kienan
/
puppet-strings
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Merge pull request #55 from iankronquist/emit-json-pdoc-23 

(PDOC-23) Emit json
This commit is contained in:
Hailee Kenney 2015-10-19 08:07:21 -07:00
parent d033183002 86414df8de
commit 93f0dd923b
16 changed files with 475 additions and 57 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
README.md
View File

@ -90,6 +90,20 @@ To document specific files:
$ puppet strings some_manifest.pp [another_if_you_feel_like_it.rb]
```
Strings can also emit the generated documentation as JSON:
```
$ puppet strings some_manifest.pp --emit-json documentation.json
```
It can also print the JSON to stdout:
```
$ puppet strings some_manifest.pp --emit-json-stdout
```
The schema for the JSON which Strings emits is [well documented](json_dom.md).
Processing is delegated to the `yardoc` tool so some options listed in `yard
help doc` are available. However, Puppet Faces do not support passing
arbitrary options through a face so these options must be specified in a

121
json_dom.md Normal file
View File

@ -0,0 +1,121 @@
The Strings JSON Interchange Schema
===================================
Strings has two flags used to emit json.
* `--emit-json $FILE` Saves json to a file.
* `--emit-json-stdout` Prints json on stdout.
Top Level Structure
-------------------
The json outputted by strings is a single object which has 4 keys representing
the different types of Puppet code and extension functions Strings reads. The
value for each key is a list of json objects representing each puppet class,
function, etc.
Here is an example of the top level structure:
```json
{
"defined_types": [...],
"puppet_classes": [...],
"puppet_functions": [...],
"puppet_types": [...],
"puppet_providers": [...]
}
```
Defined Types
-------------
Each defined type or puppet class object has the following properties and values:
* `name`: A string representing the name of the object.
* `file`: The file the object came from. A string.
* `line`: The line in the file the object came from. A number.
* `docstring`: A string. The docstring describing the object.
* `examples`: A list of strings representing the content of the examples in the
docstring.
* `signatures`: A list of function signatures which may be supported by the
object. Each function signature is a json object whose keys are the
parameter names, and whose values are the types those parameters may take.
This is extracted from the code itself.
* `parameters`: An object whose keys are the parameter names and whose values
are the parameter's types or null if it has no types. This is extracted from
the docstring.
Puppet Functions
----------------
Both puppet 4x and 3x functions are represented as json objects kept in the
`puppet_functions` list. Puppet 4x functions have every property that 3x
functions have, as well as a few extras.
Puppet 3x functions have:
* `name`: A string representing the name of the
* `file`: The file the object came from. A string.
* `line`: The line in the file the object came from. A number.
* `docstring`: A string. The docstring describing our object.
* `function_api_version`: the number 3.
* `documented_params`: A object whose keys are the parameters which were
* documented and whose values are the types they may take, or null.
* `examples`: A list of strings representing the content of the examples in the
docstring.
Puppet 4x functions have everything 3x functions do as well as:
* The `function_api_version` is the number 4, not 3 (surprise!).
* `signatures`: A list of function signatures which may be supported by the
object. Each function signature is a json object whose keys are the parameter
names, and whose values are the types those parameters may take. This is
extracted from the code itself.
Puppet Types
------------
Each puppet type object has the following properties and values:
* `name`: A string representing the name of the object
* `file`: The file the object came from. A string.
* `line`: The line in the file the object came from. A number.
* `docstring`: A string. The docstring describing our object.
* `examples`: A list of strings representing the content of the examples in the
docstring.
* `parameters`: A list of objects with the following shape:
* `allowed_vales`: a list of strings representing the allowed values.
* `default`: a string or null.
* `docstring`: The docstring.
* `name`: the parameter name.
* `properties`: A list of objects with a shape very similar to parameters but
also including:
* `namevar`: A boolean.
* `features`: A list of objects representing possible features. They have the
following shape:
* `docstring`: The description of the feature.
* `methods`: null or a list of the available methods as strings.
* `name`: The feature's name.
Puppet Providers
----------------
Each puppet provider object has the following properties and values:
* `name`: A string representing the name of the object
* `file`: The file the object came from. A string.
* `line`: The line in the file the object came from. A number.
* `docstring`: A string. The docstring describing the object.
* `examples`: A list of strings representing the content of the examples in the
docstring.
* `commands`: A list of the names of the commands available.
* `confines`: An object whose keys are the confine keys and whose values are
the confine values.
* `defaults`: Similar to above.
* `features`: A list of strings representing the features this provider
supports.
* `type_name`: The type this provider accompanies.

7
lib/puppet/face/strings.rb
View File

@ -22,6 +22,13 @@ Puppet::Face.define(:strings, '0.0.1') do
action(:yardoc) do
default
option "--emit-json-stdout" do
summary "Print json representation of the documentation to stdout"
end
option "--emit-json FILE" do
summary "Write json representation of the documentation to FILE"
end
summary "Generate YARD documentation from files."
arguments "[manifest_file.pp ...]"

1
lib/puppet_x/puppetlabs/strings.rb
View File

@ -29,6 +29,7 @@ module PuppetX::PuppetLabs
# aspects of puppet code in YARD's Registry
module CodeObjects
require 'puppet_x/puppetlabs/strings/yard/code_objects/puppet_namespace_object'
require 'puppet_x/puppetlabs/strings/yard/code_objects/method_object'
require 'puppet_x/puppetlabs/strings/yard/code_objects/defined_type_object'
require 'puppet_x/puppetlabs/strings/yard/code_objects/host_class_object'
require 'puppet_x/puppetlabs/strings/yard/code_objects/type_object'

8
lib/puppet_x/puppetlabs/strings/util.rb
View File

@ -14,12 +14,18 @@ module PuppetX::PuppetLabs::Strings::Util
# YARD options are passed to it. The best way to approach this problem is
# by using the `.yardopts` file. YARD will autoload any options placed in
# that file.
args.pop
options = args.pop
YARD::Config.options = YARD::Config.options.merge(options)
# For now, assume the remaining positional args are a list of manifest
# and ruby files to parse.
yard_args = (args.empty? ? MODULE_SOURCEFILES : args)
# If json is going to be emitted to stdout, suppress statistics.
if options[:emit_json_stdout]
yard_args.push('--no-stats')
end
# This line monkeypatches yard's progress indicator so it doesn't write
# all over the terminal. This should definitely not be in real code, but
# it's very handy for debugging with pry

26
lib/puppet_x/puppetlabs/strings/yard/code_objects/defined_type_object.rb
View File

@ -1,7 +1,33 @@
require 'json'
class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::DefinedTypeObject < PuppetX::PuppetLabs::Strings::YARD::CodeObjects::PuppetNamespaceObject
# A list of parameters attached to this class.
# @return [Array<Array(String, String)>]
attr_accessor :parameters
attr_accessor :type_info
def to_s
name.to_s
end
def to_json(*a)
{
"name" => @name,
"file" => file,
"line" => line,
"parameters" => Hash[@parameters],
"docstring" => Puppet::Util::Docs.scrub(@docstring),
"signatures" => @type_info.map do |signature|
signature.map do |key, value|
{
"name" => key,
"type" => value,
}
end
end,
"examples" => self.tags.map do |tag|
tag.text if tag.tag_name == 'example'
end.compact,
}.to_json(*a)
end
end

2
lib/puppet_x/puppetlabs/strings/yard/code_objects/host_class_object.rb
View File

@ -2,8 +2,6 @@ class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::HostClassObject < PuppetX
# The {HostClassObject} that this class inherits from, if any.
# @return [HostClassObject, Proxy, nil]
attr_accessor :parent_class
attr_accessor :type_info
# NOTE: `include_mods` is never used as it makes no sense for Puppet, but
# this is called by `YARD::Registry` and it will pass a parameter.

62
lib/puppet_x/puppetlabs/strings/yard/code_objects/method_object.rb Normal file
View File

@ -0,0 +1,62 @@
class YARD::CodeObjects::MethodObject
# Override to_s and to_json methods in Yard's MethodObject so that they
# return output formatted as I like for puppet 3x and 4x methods.
def to_s
if self[:puppet_4x_function] || self[:puppet_3x_function]
name.to_s
else
super
end
end
def to_json(*a)
if self[:puppet_4x_function]
{
"name" => @name,
"file" => file,
"line" => line,
"function_api_version" => 4,
"docstring" => Puppet::Util::Docs.scrub(@docstring),
"examples" => self.tags.map do |tag|
tag.text if tag.tag_name == 'example'
end.compact,
"documented_params" => @parameters.map do |tuple|
{
"name" => tuple[0],
"type" => tuple[1],
}
end,
"signatures" => @type_info.map do |signature|
signature.map do |key, value|
{
"name" => key,
"type" => value,
}
end
end,
}.to_json(*a)
elsif self[:puppet_3x_function]
{
"name" => @name,
"file" => file,
"line" => line,
"function_api_version" => 3,
"docstring" => Puppet::Util::Docs.scrub(@docstring),
"documented_params" => @parameters.map do |tuple|
{
"name" => tuple[0],
"type" => tuple[1],
}
end,
"examples" => self.tags.map do |tag|
tag.text if tag.tag_name == 'example'
end.compact,
}.to_json(*a)
else
super
end
end
end

19
lib/puppet_x/puppetlabs/strings/yard/code_objects/provider_object.rb
View File

@ -2,4 +2,23 @@ class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::ProviderObject < PuppetX:
# A list of parameters attached to this class.
# @return [Array<Array(String, String)>]
attr_accessor :parameters
def to_json(*a)
{
"name" => @name,
"type_name" => @type_name,
"file" => file,
"line" => line,
"docstring" => Puppet::Util::Docs.scrub(@docstring),
"commands" => @commands,
"confines" => @confines,
"defaults" => @defaults,
"features" => @features,
"examples" => self.tags.map do |tag|
tag.text if tag.tag_name == 'example'
end.compact,
}.to_json(*a)
end
end

18
lib/puppet_x/puppetlabs/strings/yard/code_objects/puppet_namespace_object.rb
View File

@ -1,11 +1,27 @@
class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::PuppetNamespaceObject < YARD::CodeObjects::NamespaceObject
attr_accessor :type_info
# NOTE: `YARD::Registry#resolve` requires a method with this signature to
# be present on all subclasses of `NamespaceObject`.
def inheritance_tree(include_mods = false)
[self]
end
attr_accessor :type_info
def to_s
name.to_s
end
def to_json(*a)
{
"name" => @name,
"file" => file,
"line" => line,
"docstring" => @docstring,
"examples" => self.tags.map do |tag|
tag.text if tag.tag_name == 'example'
end.compact,
}.to_json(*a)
end
# FIXME: We used to override `self.new` to ensure no YARD proxies were
# created for namespaces segments that did not map to a host class or

37
lib/puppet_x/puppetlabs/strings/yard/code_objects/type_object.rb
View File

@ -2,4 +2,41 @@ class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::TypeObject < PuppetX::Pup
# A list of parameters attached to this class.
# @return [Array<Array(String, String)>]
attr_accessor :parameters
def to_json(*a)
{
"name" => @name,
"file" => file,
"line" => line,
"docstring" => Puppet::Util::Docs.scrub(@docstring),
"parameters" => @parameter_details.map do |obj|
{
"allowed_values" => obj[:allowed_values] ? obj[:allowed_values].flatten : [],
"default" => obj[:default],
"docstring" => Puppet::Util::Docs.scrub(obj[:desc] || ''),
"namevar" => obj[:namevar],
"name" => obj[:name],
}
end,
"examples" => self.tags.map do |tag|
tag.text if tag.tag_name == 'example'
end.compact,
"properties" => @property_details.map do |obj|
{
"allowed_values" => obj[:allowed_values] ? obj[:allowed_values].flatten : [],
"default" => obj[:default],
"docstring" => Puppet::Util::Docs.scrub(obj[:desc] || ''),
"name" => obj[:name],
}
end,
"features" => @features.map do |obj|
{
"docstring" => Puppet::Util::Docs.scrub(obj[:desc] || ''),
"methods" => obj[:methods],
"name" => obj[:name],
}
end,
}.to_json(*a)
end
end

1
lib/puppet_x/puppetlabs/strings/yard/handlers/puppet_3x_function_handler.rb
View File

@ -10,6 +10,7 @@ class PuppetX::PuppetLabs::Strings::YARD::Handlers::Puppet3xFunctionHandler < YA
name, options = @heredoc_helper.process_parameters statement
obj = MethodObject.new(function_namespace, name)
obj[:puppet_3x_function] = true
register obj
if options['doc']

22
lib/puppet_x/puppetlabs/strings/yard/handlers/type_handler.rb
View File

@ -78,11 +78,9 @@ class PuppetX::PuppetLabs::Strings::YARD::Handlers::PuppetTypeHandler < YARD::Ha
parameter_details = []
property_details = []
features = []
obj = TypeObject.new(:root, "#{name}_type") do |o|
# FIXME: This block gets yielded twice for whatever reason
parameter_details = []
property_details = []
o.parameters = []
obj = TypeObject.new(:root, name)
obj.parameters = []
# Find the do block following the Type.
do_block = statement.jump(:do_block)
# traverse the do block's children searching for function calls whose
@ -105,7 +103,7 @@ class PuppetX::PuppetLabs::Strings::YARD::Handlers::PuppetTypeHandler < YARD::Ha
param_name = node.children[1].jump(:kw)
end
param_name = param_name.source
o.parameters << [param_name, nil]
obj.parameters << [param_name, nil]
parameter_details << {:name => param_name,
:desc => fetch_description(node), :exists? => true,
:puppet_type => true,
@ -131,7 +129,17 @@ class PuppetX::PuppetLabs::Strings::YARD::Handlers::PuppetTypeHandler < YARD::Ha
}
elsif is_feature? node
features << get_feature(node)
end
elsif is_a_func_call_named? 'ensurable', node
# Someone could call the ensurable method and create an ensure
# property. If that happens, they it will be documented twice. Serves
# them right.
property_details << {:name => 'ensure',
:desc => '', :exists? => true,
:default => nil,
:puppet_type => true,
:property => true,
:allowed_values => [],
}
end
end
obj.parameter_details = parameter_details

85
lib/puppet_x/puppetlabs/strings/yard/json_registry_store.rb Normal file
View File

@ -0,0 +1,85 @@
module YARD
class JsonRegistryStore < RegistryStore
def save(merge=true, file=nil)
super
@serializer = Serializers::JsonSerializer.new(@file)
sdb = Registry.single_object_db
if sdb == true || sdb == nil
serialize_output_schema(@store)
else
values(false).each do |object|
serialize_output_schema(object)
end
end
true
end
# @param obj [Hash] A hash representing the registry or part of the
# registry.
def serialize_output_schema(obj)
schema = {
:puppet_functions => [],
:puppet_providers => [],
:puppet_classes => [],
:defined_types => [],
:puppet_types => [],
}
schema[:puppet_functions] += obj.select do |key, val|
val.type == :method and (val['puppet_4x_function'] or
val['puppet_3x_function'])
end.values
schema[:puppet_classes] += obj.select do |key, val|
val.type == :hostclass
end.values
schema[:defined_types] += obj.select do |key, val|
val.type == :definedtype
end.values
schema[:puppet_providers] += obj.select do |key, val|
val.type == :provider
end.values
schema[:puppet_types] += obj.select do |key, val|
val.type == :type
end.values
@serializer.serialize(schema.to_json)
end
end
# Override the serializer because it puts the data at a wacky path and, more
# importantly, marshals the data with a bunch of non-printable characters.
module Serializers
class JsonSerializer < YardocSerializer
def initialize o
super
@options = {
:basepath => '.',
:extension => 'json',
}
@extension = 'json'
@basepath = '.'
end
def serialize(data)
if YARD::Config.options[:emit_json]
path = YARD::Config.options[:emit_json]
log.debug "Serializing json to #{path}"
File.open!(path, "wb") {|f| f.write data }
end
if YARD::Config.options[:emit_json_stdout]
puts data
end
end
end
end
end

17
lib/puppet_x/puppetlabs/strings/yard/monkey_patches.rb
View File

@ -1,4 +1,5 @@
require 'yard'
require File.join(File.dirname(__FILE__), './json_registry_store')
# TODO: As far as I can tell, monkeypatching is the officially recommended way
# to extend these tools to cover custom usecases. Follow up on the YARD mailing
@ -49,3 +50,19 @@ class YARD::Logger
f.close()
end
end
# 15:04:42 radens | lsegal: where would you tell yard to use your custom RegistryStore?
# 15:09:54 @lsegal | https://github.com/lsegal/yard/blob/master/lib/yard/registry.rb#L428-L435
# 15:09:54 @lsegal | you would set that attr on Registry
# 15:09:54 @lsegal | it might be worth expanding that API to swap out the store class used
# 15:10:49 @lsegal | specifically
# | https://github.com/lsegal/yard/blob/master/lib/yard/registry.rb#L190 and
# | replace RegistryStore there with a storage_class attr
module YARD::Registry
class << self
def clear
self.thread_local_store = YARD::JsonRegistryStore.new
end
end
end

2
spec/unit/puppet_x/puppetlabs/strings/yard/type_handler_spec.rb
View File

@ -7,7 +7,7 @@ describe PuppetX::PuppetLabs::Strings::YARD::Handlers::PuppetTypeHandler do
include StringsSpec::Parsing
def the_type()
YARD::Registry.at("file_type")
YARD::Registry.at("file")
end
it "should have the proper docstring" do