Merge pull request #55 from iankronquist/emit-json-pdoc-23

(PDOC-23) Emit json

README.md

@@ -90,6 +90,20 @@ To document specific files:
 $ puppet strings some_manifest.pp [another_if_you_feel_like_it.rb]
 ```
 
+Strings can also emit the generated documentation as JSON:
+
+```
+$ puppet strings some_manifest.pp --emit-json documentation.json
+```
+
+It can also print the JSON to stdout:
+
+```
+$ puppet strings some_manifest.pp --emit-json-stdout
+```
+
+The schema for the JSON which Strings emits is [well documented](json_dom.md).
+
 Processing is delegated to the `yardoc` tool so some options listed in `yard
 help doc` are available.  However, Puppet Faces do not support passing
 arbitrary options through a face so these options must be specified in a

json_dom.md

@@ -0,0 +1,121 @@
+The Strings JSON Interchange Schema
+===================================
+
+Strings has two flags used to emit json.
+* `--emit-json $FILE` Saves json to a file.
+* `--emit-json-stdout` Prints json on stdout.
+
+Top Level Structure
+-------------------
+
+The json outputted by strings is a single object which has 4 keys representing
+the different types of Puppet code and extension functions Strings reads. The
+value for each key is a list of json objects representing each puppet class,
+function, etc.
+Here is an example of the top level structure:
+
+```json
+{
+
+"defined_types": [...],
+
+"puppet_classes": [...],
+
+"puppet_functions": [...],
+
+"puppet_types": [...],
+
+"puppet_providers": [...]
+
+}
+```
+
+Defined Types
+-------------
+
+Each defined type or puppet class object has the following properties and values:
+
+* `name`: A string representing the name of the object.
+* `file`: The file the object came from. A string.
+* `line`: The line in the file the object came from. A number.
+* `docstring`: A string. The docstring describing the object.
+* `examples`: A list of strings representing the content of the examples in the
+   docstring.
+* `signatures`: A list of function signatures which may be supported by the
+  object. Each function signature is a json object whose keys are the
+  parameter names, and whose values are the types those parameters may take.
+  This is extracted from the code itself.
+* `parameters`: An object whose keys are the parameter names and whose values
+  are the parameter's types or null if it has no types. This is extracted from
+  the docstring.
+
+
+Puppet Functions
+----------------
+
+Both puppet 4x and 3x functions are represented as json objects kept in the
+`puppet_functions` list. Puppet 4x functions have every property that 3x
+functions have, as well as a few extras.
+
+Puppet 3x functions have:
+
+* `name`: A string representing the name of the 
+* `file`: The file the object came from. A string.
+* `line`: The line in the file the object came from. A number.
+* `docstring`: A string. The docstring describing our object.
+* `function_api_version`: the number 3.
+* `documented_params`: A object whose keys are the parameters which were
+* documented and whose values are the types they may take, or null.
+* `examples`: A list of strings representing the content of the examples in the
+   docstring.
+
+Puppet 4x functions have everything 3x functions do as well as:
+
+* The `function_api_version` is the number 4, not 3 (surprise!).
+* `signatures`: A list of function signatures which may be supported by the
+  object. Each function signature is a json object whose keys are the parameter
+  names, and whose values are the types those parameters may take. This is
+  extracted from the code itself.
+
+Puppet Types
+------------
+
+Each puppet type object has the following properties and values:
+
+* `name`: A string representing the name of the object
+* `file`: The file the object came from. A string.
+* `line`: The line in the file the object came from. A number.
+* `docstring`: A string. The docstring describing our object.
+* `examples`: A list of strings representing the content of the examples in the
+   docstring.
+* `parameters`: A list of objects with the following shape:
+    * `allowed_vales`: a list of strings representing the allowed values.
+    * `default`: a string or null.
+    * `docstring`: The docstring.
+    * `name`: the parameter name.
+* `properties`: A list of objects with a shape very similar to parameters but
+  also including:
+    * `namevar`: A boolean.
+* `features`: A list of objects representing possible features. They have the
+  following shape:
+    * `docstring`: The description of the feature.
+    * `methods`: null or a list of the available methods as strings.
+    * `name`: The feature's name.
+
+Puppet Providers
+----------------
+Each puppet provider object has the following properties and values:
+
+* `name`: A string representing the name of the object
+* `file`: The file the object came from. A string.
+* `line`: The line in the file the object came from. A number.
+* `docstring`: A string. The docstring describing the object.
+* `examples`: A list of strings representing the content of the examples in the
+   docstring.
+* `commands`: A list of the names of the commands available.
+* `confines`: An object whose keys are the confine keys and whose values are
+  the confine values.
+* `defaults`: Similar to above.
+* `features`: A list of strings representing the features this provider
+  supports.
+* `type_name`: The type this provider accompanies.

lib/puppet/face/strings.rb

@@ -22,6 +22,13 @@ Puppet::Face.define(:strings, '0.0.1') do
   action(:yardoc) do
     default
 
+    option "--emit-json-stdout" do
+      summary "Print json representation of the documentation to stdout"
+    end
+    option "--emit-json FILE" do
+      summary "Write json representation of the documentation to FILE"
+    end
+
     summary "Generate YARD documentation from files."
     arguments "[manifest_file.pp ...]"
 

lib/puppet_x/puppetlabs/strings.rb

@@ -29,6 +29,7 @@ module PuppetX::PuppetLabs
       # aspects of puppet code in YARD's Registry
       module CodeObjects
         require 'puppet_x/puppetlabs/strings/yard/code_objects/puppet_namespace_object'
+        require 'puppet_x/puppetlabs/strings/yard/code_objects/method_object'
         require 'puppet_x/puppetlabs/strings/yard/code_objects/defined_type_object'
         require 'puppet_x/puppetlabs/strings/yard/code_objects/host_class_object'
         require 'puppet_x/puppetlabs/strings/yard/code_objects/type_object'

lib/puppet_x/puppetlabs/strings/util.rb

@@ -14,12 +14,18 @@ module PuppetX::PuppetLabs::Strings::Util
     # YARD options are passed to it. The best way to approach this problem is
     # by using the `.yardopts` file. YARD will autoload any options placed in
     # that file.
-    args.pop
+    options = args.pop
+    YARD::Config.options = YARD::Config.options.merge(options)
 
     # For now, assume the remaining positional args are a list of manifest
     # and ruby files to parse.
     yard_args = (args.empty? ? MODULE_SOURCEFILES : args)
 
+    # If json is going to be emitted to stdout, suppress statistics.
+    if options[:emit_json_stdout]
+      yard_args.push('--no-stats')
+    end
+
     # This line monkeypatches yard's progress indicator so it doesn't write
     # all over the terminal. This should definitely not be in real code, but
     # it's very handy for debugging with pry

lib/puppet_x/puppetlabs/strings/yard/code_objects/defined_type_object.rb

@@ -1,7 +1,33 @@
+require 'json'
+
 class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::DefinedTypeObject < PuppetX::PuppetLabs::Strings::YARD::CodeObjects::PuppetNamespaceObject
   # A list of parameters attached to this class.
   # @return [Array<Array(String, String)>]
   attr_accessor :parameters
   attr_accessor :type_info
 
+  def to_s
+    name.to_s
+  end
+
+  def to_json(*a)
+    {
+      "name"             => @name,
+      "file"             => file,
+      "line"             => line,
+      "parameters"       => Hash[@parameters],
+      "docstring"        => Puppet::Util::Docs.scrub(@docstring),
+      "signatures"       => @type_info.map do |signature|
+        signature.map do |key, value|
+          {
+            "name" => key,
+            "type" => value,
+          }
+        end
+      end,
+      "examples"              => self.tags.map do |tag|
+          tag.text if tag.tag_name == 'example'
+      end.compact,
+    }.to_json(*a)
+  end
 end

lib/puppet_x/puppetlabs/strings/yard/code_objects/host_class_object.rb

@@ -2,8 +2,6 @@ class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::HostClassObject < PuppetX
   # The {HostClassObject} that this class inherits from, if any.
   # @return [HostClassObject, Proxy, nil]
   attr_accessor :parent_class
-  attr_accessor :type_info
-
 
   # NOTE: `include_mods` is never used as it makes no sense for Puppet, but
   # this is called by `YARD::Registry` and it will pass a parameter.

lib/puppet_x/puppetlabs/strings/yard/code_objects/method_object.rb

@@ -0,0 +1,62 @@
+class YARD::CodeObjects::MethodObject
+
+  # Override to_s and to_json methods in Yard's MethodObject so that they
+  # return output formatted as I like for puppet 3x and 4x methods.
+  def to_s
+    if self[:puppet_4x_function] || self[:puppet_3x_function]
+      name.to_s
+    else
+      super
+    end
+  end
+
+  def to_json(*a)
+    if self[:puppet_4x_function]
+      {
+        "name"                  => @name,
+        "file"                  => file,
+        "line"                  => line,
+        "function_api_version"  => 4,
+        "docstring"             => Puppet::Util::Docs.scrub(@docstring),
+        "examples"              => self.tags.map do |tag|
+            tag.text if tag.tag_name == 'example'
+          end.compact,
+        "documented_params"     => @parameters.map do |tuple|
+          {
+            "name" => tuple[0],
+            "type" => tuple[1],
+          }
+        end,
+       "signatures"       => @type_info.map do |signature|
+          signature.map do |key, value|
+            {
+              "name" => key,
+              "type" => value,
+            }
+          end
+        end,
+      }.to_json(*a)
+    elsif self[:puppet_3x_function]
+      {
+        "name"                  => @name,
+        "file"                  => file,
+        "line"                  => line,
+        "function_api_version"  => 3,
+        "docstring"             => Puppet::Util::Docs.scrub(@docstring),
+        "documented_params"     => @parameters.map do |tuple|
+          {
+            "name" => tuple[0],
+            "type" => tuple[1],
+          }
+        end,
+        "examples"              => self.tags.map do |tag|
+            tag.text if tag.tag_name == 'example'
+          end.compact,
+      }.to_json(*a)
+    else
+      super
+    end
+  end
+
+
+end

lib/puppet_x/puppetlabs/strings/yard/code_objects/provider_object.rb

@@ -2,4 +2,23 @@ class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::ProviderObject < PuppetX:
   # A list of parameters attached to this class.
   # @return [Array<Array(String, String)>]
   attr_accessor :parameters
+
+  def to_json(*a)
+    {
+      "name"             => @name,
+      "type_name"        => @type_name,
+      "file"             => file,
+      "line"             => line,
+      "docstring"        => Puppet::Util::Docs.scrub(@docstring),
+      "commands"         => @commands,
+      "confines"         => @confines,
+      "defaults"         => @defaults,
+      "features"         => @features,
+      "examples"              => self.tags.map do |tag|
+          tag.text if tag.tag_name == 'example'
+      end.compact,
+    }.to_json(*a)
+  end
+
+
 end

lib/puppet_x/puppetlabs/strings/yard/code_objects/puppet_namespace_object.rb

@@ -1,11 +1,27 @@
 class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::PuppetNamespaceObject < YARD::CodeObjects::NamespaceObject
+
+  attr_accessor :type_info
   # NOTE: `YARD::Registry#resolve` requires a method with this signature to
   # be present on all subclasses of `NamespaceObject`.
   def inheritance_tree(include_mods = false)
     [self]
   end
 
-  attr_accessor :type_info
+  def to_s
+    name.to_s
+  end
+
+  def to_json(*a)
+    {
+      "name"             => @name,
+      "file"             => file,
+      "line"             => line,
+      "docstring"        => @docstring,
+      "examples"              => self.tags.map do |tag|
+          tag.text if tag.tag_name == 'example'
+      end.compact,
+    }.to_json(*a)
+  end
 
   # FIXME: We used to override `self.new` to ensure no YARD proxies were
   # created for namespaces segments that did not map to a host class or

lib/puppet_x/puppetlabs/strings/yard/code_objects/type_object.rb

@@ -2,4 +2,41 @@ class PuppetX::PuppetLabs::Strings::YARD::CodeObjects::TypeObject < PuppetX::Pup
   # A list of parameters attached to this class.
   # @return [Array<Array(String, String)>]
   attr_accessor :parameters
+
+  def to_json(*a)
+    {
+      "name"             => @name,
+      "file"             => file,
+      "line"             => line,
+      "docstring"        => Puppet::Util::Docs.scrub(@docstring),
+      "parameters"       => @parameter_details.map do |obj|
+        {
+          "allowed_values" => obj[:allowed_values] ? obj[:allowed_values].flatten : [],
+          "default"        => obj[:default],
+          "docstring"      => Puppet::Util::Docs.scrub(obj[:desc] || ''),
+          "namevar"        => obj[:namevar],
+          "name"           => obj[:name],
+        }
+      end,
+      "examples"              => self.tags.map do |tag|
+          tag.text if tag.tag_name == 'example'
+      end.compact,
+      "properties"         => @property_details.map do |obj|
+        {
+          "allowed_values" => obj[:allowed_values] ? obj[:allowed_values].flatten : [],
+          "default"        => obj[:default],
+          "docstring"      => Puppet::Util::Docs.scrub(obj[:desc] || ''),
+          "name"           => obj[:name],
+        }
+      end,
+      "features"         => @features.map do |obj|
+        {
+          "docstring" => Puppet::Util::Docs.scrub(obj[:desc] || ''),
+          "methods"   => obj[:methods],
+          "name"      => obj[:name],
+        }
+      end,
+    }.to_json(*a)
+  end
+
 end

lib/puppet_x/puppetlabs/strings/yard/handlers/puppet_3x_function_handler.rb

@@ -10,6 +10,7 @@ class PuppetX::PuppetLabs::Strings::YARD::Handlers::Puppet3xFunctionHandler < YA
     name, options = @heredoc_helper.process_parameters statement
 
     obj = MethodObject.new(function_namespace, name)
+    obj[:puppet_3x_function] = true
 
     register obj
     if options['doc']

lib/puppet_x/puppetlabs/strings/yard/handlers/type_handler.rb

@@ -78,60 +78,68 @@ class PuppetX::PuppetLabs::Strings::YARD::Handlers::PuppetTypeHandler < YARD::Ha
     parameter_details = []
     property_details = []
     features = []
-    obj = TypeObject.new(:root, "#{name}_type") do |o|
+    obj = TypeObject.new(:root, name)
-      # FIXME: This block gets yielded twice for whatever reason
+    obj.parameters = []
-      parameter_details = []
-      property_details = []
-      o.parameters = []
-      # Find the do block following the Type.
-      do_block = statement.jump(:do_block)
-      # traverse the do block's children searching for function calls whose
-      # identifier is newparam (we're calling the newparam function)
-      do_block.traverse do |node|
-        if is_param? node
-          # The first member of the parameter tuple is the parameter name.
-          # Find the second identifier node under the fcall tree. The first one
-          # is 'newparam', the second one is the function name.
-          # Get its source.
-          # The second parameter is nil because we cannot infer types for these
-          # functions. In fact, that's a silly thing to ask because ruby
-          # types were deprecated with puppet 4 at the same time the type
-          # system was created.
 
-          # Because of a ripper bug a symbol identifier is sometimes incorrectly parsed as a keyword.
+    # Find the do block following the Type.
-          # That is, the symbol `:true` will be represented as s(:symbol s(:kw, true...
+    do_block = statement.jump(:do_block)
-          param_name = node.children[1].jump(:ident)
+    # traverse the do block's children searching for function calls whose
-          if param_name == node.children[1]
+    # identifier is newparam (we're calling the newparam function)
-            param_name = node.children[1].jump(:kw)
+    do_block.traverse do |node|
-          end
+      if is_param? node
-          param_name = param_name.source
+        # The first member of the parameter tuple is the parameter name.
-          o.parameters << [param_name, nil]
+        # Find the second identifier node under the fcall tree. The first one
-          parameter_details << {:name => param_name,
+        # is 'newparam', the second one is the function name.
-            :desc => fetch_description(node), :exists? => true,
+        # Get its source.
-            :puppet_type => true,
+        # The second parameter is nil because we cannot infer types for these
-            :default => fetch_default(node),
+        # functions. In fact, that's a silly thing to ask because ruby
-            :namevar => is_namevar?(node, param_name, name),
+        # types were deprecated with puppet 4 at the same time the type
-            :parameter => true,
+        # system was created.
-            :allowed_values => get_parameter_allowed_values(node),
+
-          }
+        # Because of a ripper bug a symbol identifier is sometimes incorrectly parsed as a keyword.
-        elsif is_prop? node
+        # That is, the symbol `:true` will be represented as s(:symbol s(:kw, true...
-          # Because of a ripper bug a symbol identifier is sometimes incorrectly parsed as a keyword.
+        param_name = node.children[1].jump(:ident)
-          # That is, the symbol `:true` will be represented as s(:symbol s(:kw, true...
+        if param_name == node.children[1]
-          prop_name = node.children[1].jump(:ident)
+          param_name = node.children[1].jump(:kw)
-          if prop_name == node.children[1]
-            prop_name = node.children[1].jump(:kw)
-          end
-          prop_name = prop_name.source
-          property_details << {:name => prop_name,
-            :desc => fetch_description(node), :exists? => true,
-            :default => fetch_default(node),
-            :puppet_type => true,
-            :property => true,
-            :allowed_values => get_property_allowed_values(node),
-            }
-        elsif is_feature? node
-          features << get_feature(node)
         end
+        param_name = param_name.source
+        obj.parameters << [param_name, nil]
+        parameter_details << {:name => param_name,
+          :desc => fetch_description(node), :exists? => true,
+          :puppet_type => true,
+          :default => fetch_default(node),
+          :namevar => is_namevar?(node, param_name, name),
+          :parameter => true,
+          :allowed_values => get_parameter_allowed_values(node),
+        }
+      elsif is_prop? node
+        # Because of a ripper bug a symbol identifier is sometimes incorrectly parsed as a keyword.
+        # That is, the symbol `:true` will be represented as s(:symbol s(:kw, true...
+        prop_name = node.children[1].jump(:ident)
+        if prop_name == node.children[1]
+          prop_name = node.children[1].jump(:kw)
+        end
+        prop_name = prop_name.source
+        property_details << {:name => prop_name,
+          :desc => fetch_description(node), :exists? => true,
+          :default => fetch_default(node),
+          :puppet_type => true,
+          :property => true,
+          :allowed_values => get_property_allowed_values(node),
+        }
+      elsif is_feature? node
+        features << get_feature(node)
+      elsif is_a_func_call_named? 'ensurable', node
+        # Someone could call the ensurable method and create an ensure 
+        # property. If that happens, they it will be documented twice. Serves
+        # them right.
+        property_details << {:name => 'ensure',
+          :desc => '', :exists? => true,
+          :default => nil,
+          :puppet_type => true,
+          :property => true,
+          :allowed_values => [],
+        }
       end
     end
     obj.parameter_details = parameter_details

lib/puppet_x/puppetlabs/strings/yard/json_registry_store.rb

@@ -0,0 +1,85 @@
+module YARD
+
+  class JsonRegistryStore < RegistryStore
+    def save(merge=true, file=nil)
+      super
+
+      @serializer = Serializers::JsonSerializer.new(@file)
+
+      sdb = Registry.single_object_db
+      if sdb == true || sdb == nil
+        serialize_output_schema(@store)
+      else
+        values(false).each do |object|
+          serialize_output_schema(object)
+        end
+      end
+      true
+    end
+
+    # @param obj [Hash] A hash representing the registry or part of the
+    # registry.
+    def serialize_output_schema(obj)
+
+        schema = {
+          :puppet_functions => [],
+          :puppet_providers => [],
+          :puppet_classes => [],
+          :defined_types => [],
+          :puppet_types => [],
+        }
+
+        schema[:puppet_functions] += obj.select do |key, val|
+          val.type == :method and (val['puppet_4x_function'] or
+                                   val['puppet_3x_function'])
+        end.values
+
+        schema[:puppet_classes] += obj.select do |key, val|
+          val.type == :hostclass
+        end.values
+
+        schema[:defined_types] += obj.select do |key, val|
+          val.type == :definedtype
+        end.values
+
+        schema[:puppet_providers] += obj.select do |key, val|
+          val.type == :provider
+        end.values
+
+        schema[:puppet_types] += obj.select do |key, val|
+          val.type == :type
+        end.values
+
+        @serializer.serialize(schema.to_json)
+    end
+  end
+
+  # Override the serializer because it puts the data at a wacky path and, more
+  # importantly, marshals the data with a bunch of non-printable characters.
+  module Serializers
+    class JsonSerializer < YardocSerializer
+
+      def initialize o
+        super
+        @options = {
+          :basepath => '.',
+          :extension => 'json',
+        }
+        @extension = 'json'
+        @basepath = '.'
+      end
+      def serialize(data)
+
+        if YARD::Config.options[:emit_json]
+          path = YARD::Config.options[:emit_json]
+          log.debug "Serializing json to #{path}"
+          File.open!(path, "wb") {|f| f.write data }
+        end
+        if YARD::Config.options[:emit_json_stdout]
+          puts data
+        end
+      end
+    end
+  end
+
+end

lib/puppet_x/puppetlabs/strings/yard/monkey_patches.rb

@@ -1,4 +1,5 @@
 require 'yard'
+require File.join(File.dirname(__FILE__), './json_registry_store')
 
 # TODO: As far as I can tell, monkeypatching is the officially recommended way
 # to extend these tools to cover custom usecases. Follow up on the YARD mailing
@@ -49,3 +50,19 @@ class YARD::Logger
     f.close()
   end
 end
+
+
+# 15:04:42       radens | lsegal: where would you tell yard to use your custom RegistryStore?
+# 15:09:54      @lsegal | https://github.com/lsegal/yard/blob/master/lib/yard/registry.rb#L428-L435
+# 15:09:54      @lsegal | you would set that attr on Registry
+# 15:09:54      @lsegal | it might be worth expanding that API to swap out the store class used
+# 15:10:49      @lsegal | specifically
+#                       | https://github.com/lsegal/yard/blob/master/lib/yard/registry.rb#L190 and
+#                       | replace RegistryStore there with a storage_class attr
+module YARD::Registry
+  class << self
+  def clear
+    self.thread_local_store = YARD::JsonRegistryStore.new
+  end
+  end
+end

spec/unit/puppet_x/puppetlabs/strings/yard/type_handler_spec.rb

@@ -7,7 +7,7 @@ describe PuppetX::PuppetLabs::Strings::YARD::Handlers::PuppetTypeHandler do
   include StringsSpec::Parsing
 
   def the_type()
-    YARD::Registry.at("file_type")
+    YARD::Registry.at("file")
   end
 
   it "should have the proper docstring" do