Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Eric Thompson updated an issue Puppet / PUP-5911 Define the content of an .xpp file Change By: Eric Thompson QA Status: Reviewed Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file I think we have described the content well enough now. There are other details to work out about the placement of the .xpp files. That is being discussed now. I consider this ticket done although there is some writeup still to do in the form of a PRFC, and a PR against the specifications repository. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title
Henrik Lindberg updated an issue
Puppet / PUP-5911
Define the content of an .xpp file
Change By:
Henrik Lindberg
We need to specify the content of an .xpp file - it will naturally contain the serialized AST, but since the action of writing it, and reading it is asynchronous it is important that the encoding used in the .xpp can be learned by inspecting it. The idea is that the parser checks if there is an .xpp file and if it is newer that the corresponding .pp file. If so it will use the .xpp. There could potentially be a very long time between the .xpp is created and the time it is being used (several years). We may have made advancements even if the technology used to serialize is still based on the same technology (say MsgPack) we benefit from knowing up front about the version. We may also in the future use a different encoding; a better technology than MsgPack say) we do not want to change the extension for that. We may also want to conduct experiments with different formats, benchmarking etc. Having the parser check for more than one name/extension make it a lot more expensive.Hence - the .xpp file must have some kind of envelope that all .xpp files have. It must be something simple like a first line in ascii containing a protocol URI that defines the format (i.e. the content of the envelope is described in the content that URI identifies. This is also how we at runtime can map the content to a deserializer.UpdateAn {{.xpp}} file contains a header/envelope that consists of a shebang line followed by any number of lines up to and including a terminating {{\r?\n\r?\n}} sequence. The first byte after the end of header mark is part of the serialized payload.The first line consists of:{code}#!/opt/puppetlabs/bin/xpp +pcore {code}Where the is the base encoding used for the payload:* application/ vnd.puppet.pcore+ json* application/ vnd.puppet.pcore+ msgpack* application/ vnd.puppet.pcore+ yaml* application/vnd.puppet.pcore+ text See more here: https://en.wikipedia.org/wiki/Media_type We cannot use the standard mime types (e.g. application/json) as there is no standard way of telling that it is also pcore (it is not allowed to use a non IANA registered suffix on a type in the standard tree. It is however allowed to register "vnd." types where the suffixes are included.(There is no standard mime type for msgpack).Parameters can follow the mimetype and it consists of a list of {{;name=value}}. We may use this to describe the content of text, if json/yaml is in human readable form, epxanded/debug form, etc.
Add Comment
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title
Henrik Lindberg updated an issue
Puppet / PUP-5911
Define the content of an .xpp file
Change By:
Henrik Lindberg
We need to specify the content of an .xpp file - it will naturally contain the serialized AST, but since the action of writing it, and reading it is asynchronous it is important that the encoding used in the .xpp can be learned by inspecting it. The idea is that the parser checks if there is an .xpp file and if it is newer that the corresponding .pp file. If so it will use the .xpp. There could potentially be a very long time between the .xpp is created and the time it is being used (several years). We may have made advancements even if the technology used to serialize is still based on the same technology (say MsgPack) we benefit from knowing up front about the version. We may also in the future use a different encoding; a better technology than MsgPack say) we do not want to change the extension for that. We may also want to conduct experiments with different formats, benchmarking etc. Having the parser check for more than one name/extension make it a lot more expensive.Hence - the .xpp file must have some kind of envelope that all .xpp files have. It must be something simple like a first line in ascii containing a protocol URI that defines the format (i.e. the content of the envelope is described in the content that URI identifies. This is also how we at runtime can map the content to a deserializer. UpdateAn {{.xpp}} file contains a header/envelope that consists of a shebang line followed by any number of lines up to and including a terminating {{\r?\n\r?\n}} sequence. The first byte after the end of header mark is part of the serialized payload.The first line consists of:{code}#!/opt/puppetlabs/bin/xpp +pcore{code}Where the is the base encoding for the payload:* application/json* application/msgpack* application/yaml* text
Add Comment
This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg assigned an issue to Henrik Lindberg Puppet / PUP-5911 Define the content of an .xpp file Change By: Henrik Lindberg Assignee: Henrik Lindberg Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Thomas Hallgren commented on PUP-5911 Re: Define the content of an .xpp file Using \n\n sounds like a good idea. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file ok, I can buy that, there is enough power there to describe the variations. Do you agree to the \n\n delimiter (one blank line) to ensure we have headroom should the need arise? (It is horribly difficult to add later, and the cost is one byte). Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Thomas Hallgren commented on PUP-5911 Re: Define the content of an .xpp file I think that is overly verbose. The MIME type stats that it's JSON and pcore. What's left is the protocol used and a version. The latter should reside inside of the actual content (i.e. not an xpp file concern). #!/bin/viewpcore application/json+pcore and, if there are several variants of json/pcore #!/bin/viewpcore application/json+pcore;debug=true Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@goog
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file If we use say application/json+pcore (which i like) then the protocol URI which defines which PCore model and serializer details that were used/should be used) needs to be in the Pcore serialization that follows. The "pcore" addition to the mime type does not include everything that needs to be known. We could keep it on the same line, or use multiple lines (as suggested above) to set additional properties. I.e. we could have #!/bin/viewpcore application/json+pcore protocol=http://puppet.com/pcore/serializer/readable_json_1_0.pcore where indicates binary bytes or textual bytes Compare that against: #!/bin/viewpcore application/json+pcore protocol=http://puppet.com/pcore/serializer/machine_2_machine_json_1_0.pcore
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Thomas Hallgren commented on PUP-5911 Re: Define the content of an .xpp file I'm in favor of using a shebang line with two arguments, the executable and the MIME encoding. It must end according to MIME standard so application/json+pcore is better than splitting it into application/json pcore The serializer propotype is already capable of serializing the AST with both Locator and source code included. My preference when it comes to warnings/errors would be to use issue objects (symbolic issue codes + parameters), not pre-formatted strings. Although symbols are more expensive, they also tabulate very efficiently and I don't think we don't need to bump the protocol when we add new issues. Only thing needed is an issue reporter that understands that it should report unrecognized issues in a decent way (i.e. output the symbol and the parameter values). Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file We need to decide if we want to use arguments to Issue instead of baked instances of Diagnostic. The reason for this is that it is easier to ensure 100% compatibility if the native parser (or any other parser) generates issues codes + arguments rather than pre formatted error messages. This also makes it possible to have a later binding to human language. No sure how precise we want the modelling of that to be. Do we have an ENUM with all the error codes? Do we produce a model of the issues so that they are fully modeled? (They have a name, and attributes), they all share some common attributes. We can write a transformer that generates the PCore model that describes the issues. This however requires us to bump the protocol each time we add an error code. That pain is worse than loss of precision type wise. A concern though is that the arguments to issues must be able to pass in a reference to an AST node (which should be doable) as it is Issue and the DiagnoticProducer's responsibility to figure out the file / line from the given issue. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file When discussing the use cases where an xpp file is used we discovered that: It must contain the information that is obtained as the ParseResult from the current parser: The AST model (if it could be produced, otherwise undef The Locator - which includes the filename and line offset index The source code (it is in the Program AST top level node) An array with warnings Diagnostic instances An array with error Diagnostic instances There is leeway in how to model the errors/warnings - they could all be in one array and with a severity in the diagnostic, the error and warning arrays/counts can be derived from the all_diagnostics array. Unless the file contains all of those things it cannot work as a replacement result for doing the parse in Ruby. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file Basically there are two things needed in the envelope: The basic encoding of the body The protocol URI Using the shebang style this would be expressed as: #!/opt/puppetlabs/bin/viewxpp The mime encoding will describe the fundamental encoding of the body - e.g. application/json application/yaml application/msgpack text/* (e.g. /plan, /s-expr) If we want to we could use + in the mime content-type instead of having a separate protocol URI (or just a simplified hint say application/json+pcore). The mime content type information is intended for use cases where a piece of logic needs to understand what the body consists of, but has no interest in the actual content other than as a payload. Also, if we are to follow the MIME standard the envelope should extend to the first \\r?\\n\\r?n sequence (i.e. to the end of the first blank line). This gives us headroom for the future if we want to add things to the envelope and one line is not enough/difficult to use. Add Comment
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file An alternative header could be a HTTP response header as it is well defined and has rich semantics. Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg commented on PUP-5911 Re: Define the content of an .xpp file I can think of a couple of solutions: The first line (all bytes up to first 0x10 byte) is an envelope on the format #!/opt/puppetlabs/bin/viewxpp The .xpp file is an archive (zip etc). In this format, the protocol itself can be written instead of a file with the protocol URI in. The first written serialized value must be a MsgPack string, and this string is the protocol URI. Other binary formats must start with a MsgPack String exactly the same way even if they are not based on MsgPack. Require that there is an xpp-manifest file in the same directory as the .xpp file(s) that contains a mapping of file to protocol. Write all .xpp files in one directory to a single .xpp archive containing all of them, a manifest in the archive describes their content. There are pro's and cons with all of these. The shebang line is easy to handle. It is also self documenting if we include a script "viewxpp" that can cat the content. This should have very little performance overhead if MsgPack can continue reading on an already opened stream that has been advanced to first byte after the newline. This does not work if the file is a textual serialization (no comments in a JSON). A zip archive may be heavy to use. May be difficult to handle with MsgPack and may even require unpacking to a file first (which beats the purpose of being a fast loading format. This also works even if the serialization is textual. A compromise, works fine when reading with any MsgPack, but causes problems for other formats. Thou
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg updated an issue Puppet / PUP-5911 Define the content of an .xpp file Change By: Henrik Lindberg Sprint: Language 2016-03-09 (Burn CF) Add Comment This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-5911) Define the content of an .xpp file
Title: Message Title Henrik Lindberg created an issue Puppet / PUP-5911 Define the content of an .xpp file Issue Type: Task Assignee: Unassigned Created: 2016/02/16 5:58 PM Priority: Normal Reporter: Henrik Lindberg We need to specify the content of an .xpp file - it will naturally contain the serialized AST, but since the action of writing it, and reading it is asynchronous it is important that the encoding used in the .xpp can be learned by inspecting it. The idea is that the parser checks if there is an .xpp file and if it is newer that the corresponding .pp file. If so it will use the .xpp. There could potentially be a very long time between the .xpp is created and the time it is being used (several years). We may have made advancements even if the technology used to serialize is still based on the same technology (say MsgPack) we benefit from knowing up front about the version. We may also in the future use a different encoding; a better technology than MsgPack say) we do not want to change the extension for that. We may also want to conduct experiments with different formats, benchmarking etc. Having the parser check for more than one name/extension make it a lot more expensive. Hence - the .xpp file must have some kind of envelope that all .xpp files have. It must be something simple like a first line in ascii containing a protocol URI that defines the format (i.e. the content of the envelope is described in the content that URI identifies. This is also how we at runtime can map the content to a deserializer.
