Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg assigned an issue to Unassigned Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Henrik Lindberg Assignee: Henrik Lindberg Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Thomas Hallgren assigned an issue to Henrik Lindberg Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Thomas Hallgren Assignee: Thomas Hallgren Henrik Lindberg Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg assigned an issue to Thomas Hallgren Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Henrik Lindberg Assignee: Henrik Lindberg Thomas Hallgren Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Thomas Hallgren assigned an issue to Henrik Lindberg Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Thomas Hallgren Assignee: Henrik Lindberg Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Thomas Hallgren assigned an issue to Henrik Lindberg Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Thomas Hallgren Assignee: Thomas Hallgren Henrik Lindberg Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg assigned an issue to Thomas Hallgren Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Henrik Lindberg Assignee: Thomas Hallgren Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Craig Gomes updated an issue Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Craig Gomes Sprint: PDS 2016-10-19, PDS 2016-11-02 , PDS 2016-11-16 Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Craig Gomes updated an issue Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Craig Gomes Sprint: PDS 2016-10-19 , PDS 2016-11-02 Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg commented on PUP-6780 Re: modify the lookup helper implementation to support v5 features Use option #2 - that is better. Wer can work on the UX of the explain output later if it turns out to be too much data or hard to understand. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Thomas Hallgren commented on PUP-6780 Re: modify the lookup helper implementation to support v5 features Question regarding explain and lookup options. When the lookup explanation was introduced, we decided to exempt the lookup of the lookup_options key from the explanation and instead provide a specific --explain-options option to the lookup command. This creates a problem. A data_hash function is called only once and will produce the full hash (it will include the lookup_options key if present). If the explanation support is disabled while the framework performs its initial assembly of the lookup_options hash, this means that whatever a data_hash sends to the explainer will be lost. This effectively renders the Context#explain() method useless for all data_hash methods since the initial call to produce the hash always origins from the lookup_options lookup. I see two possibilities for fixing this: 1. When explanation support is turned on, a data_hash function will always be called twice. Once for lookup_options with the explanation support disabled, and then once again, when the real lookup is made, with the explanation support enabled. 2. Always include the lookup_options in the explanation output and deprecate the --explain-options flag. I would prefer #2. My motivation is that a) it's inconsistent to use a different behavior when explanation is turned on, b) the explanation is also used for debugging, and c) although fairly verbose, it's actually nice to always see the lookup_options output. It sheds light on in what order things happen and that's the whole point with the explanation support. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg commented on PUP-6780 Re: modify the lookup helper implementation to support v5 features In your example with 'first found', the higher priority entry defines all of 'm::x' - that is, 'first found' is applied to the root key. And agree that lookup_options only applies to root keys. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title
Thomas Hallgren commented on PUP-6780
Re: modify the lookup helper implementation to support v5 features
I have a couple of issues in how to resolve merges in combination with functions that digs versus functions that doesn't.
In order to support both types of functions, the key must be passed in full down to the provider that makes the choice whether to make the called function perform the subkey resolution (data_dig) or not (data_hash and lookup_key). In case of the latter, the caller will instead resolve the key. So far, so good.
The problem arises when we want to perform merges. Let's assume that we have two hash_data functions configured, one for the environment and one for a module. They produce the following hashes: Environment
{
m::x => { a => 'env xa' }
}
{
'm::x' => { a => 'm xa', b => 'm xb' },
}
Lookup will use the merge strategy 'first found' which means that a lookup for 'm::x' will result in the hash from the environment and a lookup for 'm::x.a' will result in 'env xa'. Now the question arises, what should happen when we lookup 'm::x.b' ? There are two options:
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg commented on PUP-6780 Re: modify the lookup helper implementation to support v5 features The ticket PUP-6511 is for the format of the lookup.conf - posting there about concerns regarding use of hocon. Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title
Henrik Lindberg commented on PUP-6780
Re: modify the lookup helper implementation to support v5 features
In general, the features of a "data language" like hocon are at a lower level. The right choice IMO is to just read the hocon data in an opaque way for data just as other formats may contain other interpolation or reference features. The worst case then is getting two levels of interpolation and that there may be interference between the two syntaxes and more things to escape. (Something like the "verbatim" feature we discussed does not help with stopping hocon interpolation from taking place I assume since it is at a higher level).
For lookup.conf - which is the configuration format it is more a UX issue than a technical. How we do this is also related to a future rewrite of "puppet settings"
1.a We decided letting the format/encoding in the file decide which kind of interpolation to use by the framework when it gets a string value. Since we only support current "hiera" % style at the start that is what you get. If you decided to use hocon interpolation that is done at the lower level long before lookup's interpolation.
1.b Same here - we decided to make this configurable and that it is really determined by the data format/encoding
1.c Expressing PP style interpolation in Hocon seems like a challenge - can hocon interpolation be turned off ?
2. The %{} notation needs to be valid since that is the only interpolation we have. The ${} syntax in Hocon is at lower level.
3. Digging - is that really digging, rather than an alternative way of setting a value in a hash? We still need to do the digging in lookup at higher levels during a lookup since not all formats supports that. They must all behave the same way since at the user level the support cannot vary depending on data format. (Also note, an implementation of a data_dig style data provider function must implement digging with lookup semantics).
Ideally, users should only have to learn one interpolation syntax. Since we already have Puppet's it would be ideal if that could be used everywhere. The only place we can change is in lookup.conf, but then only if it does not clash with Hocon interpolation.
Add Comment
This message was sent by Atlassian JIRA
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title
Thomas Hallgren commented on PUP-6780
Re: modify the lookup helper implementation to support v5 features
Some concerns around Hocon support.
We currently allow interpolation in the hiera.yaml config file and we control this interpolation our selves. The format uses is % {xxx}. We forbid the use of the "hiera function" syntax (i.e. %{func("xxx")}). Hocon has it's own interpolation format which is similar (but not equal to) puppet. It uses ${xxx}
syntax. One problem with this is that the '% {xxx}
' syntax is used throughout by lookup. I.e. it's consistently used. This raises a couple of questions.
1. Hocon seems well designed. It should be no problem to use the ${} support that it provides for the config files. But if we do: 1a. What about a Hocon provider? Should it too use the ${} instead of the %{} syntax? 1b. What about the general support in lookup. Should it different and still use %{}? 1c. What about the differences between PP interpolation and Hocon interpolation (becomes especially interesting if we introduce a PP based provider). 2. If we want to retain support for %{} syntax in Hocon, what about the ${} syntax? Should it be interchangeable or have different meaning?
Hocon also understands keys with dot notation and treats this:
foo.bar : 42
equivalent to:
foo { bar : 42 }
Do we utilize that or should "digging" still be performed by lookup?
Add Comment
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title
Thomas Hallgren updated an issue
Puppet / PUP-6780
modify the lookup helper implementation to support v5 features
Change By:
Thomas Hallgren
Comment:
Some concerns around Hocon support.We currently allow interpolation in the {{hiera.yaml}} config file and we control this interpolation our selves. The format uses is %{xxx}. We forbid the use of the "hiera function" syntax (i.e. %{func("xxx")}). Hocon has it's own interpolation format which is similar (but not equal to) puppet. It uses ${xxx} syntax.
Add Comment
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title
Thomas Hallgren commented on PUP-6780
Re: modify the lookup helper implementation to support v5 features
Some concerns around Hocon support.
We currently allow interpolation in the hiera.yaml config file and we control this interpolation our selves. The format uses is % {xxx}. We forbid the use of the "hiera function" syntax (i.e. %{func("xxx")}). Hocon has it's own interpolation format which is similar (but not equal to) puppet. It uses ${xxx}
syntax.
Add Comment
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
--
You received this message because you are subscribed to the Google Groups "Puppet Bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-bugs@googlegroups.com.
Visit this group at https://groups.google.com/group/puppet-bugs.
For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg assigned an issue to Thomas Hallgren Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Change By: Henrik Lindberg Assignee: Thomas Hallgren Add Comment This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at https://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.
Jira (PUP-6780) modify the lookup helper implementation to support v5 features
Title: Message Title Henrik Lindberg created an issue Puppet / PUP-6780 modify the lookup helper implementation to support v5 features Issue Type: New Feature Assignee: Unassigned Created: 2016/10/04 3:36 PM Priority: Normal Reporter: Henrik Lindberg Since at runtime, we need to support both the v4 and v5 set of functionality (by reading the hiera.yaml in version 4, and lookup.conf in version 5), and then configuring the lookup layers/data providers, this is easiest to do by simply adding support for the new feature to the lookup runtime/helper. It needs to support the lookup.conf format and the three function APIs such that: 'data_hash' functions are called with the specified API, defaulting interpolation to be on and compatible with hiera / current lookup interpolation 'lookup_key' functions are called with the specified API, using no interpolation on returned strings by default (verbatim) 'data_dig' functions are called with the specified API, using no interpolation on returned strings by default (verbatim) Each call made should provide the options (immutable) to the function and should supply an invocation object to the function where the cache in the invocation is for that part (i.e. that specific call to the function in the hierarchy (this because the same function may be used in another entry and each entry must get its own cache).
