Re: minifi-cpp config.yml case-sensitivity

[Andy Christianson]

My idea of renaming the keys shouldn’t require any changes to yaml-cpp. The 
pseudocode would be (assuming a simplification that the whole tree is a map of 
maps, plus leaf values):

normalize(yamlTree):

  if is_leaf(yamlTree):
ret yamlTree

  newTree = {}

  for key in yamlTree: 
newKey = to_lower(key)
newTree[newKey] = normalize(yamlTree[key]) 

  ret newTree

So we would just normalize all key keys in the parsed yaml tree before mapping 
the yaml config values to minifi internal config structures.

-Andy

On 7/13/17, 1:53 PM, "Marc"  wrote:

Hey Andy,
The concern I had was not with performance -- as you said, only at load
and reload time -- but more with maintainability.

 If we make the change to the third party directory we'll have the
concern that we lose this capability when someone copies a new version of
yaml-cpp into our base directory, perhaps even at review. I'm not a fan of
maintaining someone else's code.

 Can we alleviate this concern by not making the change directly to
their code in the third party directory?. I briefly entertained the thought
of using a git submodule and custom branch, but there we would have the
same issue. My thought is can we do this in such a way that a copy/paste of
an updated version makes it glaringly obvious that there are changes here.
I don't think a section of the readme is likely to suffice.

On Thu, Jul 13, 2017 at 1:40 PM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> On initial investigation, it doesn’t seem trivial to enable support for
> this in yaml-cpp. It might be feasible to walk the entire parsed yaml tree
> and map keys to a to_lower or to_upper key. It would come with some
> computational and possibly memory cost, but only at config load time, 
which
> shouldn’t be in the hot path. IMO, the usability improvement would be 
worth
> the cost.
>
> On 7/13/17, 11:16 AM, "Marc"  wrote:
>
> Hi Andy,
>   I believe this makes sense if Yaml-CPP supports the option, but I
> would
> be cautious if we have to change dependencies. If Yaml-CPP does not
> support
> this are you proposing to make the change to our third party 
directory?
>
> On Thu, Jul 13, 2017 at 11:09 AM, Andy Christianson <
> achristian...@hortonworks.com> wrote:
>
> > All,
> >
> > It appears that the properties in the config.yml are case-sensitive,
> and
> > to add to it, they’re inconsistent. E.g. in the example from the
> README.md
> > file, we have:
> >
> > …
> > Processors:
> > - name: GetFile
> >   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
> >   class: org.apache.nifi.processors.standard.GetFile
> >   max concurrent tasks: 1
> >   scheduling strategy: TIMER_DRIVEN
> >   scheduling period: 1 sec
> >   penalization period: 30 sec
> >   yield period: 1 sec
> >   run duration nanos: 0
> >   auto-terminated relationships list:
> >   Properties:
> >   Input Directory: /tmp/getfile
> >   Keep Source File: true
> > …
> >
> > The pragmatic course of action would seem to be making the config
> parser
> > case-insensitive, so that users have one less thing to worry about.
> >
> > If there is no opposition to this, I will enter a Jira ticket to
> make the
> > config parser case-insensitive.
> >
> > -Andy
> >
> >
>
>
>

Re: minifi-cpp config.yml case-sensitivity

[Marc]

Hey Andy,
The concern I had was not with performance -- as you said, only at load
and reload time -- but more with maintainability.

 If we make the change to the third party directory we'll have the
concern that we lose this capability when someone copies a new version of
yaml-cpp into our base directory, perhaps even at review. I'm not a fan of
maintaining someone else's code.

 Can we alleviate this concern by not making the change directly to
their code in the third party directory?. I briefly entertained the thought
of using a git submodule and custom branch, but there we would have the
same issue. My thought is can we do this in such a way that a copy/paste of
an updated version makes it glaringly obvious that there are changes here.
I don't think a section of the readme is likely to suffice.

On Thu, Jul 13, 2017 at 1:40 PM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> On initial investigation, it doesn’t seem trivial to enable support for
> this in yaml-cpp. It might be feasible to walk the entire parsed yaml tree
> and map keys to a to_lower or to_upper key. It would come with some
> computational and possibly memory cost, but only at config load time, which
> shouldn’t be in the hot path. IMO, the usability improvement would be worth
> the cost.
>
> On 7/13/17, 11:16 AM, "Marc"  wrote:
>
> Hi Andy,
>   I believe this makes sense if Yaml-CPP supports the option, but I
> would
> be cautious if we have to change dependencies. If Yaml-CPP does not
> support
> this are you proposing to make the change to our third party directory?
>
> On Thu, Jul 13, 2017 at 11:09 AM, Andy Christianson <
> achristian...@hortonworks.com> wrote:
>
> > All,
> >
> > It appears that the properties in the config.yml are case-sensitive,
> and
> > to add to it, they’re inconsistent. E.g. in the example from the
> README.md
> > file, we have:
> >
> > …
> > Processors:
> > - name: GetFile
> >   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
> >   class: org.apache.nifi.processors.standard.GetFile
> >   max concurrent tasks: 1
> >   scheduling strategy: TIMER_DRIVEN
> >   scheduling period: 1 sec
> >   penalization period: 30 sec
> >   yield period: 1 sec
> >   run duration nanos: 0
> >   auto-terminated relationships list:
> >   Properties:
> >   Input Directory: /tmp/getfile
> >   Keep Source File: true
> > …
> >
> > The pragmatic course of action would seem to be making the config
> parser
> > case-insensitive, so that users have one less thing to worry about.
> >
> > If there is no opposition to this, I will enter a Jira ticket to
> make the
> > config parser case-insensitive.
> >
> > -Andy
> >
> >
>
>
>

Re: minifi-cpp config.yml case-sensitivity

[Andy Christianson]

On initial investigation, it doesn’t seem trivial to enable support for this in 
yaml-cpp. It might be feasible to walk the entire parsed yaml tree and map keys 
to a to_lower or to_upper key. It would come with some computational and 
possibly memory cost, but only at config load time, which shouldn’t be in the 
hot path. IMO, the usability improvement would be worth the cost.

On 7/13/17, 11:16 AM, "Marc"  wrote:

Hi Andy,
  I believe this makes sense if Yaml-CPP supports the option, but I would
be cautious if we have to change dependencies. If Yaml-CPP does not support
this are you proposing to make the change to our third party directory?

On Thu, Jul 13, 2017 at 11:09 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> All,
>
> It appears that the properties in the config.yml are case-sensitive, and
> to add to it, they’re inconsistent. E.g. in the example from the README.md
> file, we have:
>
> …
> Processors:
> - name: GetFile
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 1 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list:
>   Properties:
>   Input Directory: /tmp/getfile
>   Keep Source File: true
> …
>
> The pragmatic course of action would seem to be making the config parser
> case-insensitive, so that users have one less thing to worry about.
>
> If there is no opposition to this, I will enter a Jira ticket to make the
> config parser case-insensitive.
>
> -Andy
>
>

Re: minifi-cpp config.yml case-sensitivity

[Marc]

Hi Andy,
  I believe this makes sense if Yaml-CPP supports the option, but I would
be cautious if we have to change dependencies. If Yaml-CPP does not support
this are you proposing to make the change to our third party directory?

On Thu, Jul 13, 2017 at 11:09 AM, Andy Christianson <
achristian...@hortonworks.com> wrote:

> All,
>
> It appears that the properties in the config.yml are case-sensitive, and
> to add to it, they’re inconsistent. E.g. in the example from the README.md
> file, we have:
>
> …
> Processors:
> - name: GetFile
>   id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
>   class: org.apache.nifi.processors.standard.GetFile
>   max concurrent tasks: 1
>   scheduling strategy: TIMER_DRIVEN
>   scheduling period: 1 sec
>   penalization period: 30 sec
>   yield period: 1 sec
>   run duration nanos: 0
>   auto-terminated relationships list:
>   Properties:
>   Input Directory: /tmp/getfile
>   Keep Source File: true
> …
>
> The pragmatic course of action would seem to be making the config parser
> case-insensitive, so that users have one less thing to worry about.
>
> If there is no opposition to this, I will enter a Jira ticket to make the
> config parser case-insensitive.
>
> -Andy
>
>

Re: minifi-cpp config.yml case-sensitivity

[Kevin Doran]

Good catch, Andy. I agree the preferable course of action here would be to make 
MiNiFi CPP's yaml config parser case-insensitive.

Kevin

On 7/13/17, 11:09, "Andy Christianson"  wrote:

All,

It appears that the properties in the config.yml are case-sensitive, and to 
add to it, they’re inconsistent. E.g. in the example from the README.md file, 
we have:

…
Processors:
- name: GetFile
  id: 471deef6-2a6e-4a7d-912a-81cc17e3a206
  class: org.apache.nifi.processors.standard.GetFile
  max concurrent tasks: 1
  scheduling strategy: TIMER_DRIVEN
  scheduling period: 1 sec
  penalization period: 30 sec
  yield period: 1 sec
  run duration nanos: 0
  auto-terminated relationships list:
  Properties:
  Input Directory: /tmp/getfile
  Keep Source File: true
…

The pragmatic course of action would seem to be making the config parser 
case-insensitive, so that users have one less thing to worry about.

If there is no opposition to this, I will enter a Jira ticket to make the 
config parser case-insensitive.

-Andy