Re: [Puppet Users] Re: Exported resource realized by resource collector, bug?

[Henrik Lindberg]

On 2018-09-14 17:13, Julio Guevara wrote:

/The <| |> operator finds ALL resources (regular, virtual and exported)
that match. It realizes those that are not already realized.

The <<| |>> operator finds ONLY exported resources. /
/
/
If that is the case, maybe documentation is miss-leading since it makes 
a clear distinction between the two types of collectors. 


Yes, it is as it does not say that it works on all kinds of resources 
which allows the reader to think that it is linked to them being 
virtual. Feel free to add a ticket for documentation in Puppet's Jira.


I will see If I 
can come up with another solution for my code but with such limited set 
of logical operators it will be hard. I think It would be easier to just 
make a query with puppetdb_query and then import exported resources that 
match my criteria correctly.


That is what I recommend and what most users end up doing when their use 
case is not super trivial and works with the limitations of the collectors.


Best,
- henrik



Thanks

On Fri, Sep 14, 2018 at 8:07 AM Henrik Lindberg 
mailto:henrik.lindb...@puppet.com>> wrote:


On 2018-09-14 14:47, jcbollinger wrote:
 >
 >
 > On Friday, September 14, 2018 at 3:10:59 AM UTC-5, Johan De Wit
wrote:
 >
 >     Hi,
 >
 >     check this ticket. https://tickets.puppetlabs.com/browse/PUP-6723
 >     
 >
 >
 >     It is hard to explain, but the 'and' a resource collector
does niot
 >     behave like the 'boolean and' as we expect this.
 >
 >
 >
 > I don't follow, Jo.  The issue described here is not about the
search
 > expression, but the fact that an /ordinary/ resource collector
imports
 > /exported/ resources declared by the node.  The presence or
details of a
 > search expression is not relevant -- under no circumstances
should this
 > happen.
 >
 > If this behavior is in fact happening, then I'd guess that Puppet is
 > failing to distinguish between (1) resources exported by the
target node
 > and not imported, and (2) resources exported by any arbitrary
node and
 > imported for the target node.  Alternatively, perhaps Puppet is more
 > simply just failing to distinguish between exported and virtual
 > resources declared by the target node.
 >

The <| |> operator finds ALL resources (regular, virtual and exported)
that match. It realizes those that are not already realized.

The <<| |>> operator finds ONLY exported resources.

The query must thus be written to take that into account.
And THAT is where it becomes important how the != and == operators
actually work.

(I am in no way defending the way it works, but it will be very
difficult to change without difficult to find silent backwards
incompatibilities).

- henrik

 >
 > John
 >
 > --
 > You received this message because you are subscribed to the Google
 > Groups "Puppet Users" group.
 > To unsubscribe from this group and stop receiving emails from it,
send
 > an email to puppet-users+unsubscr...@googlegroups.com

 > >.
 > To view this discussion on the web visit
 >

https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com

 >

.
 > For more options, visit https://groups.google.com/d/optout.


-- 


Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

-- 
You received this message because you are subscribed to a topic in

the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/puppet-users/6lgqC1HmKQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
puppet-users+unsubscr...@googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-users/pngbmn%24lqt%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google 
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to puppet-users+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CADzjYHEJ3VnvKiO8nB6kmDTe4JAaupm7VLyN0ePpb0DNhkJxeQ%40mail.gmail.com 

Re: [Puppet Users] Re: Exported resource realized by resource collector, bug?

[Julio Guevara]

*The <| |> operator finds ALL resources (regular, virtual and
exported) that match. It realizes those that are not already realized.The
<<| |>> operator finds ONLY exported resources. *

If that is the case, maybe documentation is miss-leading since it makes a
clear distinction between the two types of collectors. I will see If I can
come up with another solution for my code but with such limited set of
logical operators it will be hard. I think It would be easier to just make
a query with puppetdb_query and then import exported resources that match
my criteria correctly.

Thanks

On Fri, Sep 14, 2018 at 8:07 AM Henrik Lindberg 
wrote:

> On 2018-09-14 14:47, jcbollinger wrote:
> >
> >
> > On Friday, September 14, 2018 at 3:10:59 AM UTC-5, Johan De Wit wrote:
> >
> > Hi,
> >
> > check this ticket. https://tickets.puppetlabs.com/browse/PUP-6723
> > 
> >
> >
> > It is hard to explain, but the 'and' a resource collector does niot
> > behave like the 'boolean and' as we expect this.
> >
> >
> >
> > I don't follow, Jo.  The issue described here is not about the search
> > expression, but the fact that an /ordinary/ resource collector imports
> > /exported/ resources declared by the node.  The presence or details of a
> > search expression is not relevant -- under no circumstances should this
> > happen.
> >
> > If this behavior is in fact happening, then I'd guess that Puppet is
> > failing to distinguish between (1) resources exported by the target node
> > and not imported, and (2) resources exported by any arbitrary node and
> > imported for the target node.  Alternatively, perhaps Puppet is more
> > simply just failing to distinguish between exported and virtual
> > resources declared by the target node.
> >
>
> The <| |> operator finds ALL resources (regular, virtual and exported)
> that match. It realizes those that are not already realized.
>
> The <<| |>> operator finds ONLY exported resources.
>
> The query must thus be written to take that into account.
> And THAT is where it becomes important how the != and == operators
> actually work.
>
> (I am in no way defending the way it works, but it will be very
> difficult to change without difficult to find silent backwards
> incompatibilities).
>
> - henrik
>
> >
> > John
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Puppet Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to puppet-users+unsubscr...@googlegroups.com
> > .
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com
> > <
> https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com?utm_medium=email&utm_source=footer
> >.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/6lgqC1HmKQk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/pngbmn%24lqt%241%40blaine.gmane.org
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CADzjYHEJ3VnvKiO8nB6kmDTe4JAaupm7VLyN0ePpb0DNhkJxeQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Exported resource realized by resource collector, bug?

[Henrik Lindberg]

On 2018-09-14 14:47, jcbollinger wrote:



On Friday, September 14, 2018 at 3:10:59 AM UTC-5, Johan De Wit wrote:

Hi,

check this ticket. https://tickets.puppetlabs.com/browse/PUP-6723



It is hard to explain, but the 'and' a resource collector does niot
behave like the 'boolean and' as we expect this.



I don't follow, Jo.  The issue described here is not about the search 
expression, but the fact that an /ordinary/ resource collector imports 
/exported/ resources declared by the node.  The presence or details of a 
search expression is not relevant -- under no circumstances should this 
happen.


If this behavior is in fact happening, then I'd guess that Puppet is 
failing to distinguish between (1) resources exported by the target node 
and not imported, and (2) resources exported by any arbitrary node and 
imported for the target node.  Alternatively, perhaps Puppet is more 
simply just failing to distinguish between exported and virtual 
resources declared by the target node.




The <| |> operator finds ALL resources (regular, virtual and exported) 
that match. It realizes those that are not already realized.


The <<| |>> operator finds ONLY exported resources.

The query must thus be written to take that into account.
And THAT is where it becomes important how the != and == operators
actually work.

(I am in no way defending the way it works, but it will be very 
difficult to change without difficult to find silent backwards 
incompatibilities).


- henrik



John

--
You received this message because you are subscribed to the Google 
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to puppet-users+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/pngbmn%24lqt%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Exported resource realized by resource collector, bug?

[jcbollinger]

On Friday, September 14, 2018 at 3:10:59 AM UTC-5, Johan De Wit wrote:
>
> Hi, 
>
> check this ticket.  https://tickets.puppetlabs.com/browse/PUP-6723
>
>
> It is hard to explain, but the 'and' a resource collector does niot behave 
> like the 'boolean and' as we expect this.
>


I don't follow, Jo.  The issue described here is not about the search 
expression, but the fact that an *ordinary* resource collector imports 
*exported* resources declared by the node.  The presence or details of a 
search expression is not relevant -- under no circumstances should this 
happen.

If this behavior is in fact happening, then I'd guess that Puppet is 
failing to distinguish between (1) resources exported by the target node 
and not imported, and (2) resources exported by any arbitrary node and 
imported for the target node.  Alternatively, perhaps Puppet is more simply 
just failing to distinguish between exported and virtual resources declared 
by the target node.


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/a4c43871-cd4e-4e40-a23b-5fa44bfc0dfc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Exported resource realized by resource collector, bug?

[Henrik Lindberg]

On 2018-09-14 10:10, Johan De Wit wrote:

Hi,

check this ticket. https://tickets.puppetlabs.com/browse/PUP-6723


It is hard to explain, but the 'and' a resource collector does niot 
behave like the 'boolean and' as we expect this.




It is not AND that is different; it is the == and != query operators 
that must be understood. The == acts as "in" but != is an *exact* match 
operator just like in puppet.


So, to find those that have neither 'pre' nor 'post' you would want to
write the following, only you cannot because the NOT operator is not 
supported !!!


<| not(tag == 'pre' or tag == 'post') |>  # gives error for not()

The expression:

  tag != 'pre' and tag != 'post'

works only if tag is not an array (which it almost always is)

Consider this:

@notify { 'a': tag => 'blue' }
@notify { 'b': tag => 'green'}
@notify { 'c': tag => ['blue', 'green'] }
@notify { 'd': tag => 'red' }
@notify { 'e': tag => ['red', 'blue'] }

Notify <| tag != 'blue' and tag != 'green' |> { }

It will realize c, d, and e.

c because the array is != 'blue' and is also != 'green'
d because 'red' != 'blue' and 'red != 'green'
e because again, the array is != to either 'blue' nor 'green'

If you think that you can compare using an array, like this:

<| tag != ['blue', 'green'] |>

then you will also be disappointed because that is also not supported
and will give you an error since array values are not supported in the 
query.


The documentation actually points out that the behavior is undefined for 
arrays and hashes. That is its way of saying "it is a mess". How it

actually works (since way back) is what I described above.

You need to find a different way of structuring your logic.
I doubt that virtual/exported resources and features around those will 
receive much love as there has been talks about dropping both virtual 
and exported resources all together (at the moment we have no better 
replacement for the features they enable, so this will not happen over 
night).


Worth noting is also that using tags extensively is not a very robust 
solution since there is no way you can know if your tags are unique or 
not or if you will find elements by chance that are tagged with 
something from part of a name in some scope. If you must use tags,

use tags that are as unique as possible. (More than one user have been
bitten by this).

Sorry for being the bearer of bad news.
Best,

- henrik



Grts

Jo


-Original message-
*From:* jcbollinger 
*Sent:* Thursday 13th September 2018 15:50
*To:* Puppet Users 
*Subject:* [Puppet Users] Re: Exported resource realized by resource
collector, bug?



On Wednesday, September 12, 2018 at 3:20:12 PM UTC-5, Julio Guevara
wrote:

So I'm working with puppet 5.5.1 and I have encountered what I
think is a bug.

So on my site.pp i have the following line for ordering my
firewall rules:

site.pp
-
if $::kernel == 'Linux' {
# Make sure every firewall rule not pre or post is created in
the middle
Firewall <| tag != 'pre' and tag != 'post' |> {
before +> Class['profiles::fw_rules::post'],
require +> Class['profiles::fw_rules::pre'],
}
}
Then on another class I'm doing the following (This is a class
for a postgresql client):
@@firewall { "222 tcp:5432 pgpool from ${::ipaddress}/32" :
action => 'accept',
source => "${::ipaddress}/32",
dport => '5432',
proto => 'tcp',
tag => [ $tag, 'postgresql_client' ],
}
On the postgresql servers I do:
# Grab all firewall rules created for this cluster
Firewall <<| tag == $tag and tag == 'postgresql_client' |>>

I would expect that the rule to only appear on the postgresql
server, but what ends up happening is that the rule is realized
on both the server and the client.

On the documentation for resource collector

(https://puppet.com/docs/puppet/5.5/lang_collectors.html#exported-resource-collectors

):
Collectors realize virtual resources, are used in chaining
statements, and override resource attributes.

on that very same page we have the following for exported
resource collectors:
An exported resource collector uses a modified syntax that
realizes exported resources
 and
imports resources published by other nodes.

Have anybody else found a similar issue?


I do not recall encountering this issue myself, but I concur that I
expect ordinary resource collectors to /not/ collect exported
resources, no matter what node declares them.  Although exported
resources and exported resource collectors are both syntactically
and semantically /anal

RE: [Puppet Users] Re: Exported resource realized by resource collector, bug?

[Johan De Wit]

Hi, 

check this ticket.  https://tickets.puppetlabs.com/browse/PUP-6723



It is hard to explain, but the 'and' a resource collector does niot behave like 
the 'boolean and' as we expect this.



Grts

Jo



-Original message-
From: jcbollinger 
Sent: Thursday 13th September 2018 15:50
To: Puppet Users 
Subject: [Puppet Users] Re: Exported resource realized by resource collector, 
bug?



On Wednesday, September 12, 2018 at 3:20:12 PM UTC-5, Julio Guevara wrote:
So I'm working with puppet 5.5.1 and I have encountered what I think is a bug.

So on my site.pp i have the following line for ordering my firewall rules:

site.pp
-
if $::kernel == 'Linux' {
# Make sure every firewall rule not pre or post is created in the middle
Firewall <| tag != 'pre' and tag != 'post' |> {
before +> Class['profiles::fw_rules::post'],
require +> Class['profiles::fw_rules::pre'],
}
}
Then on another class I'm doing the following (This is a class for a postgresql 
client):
@@firewall { "222 tcp:5432 pgpool from ${::ipaddress}/32" :
action => 'accept',
source => "${::ipaddress}/32",
dport => '5432',
proto => 'tcp',
tag => [ $tag, 'postgresql_client' ],
}
On the postgresql servers I do:
# Grab all firewall rules created for this cluster
Firewall <<| tag == $tag and tag == 'postgresql_client' |>>

I would expect that the rule to only appear on the postgresql server, but what 
ends up happening is that the rule is realized on both the server and the 
client.

On the documentation for resource collector 
(https://puppet.com/docs/puppet/5.5/lang_collectors.html#exported-resource-collectors):
Collectors realize virtual resources, are used in chaining statements, and 
override resource attributes. 

on that very same page we have the following for exported resource collectors:
An exported resource collector uses a modified syntax that realizes exported 
resources and imports resources published by other nodes.

Have anybody else found a similar issue?


I do not recall encountering this issue myself, but I concur that I expect 
ordinary resource collectors to not collect exported resources, no matter what 
node declares them.  Although exported resources and exported resource 
collectors are both syntactically and semantically analogous to virtual 
resources and ordinary resource collectors, they are and have always been 
distinct.  The documentation presents no reason to think otherwise, and in fact 
generally supports that position, though I am unaware of any place where it 
states it in so many words.

Therefore, if indeed it is the case that an ordinary resource collector is 
collecting exported resources, then I concur that that constitutes a bug, and a 
fairly serious one at that.  I suggest you file a bug report.

Do note, however, that in the event that an exported resource is collected for 
a given target node via an exported resource collector, I do expect that for 
all intents and purposes it is then treated as an ordinary resource, and in 
particular, that it ordinary collectors will then affect it with respect to 
parameter overrides and chaining (in the scope of the current catalog building 
job).  The thing that ordinary collectors ought not to do with respect to 
exported resources is import them.


John


-- 
 You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
email to puppet-users+unsubscr...@googlegroups.com 
 .
 To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/0834fac6-15de-4b7f-9ca1-fbce11b13754%40googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.
 

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/zarafa.5b9b6d09.5d47.268d68ba72245991%40zarafa7.open-future.be.
For more options, visit https://groups.google.com/d/optout.