Re: [Puppet Users] Re: Puppet Actions in Parallel?

2017-01-30 Thread Henrik Lindberg 

On 30/01/17 12:58, Michael Shaw wrote:

The very clear use case for me is in the puppet management of
infrastructure.

Take the case of puppet being used to automate the creation of a
database server. The Command to initiate the creation takes around 2
seconds to complete.  The process to actually create the database can
take 5 minutes.  This has a problem when you want to create a suitable
provider.

You can complete the creation immediately after the creation command
completes.  nice and quick.  However you cannot have anything else in
your puppet manifest that depends on this database server actually
existing and being accessible.

Or you could block on the creation, polling every 5 seconds to confirm
the current state of the creation task.  This is robust, and ensures
that when the creation task completes, the database is available for
other puppet actions.  This is great if you are only managing 1 database
server.  What about if there are 5 or 6? the creation of these happens
sequentially, creating a very long running script.

The best outcome would be to have a way to write  a provider to say that
all instances of this type are independent, and we can run X in parallel.

Kind regards,

Michael



We are very much aware of use cases like this where the parallelism is 
mostly waiting from puppet's perspective and waiting is by far much 
faster to do in parallel :-)


It is however very difficult to implement given the current way the 
types and providers framework is implemented. To start with, it is 
complex to make different providers talk to each other and communicate 
about a dynamic (evolving) intermediate state. To make it work you would 
need to model the different steps in the workflow as separate resources 
thus enabling waiting on an earlier step to finish (join point) if you 
need that.
You could also implement the provider as an external service that does 
the actual work and you would have providers in puppet talk to this 
asynchronous service (a *lot* of work).


A complicating factor is that Puppet is inherently single-threaded. 
There are many constructs that are not safe for multithreading.


I would very much like us to have a types & providers framework that can 
do this but it requires a major new take and a new implementation.


- henrik


On Tuesday, February 18, 2014 at 5:02:11 PM UTC, Jon Forrest wrote:

On Mon, Feb 17, 2014 at 6:20 AM, jcbollinger > wrote:

> Well, I think the question that killed this thread the first time
boils down
> to "would it really?".  The speculation at the time was that parallel
> execution would produce disappointing wall-time gains, based on the
> assertion that the catalog application process is largely I/O
bound.  There
> were also some assertions that Ruby doesn't do shared-memory
parallelism
> very well.  Nobody reported any actual analysis of any of that,
though.

Right. Without such analysis it's hard to know if this idea is worth
following
up on today.

But, one thing to keep in mind is that systems are always changing.
An I/O bound
system of today might not be I/O bound tomorrow as technological
improvements
appear. Having a computer with available resources unable to apply
to resources
to a Puppet run (or anything else) is wasteful. In time, the lack of
client parallelization
could be a competitive weakness as Puppet competes in the marketplace.
(I don't know
what the status of client parallelization is in the competition
right now).

Jon Forrest

--
You received this message because you are subscribed to the Google
Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-users+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/951c8a89-662d-4da0-9154-3c6ee4cf%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/o6nrq5%24ia6%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Puppet Actions in Parallel?

2017-01-30 Thread Michael Shaw 
The very clear use case for me is in the puppet management of 
infrastructure.

Take the case of puppet being used to automate the creation of a database 
server. The Command to initiate the creation takes around 2 seconds to 
complete.  The process to actually create the database can take 5 minutes. 
 This has a problem when you want to create a suitable provider.

You can complete the creation immediately after the creation command 
completes.  nice and quick.  However you cannot have anything else in your 
puppet manifest that depends on this database server actually existing and 
being accessible.

Or you could block on the creation, polling every 5 seconds to confirm the 
current state of the creation task.  This is robust, and ensures that when 
the creation task completes, the database is available for other puppet 
actions.  This is great if you are only managing 1 database server.  What 
about if there are 5 or 6? the creation of these happens sequentially, 
creating a very long running script.

The best outcome would be to have a way to write  a provider to say that 
all instances of this type are independent, and we can run X in parallel.

Kind regards,

Michael

On Tuesday, February 18, 2014 at 5:02:11 PM UTC, Jon Forrest wrote:
>
> On Mon, Feb 17, 2014 at 6:20 AM, jcbollinger  > wrote: 
>
> > Well, I think the question that killed this thread the first time boils 
> down 
> > to "would it really?".  The speculation at the time was that parallel 
> > execution would produce disappointing wall-time gains, based on the 
> > assertion that the catalog application process is largely I/O bound. 
>  There 
> > were also some assertions that Ruby doesn't do shared-memory parallelism 
> > very well.  Nobody reported any actual analysis of any of that, though. 
>
> Right. Without such analysis it's hard to know if this idea is worth 
> following 
> up on today. 
>
> But, one thing to keep in mind is that systems are always changing. An I/O 
> bound 
> system of today might not be I/O bound tomorrow as technological 
> improvements 
> appear. Having a computer with available resources unable to apply to 
> resources 
> to a Puppet run (or anything else) is wasteful. In time, the lack of 
> client parallelization 
> could be a competitive weakness as Puppet competes in the marketplace. 
> (I don't know 
> what the status of client parallelization is in the competition right 
> now). 
>
> Jon Forrest 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/951c8a89-662d-4da0-9154-3c6ee4cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet Users] Re: Puppet Actions in Parallel?

2014-02-18 Thread Jon Forrest 
On Mon, Feb 17, 2014 at 6:20 AM, jcbollinger  wrote:

> Well, I think the question that killed this thread the first time boils down
> to "would it really?".  The speculation at the time was that parallel
> execution would produce disappointing wall-time gains, based on the
> assertion that the catalog application process is largely I/O bound.  There
> were also some assertions that Ruby doesn't do shared-memory parallelism
> very well.  Nobody reported any actual analysis of any of that, though.

Right. Without such analysis it's hard to know if this idea is worth following
up on today.

But, one thing to keep in mind is that systems are always changing. An I/O bound
system of today might not be I/O bound tomorrow as technological improvements
appear. Having a computer with available resources unable to apply to resources
to a Puppet run (or anything else) is wasteful. In time, the lack of
client parallelization
could be a competitive weakness as Puppet competes in the marketplace.
(I don't know
what the status of client parallelization is in the competition right now).

Jon Forrest

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAF0Z9ipsPHh%2BZOC_VJM2x_74ToCD-u-imyk4Y9bXyi4wxMhYtg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Puppet Actions in Parallel?

2014-02-17 Thread Deepak Giridharagopal 
On Sat, Feb 15, 2014 at 9:00 AM, Trevor Vaughan wrote:

> Wow, that's quite some dead thread resurrection!
>
> I remember this being discussed in the past and part of the issue is that
> so many parts of what Puppet is doing are I/O bound so I'm not really sure
> what the parallelism would gain you outside of destroying your I/O channels.
>
> That said, I think it would be nice to have supported for those that do
> have systems that can take advantage of it. We may even see that a -j2
> would give just enough balance between I/O destruction and system
> acceleration.
>

There was a recent, pretty in-depth discussion in puppet-dev about a
related concept: batched application of resources:

https://groups.google.com/forum/#!msg/puppet-dev/X7RgakTGnbk/19RgHTMuLZUJ


>
> Trevor
>
>
>
>
>
>
> On Fri, Feb 14, 2014 at 11:52 PM, Andrew Pennebaker <
> andrew.penneba...@gmail.com> wrote:
>
>> This is a fantastic idea! Any progress on this?
>>
>>
>> On Sunday, March 4, 2012 10:53:23 PM UTC-5, Jon Forrest wrote:
>>>
>>> As many learning Puppet for the first time, the fact that the
>>> order of actions is undefined unless specific metaparameters
>>> like 'require' are used. Fine.
>>>
>>> This got me to thinking. The GNU make program has the "-j"
>>> option, which allows make to start more than one action
>>> in parallel if the actions are at the same dependency level.
>>> I've used this option on a 48-core machine to great benefit.
>>>
>>> So, why can't there be a similar option in the puppet agent?
>>> I can easily imagine how this could substantially reduce the
>>> length of time for a puppet run.
>>>
>>> (The make "-j" option allows an optional numeric value, which, if
>>> given, is the maximum number of actions that can be run
>>> in parallel. If no numeric is given, then there's no limit
>>> to the number of parallel actions).
>>>
>>> I did a quick review of the Puppet manual but I didn't see
>>> anything like this. Am I missing something? Is this a good
>>> idea?
>>>
>>> Cordially,
>>> Jon Forrest
>>>
>>>  --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to puppet-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/b836c960-09bc-48f5-8b5f-407d42233b10%40googlegroups.com
>> .
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
>
> --
> Trevor Vaughan
> Vice President, Onyx Point, Inc
> (410) 541-6699
> tvaug...@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CANs%2BFoV5iSNbuS%2BnxEzmU7mhQXzGtwf5p4-e4M3P1K_b_P_Fbw%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CAOjOXY2Rv1i9s5fDtXf-w9o0%3DHtYzkbMkGc-1E0uSVGcZBFj_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Puppet Actions in Parallel?

2014-02-17 Thread jcbollinger 


On Sunday, February 16, 2014 10:43:53 AM UTC-6, Trevor Vaughan wrote:
>
>
> [...] this may not happen but it would still be nice to have.
>
>
>

Well, I think the question that killed this thread the first time boils 
down to "would it really?".  The speculation at the time was that parallel 
execution would produce disappointing wall-time gains, based on the 
assertion that the catalog application process is largely I/O bound.  There 
were also some assertions that Ruby doesn't do shared-memory parallelism 
very well.  Nobody reported any actual analysis of any of that, though.

Whatever benefit there might be needs to be weighed against the costs, 
which include not just the direct costs of developing the feature, but also 
the ongoing costs of added code complexity and increased maintenance burden.

Were I PL, I would be very hesitant to devote resources to such a 
speculative project as I think this would be.  Were I a user interested in 
such a feature and having time available, I might consider having a go the 
project myself.  Working code trumps predictive analysis every time.


John

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/6bb31477-57ef-4035-9c0c-ffc0b6d757d5%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Puppet Actions in Parallel?

2014-02-16 Thread Trevor Vaughan 
It seems like that should be a relatively easy issue to overcome
conceptually anyway.

* Add a metaparamater 'parallel', that defaults to 'true' or 'false'
depending on the specific native type being managed.
** For instance, YUM and DEB packages (probably most packages) would be
mandatory 'false' and non-overridable.
** File types would default to 'true' and be overridable.

In this way, explicit ordering would always work and items that don't need
explicit ordering could be done in parallel.

This partially comes back to some of the subgraph discussions that we had
back in 2010. Once you boil your tree down to a bunch of independent
subgraphs, you can execute all of them in parallel so long as all of your
resource ordering is correct and internal consistency is met.

In the Red Hat family case: yumrepo -> yum (yum serial) -> explicit
dependencies.

Could it cause issues? Yes! But that's why these types of things are off by
default. For those of us that have the power and want the capability, let's
get the gains, for those that need a more risk-averse environment, don't
enable it.

I fear that, with the move toward making everything more procedural by
default, this may not happen but it would still be nice to have.

Trevor


On Sat, Feb 15, 2014 at 12:35 PM, Jon Forrest  wrote:

>
>
> On Friday, February 14, 2014 8:52:59 PM UTC-8, Andrew Pennebaker wrote:
>>
>> This is a fantastic idea! Any progress on this?
>>
>
> I'm the original poster. It's been a while since I posted this.
> iIt didn't generate nearly as much interest as
> I was hoping. There was some concern that even though
> various actions were at the same level in the dependency graph,
> that they would have side effects that required serial execution.
> One example of this is that there can only be one apt-get install at
> a time. There are probably others.
>
> So, it wouldn't be easy for Puppet to know which actions
> it should do safely in parallel.
>
> If any other the experts have anything to add I'd love
> to hear it.
>
> Jon Forrest
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/86d46ce2-a714-48c1-9954-b5c9b483b14a%40googlegroups.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaug...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CANs%2BFoVpO64yw4W5-dEQC_h5rWspDMtqLYZEWwji-mCM0y9%3DmA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: Puppet Actions in Parallel?

2014-02-15 Thread Jon Forrest 


On Friday, February 14, 2014 8:52:59 PM UTC-8, Andrew Pennebaker wrote:
>
> This is a fantastic idea! Any progress on this?
>

I'm the original poster. It's been a while since I posted this.
iIt didn't generate nearly as much interest as
I was hoping. There was some concern that even though
various actions were at the same level in the dependency graph,
that they would have side effects that required serial execution.
One example of this is that there can only be one apt-get install at
a time. There are probably others.

So, it wouldn't be easy for Puppet to know which actions
it should do safely in parallel.

If any other the experts have anything to add I'd love
to hear it.

Jon Forrest
 

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/86d46ce2-a714-48c1-9954-b5c9b483b14a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [Puppet Users] Re: Puppet Actions in Parallel?

2014-02-15 Thread Trevor Vaughan 
Wow, that's quite some dead thread resurrection!

I remember this being discussed in the past and part of the issue is that
so many parts of what Puppet is doing are I/O bound so I'm not really sure
what the parallelism would gain you outside of destroying your I/O channels.

That said, I think it would be nice to have supported for those that do
have systems that can take advantage of it. We may even see that a -j2
would give just enough balance between I/O destruction and system
acceleration.

Trevor






On Fri, Feb 14, 2014 at 11:52 PM, Andrew Pennebaker <
andrew.penneba...@gmail.com> wrote:

> This is a fantastic idea! Any progress on this?
>
>
> On Sunday, March 4, 2012 10:53:23 PM UTC-5, Jon Forrest wrote:
>>
>> As many learning Puppet for the first time, the fact that the
>> order of actions is undefined unless specific metaparameters
>> like 'require' are used. Fine.
>>
>> This got me to thinking. The GNU make program has the "-j"
>> option, which allows make to start more than one action
>> in parallel if the actions are at the same dependency level.
>> I've used this option on a 48-core machine to great benefit.
>>
>> So, why can't there be a similar option in the puppet agent?
>> I can easily imagine how this could substantially reduce the
>> length of time for a puppet run.
>>
>> (The make "-j" option allows an optional numeric value, which, if
>> given, is the maximum number of actions that can be run
>> in parallel. If no numeric is given, then there's no limit
>> to the number of parallel actions).
>>
>> I did a quick review of the Puppet manual but I didn't see
>> anything like this. Am I missing something? Is this a good
>> idea?
>>
>> Cordially,
>> Jon Forrest
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/b836c960-09bc-48f5-8b5f-407d42233b10%40googlegroups.com
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>



-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaug...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/CANs%2BFoV5iSNbuS%2BnxEzmU7mhQXzGtwf5p4-e4M3P1K_b_P_Fbw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Puppet Users] Re: Puppet Actions in Parallel?

2014-02-14 Thread Andrew Pennebaker 
This is a fantastic idea! Any progress on this?

On Sunday, March 4, 2012 10:53:23 PM UTC-5, Jon Forrest wrote:
>
> As many learning Puppet for the first time, the fact that the
> order of actions is undefined unless specific metaparameters
> like 'require' are used. Fine.
>
> This got me to thinking. The GNU make program has the "-j"
> option, which allows make to start more than one action
> in parallel if the actions are at the same dependency level.
> I've used this option on a 48-core machine to great benefit.
>
> So, why can't there be a similar option in the puppet agent?
> I can easily imagine how this could substantially reduce the
> length of time for a puppet run.
>
> (The make "-j" option allows an optional numeric value, which, if
> given, is the maximum number of actions that can be run
> in parallel. If no numeric is given, then there's no limit
> to the number of parallel actions).
>
> I did a quick review of the Puppet manual but I didn't see
> anything like this. Am I missing something? Is this a good
> idea?
>
> Cordially,
> Jon Forrest
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-users/b836c960-09bc-48f5-8b5f-407d42233b10%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.