Re: [Puppet Users] File Size

[Trevor Vaughan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hmm...perhaps both modes could be supported.

Have one way to schedule and another to run inline.

That would be quite useful.

Trevor

On 04/08/2010 01:45 PM, Thomas Bellman wrote:
> Dan Bode wrote:
> 
>> I would prefer if puppet ran the sync. It would be nice to receive
>> puppet events for any changes made via rsync (essentially reports of
>> which files change, this would require that it is implemented in ruby).
>>
>> I can see from reading the man page that there is a --dryrun call that
>> could be used to determine rather rsync should be run or not. Is this
>> reasonable to run this to determine if Puppet should run? or is that
>> too slow?
> 
> Problem is, once you get file trees that have several tens of thousands
> of files in them, just traversing the tree to see which files are there
> and ought to be transfered can take a while.  When the target tree is
> already up to date, rsync --dry-run doesn't go any faster than without
> --dry-run.
> 
> The time taken doesn't matter much when Puppet is doing its automatic,
> unattended runs, but when you have made a change to your manifests and
> want to make a manual test run from an interactive shell, you don't want
> to wait an extra ten or fifty seconds just to see that you misspelled a
> package name...
> 
> 
> /Bellman
> 

- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaug...@onyxpoint.com
 phone: 410-541-ONYX (6699)
 pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAku+dQMACgkQyWMIJmxwHpRULQCeMWxZcFB2/EwM+m+LxLGwULSl
LVkAoLTU8x1F9YNKf+bz6uS0BgOWhCGx
=RB47
-END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

<>

Re: [Puppet Users] File Size

[Thomas Bellman]

Patrick wrote:


On Apr 8, 2010, at 2:15 AM, Thomas Bellman wrote:



This does two things: when run the first time, it actually does
an rsync of a directory tree.  And then it creates a cron job
for keeping the tree synchronized with the source.  You use the
'creates' parameter to indicate a file that you expect to exist
in the tree, so Puppet itself doesn't have to run rsync every
time.  If you mirror a large tree that could take a long time,
even if no files actually need to be transferred (I use it to
keep a mirror of CentOS and EPEL on a cluster, so we can re-install
our nodes anytime without being dependant on external servers).



It sounds like we have very different ideas how this should work.
For me, an ideal module would use the the schedule argument, and
not have anything to do with cron.


Luckily, what I have implemented actually supports that mode of
operation as well. :-)  Just don't specify the 'hour' and 'minute'
parameters, and don't use the 'creates' parameter.

Using the schedule metaparameter might not be what you really want.
If you do something like:

rsync_mirror {
mymirror: ..., schedule => nightly;
}

sometype {
foo: ..., require => Rsync_mirror[mymirror];
}

where Sometype[foo] needs some files from the rsync, if you run
your manifests on a fresh machine, and it isn't in the middle of
the night, then Sometype[foo] will fail.



I wouldn't use use any of these tags, but here are more ideas while
you are asking:

require
depends
subscribe
notify
onlyif
unless
refreshonly


Luke implemented 'require' and 'subscribe' for me long before I
started using Puppet, so I got that for free...  But 'subscribe'
probably doesn't make much sense without 'refreshonly'; should
be easy to implement, so I might do that.  I don't think it is
something I would use, though.

I suppose you would want 'notify' to send a notification when there
has been some files actually changed by the rsync.  That would only
make sense when you don't use cron to offload the work from Puppet.
But I don't think that can be done from a define with exec, so it
would need to be implemented in Ruby.  If someone can come up with
a good use case I'll think about it.

'Onlyif' and 'unless' is already implemented.

I don't know what 'depends' would do.  Explanation?


/Bellman

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Nigel Kersten]

On Thu, Apr 8, 2010 at 10:50 AM, Paul Lathrop wrote:

> On Thu, Apr 8, 2010 at 10:45 AM, Thomas Bellman 
> wrote:
> > Dan Bode wrote:
> >
> >> I would prefer if puppet ran the sync. It would be nice to receive
> puppet
> >> events for any changes made via rsync (essentially reports of which
> files
> >> change, this would require that it is implemented in ruby).
> >>
> >> I can see from reading the man page that there is a --dryrun call that
> >> could be used to determine rather rsync should be run or not. Is this
> >> reasonable to run this to determine if Puppet should run? or is that too
> >> slow?
> >
> > Problem is, once you get file trees that have several tens of thousands
> > of files in them, just traversing the tree to see which files are there
> > and ought to be transfered can take a while.  When the target tree is
> > already up to date, rsync --dry-run doesn't go any faster than without
> > --dry-run.
> >
> > The time taken doesn't matter much when Puppet is doing its automatic,
> > unattended runs, but when you have made a change to your manifests and
> > want to make a manual test run from an interactive shell, you don't want
> > to wait an extra ten or fifty seconds just to see that you misspelled a
> > package name...
>
> True, but this is one place where --tags really shines. Our manifest
> have reached the level of complexity where we *need* to use --tags for
> iterative debugging.
>

So I rarely use tags for iterative debugging, simply because I dislike the
behavior when Resource A is tagged "foo" and requires Resource B which is
not tagged "foo"

I'd use them a lot more if we could have an optional flag for automatically
pulling in requires that aren't tagged with the specified tag.

I guess I should feature request that.

Anyhow, for iterative debugging I instead have development servers set up
that allow easy patching of a pending changelist into their config, and this
can include switching off large chunks of the config.




>
> --Paul
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>


-- 
nigel

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Paul Lathrop]

On Thu, Apr 8, 2010 at 10:45 AM, Thomas Bellman  wrote:
> Dan Bode wrote:
>
>> I would prefer if puppet ran the sync. It would be nice to receive puppet
>> events for any changes made via rsync (essentially reports of which files
>> change, this would require that it is implemented in ruby).
>>
>> I can see from reading the man page that there is a --dryrun call that
>> could be used to determine rather rsync should be run or not. Is this
>> reasonable to run this to determine if Puppet should run? or is that too
>> slow?
>
> Problem is, once you get file trees that have several tens of thousands
> of files in them, just traversing the tree to see which files are there
> and ought to be transfered can take a while.  When the target tree is
> already up to date, rsync --dry-run doesn't go any faster than without
> --dry-run.
>
> The time taken doesn't matter much when Puppet is doing its automatic,
> unattended runs, but when you have made a change to your manifests and
> want to make a manual test run from an interactive shell, you don't want
> to wait an extra ten or fifty seconds just to see that you misspelled a
> package name...

True, but this is one place where --tags really shines. Our manifest
have reached the level of complexity where we *need* to use --tags for
iterative debugging.

--Paul

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Thomas Bellman]

Dan Bode wrote:

I would prefer if puppet ran the sync. It would be nice to receive 
puppet events for any changes made via rsync (essentially reports of 
which files change, this would require that it is implemented in ruby).


I can see from reading the man page that there is a --dryrun call that 
could be used to determine rather rsync should be run or not. Is this 
reasonable to run this to determine if Puppet should run? or is that too 
slow?


Problem is, once you get file trees that have several tens of thousands
of files in them, just traversing the tree to see which files are there
and ought to be transfered can take a while.  When the target tree is
already up to date, rsync --dry-run doesn't go any faster than without
--dry-run.

The time taken doesn't matter much when Puppet is doing its automatic,
unattended runs, but when you have made a change to your manifests and
want to make a manual test run from an interactive shell, you don't want
to wait an extra ten or fifty seconds just to see that you misspelled a
package name...


/Bellman

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Patrick]

On Apr 8, 2010, at 2:15 AM, Thomas Bellman wrote:

> Trevor Vaughan wrote:
> 
>> Mainly sugar around the call with the ability to twiddle all of the
>> useful rsync flags in a platform-agnostic manner where possible.
>> That would be truly awesome.
> 
> I have such a define in my module "nsc-puppet-utils" (available at
> http://www.nsc.liu.se/~bellman/nsc-puppet-utils.git).  You use it
> like this:
> 
>rsync_mirror {
>epel-5-x86_64:
>source => "rsync://mirrorhost/fedora-epel/5/x86_64/./",
>target => "/pkg/epel-5.x86_64",
>rsyncflags => ['--exclude', 'repoview', '--exclude', 'debug'],
>creates => "/pkg/epel-5.x86_64/repodata/repomd.xml",
>hour => 3, minute => 25;
>}
> 
> This does two things: when run the first time, it actually does
> an rsync of a directory tree.  And then it creates a cron job
> for keeping the tree synchronized with the source.  You use the
> 'creates' parameter to indicate a file that you expect to exist
> in the tree, so Puppet itself doesn't have to run rsync every
> time.  If you mirror a large tree that could take a long time,
> even if no files actually need to be transferred (I use it to
> keep a mirror of CentOS and EPEL on a cluster, so we can re-install
> our nodes anytime without being dependant on external servers).
> 
> Is this what you want?  Any suggestions for improvements?
> 

It sounds like we have very different ideas how this should work.
For me, an ideal module would use the the schedule argument, and not have 
anything to do with cron.


I wouldn't use use any of these tags, but here are more ideas while you are 
asking:
require
depends
subscribe
notify
onlyif
unless
refreshonly

-Patrick

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Trevor Vaughan]

That's definitely reasonable and probably worthwhile.

And having this be an internal puppet type that could parse the output
of rsync and report errors, etc... would be ideal.

Trevor

On Thu, Apr 8, 2010 at 6:22 AM, Dan Bode  wrote:
>
>
> On Thu, Apr 8, 2010 at 2:15 AM, Thomas Bellman  wrote:
>>
>> Trevor Vaughan wrote:
>>
>>> Mainly sugar around the call with the ability to twiddle all of the
>>> useful rsync flags in a platform-agnostic manner where possible.
>>>
>>> That would be truly awesome.
>>
>> I have such a define in my module "nsc-puppet-utils" (available at
>> http://www.nsc.liu.se/~bellman/nsc-puppet-utils.git).  You use it
>> like this:
>>
>>    rsync_mirror {
>>        epel-5-x86_64:
>>            source => "rsync://mirrorhost/fedora-epel/5/x86_64/./",
>>            target => "/pkg/epel-5.x86_64",
>>            rsyncflags => ['--exclude', 'repoview', '--exclude', 'debug'],
>>            creates => "/pkg/epel-5.x86_64/repodata/repomd.xml",
>>            hour => 3, minute => 25;
>>    }
>>
>> This does two things: when run the first time, it actually does
>> an rsync of a directory tree.  And then it creates a cron job
>> for keeping the tree synchronized with the source.  You use the
>> 'creates' parameter to indicate a file that you expect to exist
>> in the tree, so Puppet itself doesn't have to run rsync every
>> time.  If you mirror a large tree that could take a long time,
>> even if no files actually need to be transferred (I use it to
>> keep a mirror of CentOS and EPEL on a cluster, so we can re-install
>> our nodes anytime without being dependant on external servers).
>>
>> Is this what you want?  Any suggestions for improvements?
>
> I would prefer if puppet ran the sync. It would be nice to receive puppet
> events for any changes made via rsync (essentially reports of which files
> change, this would require that it is implemented in ruby).
>
> I can see from reading the man page that there is a --dryrun call that could
> be used to determine rather rsync should be run or not. Is this reasonable
> to run this to determine if Puppet should run? or is that too slow?
>
>>
>>        /Bellman
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To post to this group, send email to puppet-us...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> puppet-users+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/puppet-users?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>



-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaug...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Dan Bode]

On Thu, Apr 8, 2010 at 2:15 AM, Thomas Bellman  wrote:

> Trevor Vaughan wrote:
>
>  Mainly sugar around the call with the ability to twiddle all of the
>> useful rsync flags in a platform-agnostic manner where possible.
>>
>> That would be truly awesome.
>>
>
> I have such a define in my module "nsc-puppet-utils" (available at
> http://www.nsc.liu.se/~bellman/nsc-puppet-utils.git).
>  You use it
> like this:
>
>rsync_mirror {
>epel-5-x86_64:
>source => "rsync://mirrorhost/fedora-epel/5/x86_64/./",
>target => "/pkg/epel-5.x86_64",
>rsyncflags => ['--exclude', 'repoview', '--exclude', 'debug'],
>creates => "/pkg/epel-5.x86_64/repodata/repomd.xml",
>hour => 3, minute => 25;
>}
>
> This does two things: when run the first time, it actually does
> an rsync of a directory tree.  And then it creates a cron job
> for keeping the tree synchronized with the source.  You use the
> 'creates' parameter to indicate a file that you expect to exist
> in the tree, so Puppet itself doesn't have to run rsync every
> time.  If you mirror a large tree that could take a long time,
> even if no files actually need to be transferred (I use it to
> keep a mirror of CentOS and EPEL on a cluster, so we can re-install
> our nodes anytime without being dependant on external servers).
>
> Is this what you want?  Any suggestions for improvements?
>

I would prefer if puppet ran the sync. It would be nice to receive puppet
events for any changes made via rsync (essentially reports of which files
change, this would require that it is implemented in ruby).

I can see from reading the man page that there is a --dryrun call that could
be used to determine rather rsync should be run or not. Is this reasonable
to run this to determine if Puppet should run? or is that too slow?


>
>/Bellman
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-us...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Thomas Bellman]

Trevor Vaughan wrote:


Mainly sugar around the call with the ability to twiddle all of the
useful rsync flags in a platform-agnostic manner where possible.

That would be truly awesome.


I have such a define in my module "nsc-puppet-utils" (available at
http://www.nsc.liu.se/~bellman/nsc-puppet-utils.git).  You use it
like this:

rsync_mirror {
epel-5-x86_64:
source => "rsync://mirrorhost/fedora-epel/5/x86_64/./",
target => "/pkg/epel-5.x86_64",
rsyncflags => ['--exclude', 'repoview', '--exclude', 'debug'],
creates => "/pkg/epel-5.x86_64/repodata/repomd.xml",
hour => 3, minute => 25;
}

This does two things: when run the first time, it actually does
an rsync of a directory tree.  And then it creates a cron job
for keeping the tree synchronized with the source.  You use the
'creates' parameter to indicate a file that you expect to exist
in the tree, so Puppet itself doesn't have to run rsync every
time.  If you mirror a large tree that could take a long time,
even if no files actually need to be transferred (I use it to
keep a mirror of CentOS and EPEL on a cluster, so we can re-install
our nodes anytime without being dependant on external servers).

Is this what you want?  Any suggestions for improvements?


/Bellman

--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Trevor Vaughan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I'd +1000 this.

Mainly sugar around the call with the ability to twiddle all of the
useful rsync flags in a platform-agnostic manner where possible.

That would be truly awesome.

Also, the ability to natively wrap this in SSL with something like
stunnel would be even better. At a worst case, bookending it with
webrick might work somehow.

Trevor

On 04/07/2010 08:28 PM, Michael DeHaan wrote:
> On Wed, Apr 7, 2010 at 8:20 PM, Christopher Johnston  
> wrote:
>> Nfs is not always feasible in controlled environments.  I run kernels with
>> the nfs stack completely removed to cut out kernel bloat (for size).
>>
> 
> And it can be slow and annoying, yes :)
> 
>> Rsync integration into puppet directly would be attractive and very useful.
>>
> 
> What would you see this doing beyond an Exec call, or would it just be
> sugar around that call?
> I'm wondering what the "don't execute if" conditions might be, in
> other words to avoid hitting the server, or if that's meaningful.
> 

- -- 
Trevor Vaughan
 Vice President, Onyx Point, Inc.
 email: tvaug...@onyxpoint.com
 phone: 410-541-ONYX (6699)
 pgp: 0x6C701E94

- -- This account not approved for unencrypted sensitive information --
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAku9QvkACgkQyWMIJmxwHpSv2wCgoO/Rg6vugmMyVLG1iRf0BzyE
mZcAni8me2DbU0nHxRNFiiCJHFOlNdz9
=VGGe
-END PGP SIGNATURE-

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

<>

Re: [Puppet Users] File Size

[Michael DeHaan]

On Wed, Apr 7, 2010 at 8:20 PM, Christopher Johnston  wrote:
> Nfs is not always feasible in controlled environments.  I run kernels with
> the nfs stack completely removed to cut out kernel bloat (for size).
>

And it can be slow and annoying, yes :)

> Rsync integration into puppet directly would be attractive and very useful.
>

What would you see this doing beyond an Exec call, or would it just be
sugar around that call?
I'm wondering what the "don't execute if" conditions might be, in
other words to avoid hitting the server, or if that's meaningful.

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Christopher Johnston]

Nfs is not always feasible in controlled environments.  I run kernels  
with the nfs stack completely removed to cut out kernel bloat (for  
size).


Rsync integration into puppet directly would be attractive and very  
useful.


Sent from my iPhone

On Apr 7, 2010, at 8:07 PM, Michael DeHaan   
wrote:



On Wed, Apr 7, 2010 at 1:38 PM, Patrick  wrote:
I second this.  Puppet will load the whole file into ram, and  
puppet never deallocates memory.  It's almost always better to move  
big files by putting them into a package or using an "Exec" type  
with "creates."





Just to be clear, the deallocation beyond a threshold is largely a
present-version-of Ruby limitation.   That isn't to say there aren't
some other things we can do to make fileserving better -- such as the
streaming improvements in the next release (this should help a fair
amount!)

As I just mentioned one of them on the list, there are a couple
alternatives to fileserving you can look at now if you want to
transfer content.

One (not so suitable for binary content) is something like
http://github.com/reductivelabs/puppet-vcsrepo using source control.

Another recommended approach, and really it's the right thing to do in
many cases, are read only NFS mounts with copies sourcing of those
locations.   You could also, if you really wanted, use an Exec+rsync,
though I'd go the NFS (or samba, etc) approach first.

Fileserving is definitely something you'd continue to want to do with
templates and such, but not so much for app deployment.

Moving forward, I think you'll see more support and features around
alternative ways to deploy files, such as vcsrepo.   If there's
another use case around this that I'm missing, where NFS or source
control won't work, let me know.

--Michael


On Apr 7, 2010, at 10:21 AM, Daniel Kerwin wrote:


Not sure about a limit but puppet isn't very good at transfering
really big files. This may lead to memory problems afaik



--
You received this message because you are subscribed to the Google  
Groups "Puppet Users" group.

To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users+unsubscr...@googlegroups.com 
.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en 
.




--
You received this message because you are subscribed to the Google Groups "Puppet 
Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Michael DeHaan]

On Wed, Apr 7, 2010 at 1:38 PM, Patrick  wrote:
> I second this.  Puppet will load the whole file into ram, and puppet never 
> deallocates memory.  It's almost always better to move big files by putting 
> them into a package or using an "Exec" type with "creates."
>
>

Just to be clear, the deallocation beyond a threshold is largely a
present-version-of Ruby limitation.   That isn't to say there aren't
some other things we can do to make fileserving better -- such as the
streaming improvements in the next release (this should help a fair
amount!)

As I just mentioned one of them on the list, there are a couple
alternatives to fileserving you can look at now if you want to
transfer content.

One (not so suitable for binary content) is something like
http://github.com/reductivelabs/puppet-vcsrepo using source control.

Another recommended approach, and really it's the right thing to do in
many cases, are read only NFS mounts with copies sourcing of those
locations.   You could also, if you really wanted, use an Exec+rsync,
though I'd go the NFS (or samba, etc) approach first.

Fileserving is definitely something you'd continue to want to do with
templates and such, but not so much for app deployment.

Moving forward, I think you'll see more support and features around
alternative ways to deploy files, such as vcsrepo.   If there's
another use case around this that I'm missing, where NFS or source
control won't work, let me know.

--Michael

> On Apr 7, 2010, at 10:21 AM, Daniel Kerwin wrote:
>
>> Not sure about a limit but puppet isn't very good at transfering
>> really big files. This may lead to memory problems afaik
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Patrick]

I second this.  Puppet will load the whole file into ram, and puppet never 
deallocates memory.  It's almost always better to move big files by putting 
them into a package or using an "Exec" type with "creates."


On Apr 7, 2010, at 10:21 AM, Daniel Kerwin wrote:

> Not sure about a limit but puppet isn't very good at transfering
> really big files. This may lead to memory problems afaik
> 
> On Wed, Apr 7, 2010 at 6:56 PM, Seeker  wrote:
>> Hi all, Just wondering, is there a limit on the file size that you can
>> transfer with PUPPET.
>> 
>> Thank you all
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "Puppet Users" group.
>> To post to this group, send email to puppet-us...@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> puppet-users+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/puppet-users?hl=en.
>> 
>> 
> 
> 
> 
> -- 
> 
> Cheers,
> 
> Daniel
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To post to this group, send email to puppet-us...@googlegroups.com.
> To unsubscribe from this group, send email to 
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/puppet-users?hl=en.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.

Re: [Puppet Users] File Size

[Daniel Kerwin]

Not sure about a limit but puppet isn't very good at transfering
really big files. This may lead to memory problems afaik

On Wed, Apr 7, 2010 at 6:56 PM, Seeker  wrote:
> Hi all, Just wondering, is there a limit on the file size that you can
> transfer with PUPPET.
>
> Thank you all
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Puppet Users" group.
> To post to this group, send email to puppet-us...@googlegroups.com.
> To unsubscribe from this group, send email to 
> puppet-users+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/puppet-users?hl=en.
>
>



-- 

Cheers,

Daniel

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Users" group.
To post to this group, send email to puppet-us...@googlegroups.com.
To unsubscribe from this group, send email to 
puppet-users+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/puppet-users?hl=en.