On 30/09/10 16:37, Brice Figureau wrote:
> On Thu, 2010-09-30 at 06:01 -0700, Nigel Kersten wrote:
>> On Thu, Sep 30, 2010 at 1:21 AM, Brice Figureau
>> <[email protected]> wrote:
>>> On Wed, 2010-09-29 at 17:32 -0700, Jason Wright wrote:
>>>> On Wed, Sep 29, 2010 at 1:54 PM, Brice Figureau
>>>> <[email protected]> wrote:
>>>>> It would be great if you could add some debug statements to the
>>>>> lib/puppet/indirector/yaml.rb file around line 22 to show what the YAML
>>>>> look like, and/or what cache it was trying to load.
>>>>
>>>> I added
>>>>
>>>> Puppet.debug("FOO: failed to read YAML from #{file}") if yaml.nil?
>>>> or yaml.to_s == ""
>>>>
>>>> at line 19 of puppet/indirector/yaml.rb and it's logging when I run
>>>> puppet-load so it looks like something is failing in readlock().
>>>
>>> Yes that was my gut feeling too.
>>> I think part of the issue is that puppet-load asks always for the same
>>> node. In real world setups it is improbable that the master has to
>>> answer the same question at exactly the same time.
>>> So I think there is a race in the indirector yaml caching subsystem. It
>>> looks like readlock and writelock are not doing their job.
>
> I found several issues that are worth looking into:
>
> 1) Puppet::Util.sync doesn't seem thread-safe
> Two threads can enter this method at the same time for the same
> resource. Thus it might be possible to exit with two different Sync
> instance for the same resource. There are low chance with MRI
> green-threading, but this can happen under JRuby. Which means a thread
> can write the file at the same time another can read it (flock is per
> process and shouldn't lock a given thread).
Jason, can you try the following ugly patch (on the master):
diff --git a/lib/puppet/util.rb b/lib/puppet/util.rb
index bb41270..3611a42 100644
--- a/lib/puppet/util.rb
+++ b/lib/puppet/util.rb
@@ -2,6 +2,7 @@
require 'puppet/util/monkey_patches'
require 'sync'
+require 'monitor'
require 'puppet/external/lock'
module Puppet
@@ -18,12 +19,14 @@ module Util
extend Puppet::Util::POSIX
# Create a hash to store the different sync objects.
- @@syncresources = {}
+ @@syncresources = {}.extend(MonitorMixin)
# Return the sync object associated with a given resource.
def self.sync(resource)
- @@syncresources[resource] ||= Sync.new
- @@syncresources[resource]
+ @@syncresources.synchronize do
+ @@syncresources[resource] ||= Sync.new
+ @@syncresources[resource]
+ end
end
# Change the process to a different user
It should fix the 1) issue outlined above. I don't think it is the root
cause of your issue, but that's the best I could think about :(
So let's try if that fixes it, we might be lucky today :)
--
Brice Figureau
My Blog: http://www.masterzen.fr/
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.
