On 20/03/10 17:31, Nigel Kersten wrote:
> On Sat, Mar 20, 2010 at 8:46 AM, Brice Figureau
> <brice-pup...@daysofwonder.com> wrote:
>> On 20/03/10 14:56, Nigel Kersten wrote:
>>> On Sat, Mar 20, 2010 at 3:45 AM, Brice Figureau
>>> <brice-pup...@daysofwonder.com> wrote:
>>>> On 20/03/10 02:51, Nigel Kersten wrote:
>>>>> So this seemed worth another thread given the significant progress...
>>>>> just about all done by Brice as usual.... :)
>>>>>
>>>>> With jruby 1.4.0 and jetty-rackup from
>>>>> http://github.com/geekq/jetty-rackup and a simple config.ru file that
>>>>> looks like:
>>>>
>>>> I'm using without issue:
>>>> jruby 1.5.0.dev (ruby 1.8.7 patchlevel 174) (2010-03-08 1cc36d4)
>>>>
>>>>> -----------------
>>>>> $0 = "puppetmasterd"
>>>>> require 'puppet'
>>>>>
>>>>> ARGV << "--verbose"
>>>>> #ARGV << "--debug"
>>>>> #ARGV << "--trace"
>>>>> # I have a special config file for my servers
>>>>> ARGV << "--config" << "/etc/puppet/puppetmasterd.conf"
>>>>> ARGV << "--no-daemonize"
>>>>> ARGV << "--user" << "puppet"
>>>>> ARGV << "--no-ca"
>>>>>
>>>>> ARGV << "--rack"
>>>>> require 'puppet/application/puppetmasterd'
>>>>> # we're usually running inside a Rack::Builder.new {} block,
>>>>> # therefore we need to call run *here*.
>>>>> run Puppet::Application[:puppetmasterd].run
>>>>> -----------------
>>>>>
>>>>> then this little modification to jetty-rackup
>>>>> -----------------
>>>>> jetty = org.mortbay.jetty.Server.new options[:Port]
>>>>> # next 3 lines are new
>>>>> jetty.get_connectors.each do |c|
>>>>> c.set_header_buffer_size 8192
>>>>> end
>>>>> -----------------
>>>>
>>>> I was about to send you this one :-) but you figured out it.
>>>>
>>>>> and we have something that almost works.
>>>>>
>>>>> err: Could not retrieve catalog: Uncaught exception fork is unsafe and
>>>>> disabled by default on JRuby in method puppetmaster.getconfig
>>>>
>>>> It's strange I don't get that.
>>>>
>>>> Maybe we could rewrite Puppet::Util#execute for jruby to use the JVM
>>>> execute instead of forking.
>>>>
>>>>> I've enabled forking.... -J-Djruby.fork.enabled=true and crossed my
>>>>> fingers... oh look at all this...
>>>>>
>>>>> WARNING: fork is highly unlikely to be safe or stable on the JVM. Have
>>>>> fun!
>>>>> WARNING: fork is highly unlikely to be safe or stable on the JVM. Have
>>>>> fun!
>>>>> WARNING: fork is highly unlikely to be safe or stable on the JVM. Have
>>>>> fun!
>>>>>
>>>>> and we blow up earlier than we did before... Tried the latest HEAD,
>>>>> it was completely broken, so reverted to a few commits earlier, and
>>>>> made some progress... but then the actual forking looks to fail.
>>>>>
>>>>> info: Autoloaded module base
>>>>> debug: Puppet::Type::Package::ProviderRpm: Executing '/usr/bin/rpm
>>>>> --version'
>>>>> Could not autoload rpm: No child processes - No child processes
>>>>
>>>> OK, in fact looking harder, I have this call to execute, but it fails in
>>>> my JRuby version without impacting the compilation itself.
>>>>
>>>> In fact the issue comes from this code from the rpm provider:
>>>> if command('rpm')
>>>> confine :true => begin
>>>> rpm('--version')
>>>> rescue Puppet::ExecutionFailure
>>>> false
>>>> else
>>>> true
>>>> end
>>>> end
>>>>
>>>> This code is executed because each type when compiling is asked if it's
>>>> a builtin type, which in turn autoload each provider, which in turn
>>>> executes such "global" code (ie checking confine).
>>>> I don't think it is necessary to autoload the providers (type should be
>>>> enough for the master, isn't it?), but I'm afraid it isn't possible.
>>>
>>> So when I thought about this more, I got more confused. I really
>>> thought providers were pretty much ignored by the server. This seems
>>> quite undesirable to have any of this come into play at all on the
>>> server.
>>
>> I was surprised to find this too. I really thought we weren't loading
>> any provider on the server.
>>
>>> I take it you're saying that it isn't possible to just autoload the
>>> type without the provider?
>>
>> I don't see why it shouldn't be possible, but looking the type auto
>> loader code there is this:
>>
>> # Define a new type.
>> def newtype(name, options = {}, &block)
>> ...
>> # Now set up autoload any providers that might exist for this type.
>> klass.providerloader = Puppet::Util::Autoload.new(klass,
>> "puppet/provider/#{klass.name.to_s}"
>> )
>>
>> # We have to load everything so that we can figure out the default type.
>> klass.providerloader.loadall()
>> ...
>> end
>>
>> I don't think it's necessary to load the provider on the server, but
>> this last comment looks strange to me.
>>
>> I think it is safe to comment those lines in puppet/metatype/manager.rb
>> for your experiment.
>
> yeah, no difference with a quick test. I'll do some more tracing later
> to work out where this is coming from.
That means something else is loading the providers.
I'm wondering where.
I still thinks it is not necessary to load the providers on the master
(this has no real sense).
>>
>>>>
>>>> Anyway if you checkout jruby 1cc36d4 and use the default option you get
>>>> a nice stack trace but no failure of the compilation :-)
>>>
>>> ahah ;) I'll give that a try.
>>>
>>>>
>>>>> It looks like we only fork in three places in Puppet for what it's worth.
>>>>
>>>> Yes, it's:
>>>> - when daemonizing which you and I are not doing
>>>> - when executing (or generate)
>>>> - in puppetrun
>>>>
>>>> I think execute should be rewritten to use the JVM Runtime.exe or
>>>> ProcessBuilder for JRuby, instead of forking. I'll see if I can come
>>>> with a patch later in the week-end.
>>>
>>> I was thinking the same thing, I'll have a poke around too.
>>
>> Something along of:
>>
>> require 'java'
>>
>> include_class 'java.lang.Runtime'
>> include_class 'java.io.InputStreamReader'
>> include_class 'java.io.BufferedReader'
>>
>> process = Runtime.runtime.exec(command)
>> stream = BufferedReader.new(
>> InputStreamReader.new(
>> process.input_stream
>> )
>> )
>>
>> output = ""
>> while line = stream.read_line
>> output << "#{line}\n"
>> end
>>
>> process.wait_for
>>
>> should do it fine
>
> For normal execution :)
Yes, but that should be more than enough for generate() to work on a master.
> Our execute method is a little weirder with all the failonfail,
> stdinfile and combine args, some of which some of my providers rely
> on.
Yes, but I don't think you run your providers on your master. puppetd on
Jruby will be more difficult to have.
> Let's assume we're in the happy place where JRuby is working perfectly
> apart from this issue. How would we maintain this cleanly in the code
> base?
I don't think we should. I think we should indeed push patches to JRuby.
But I fear not everything will be supported (ie fork for instance might
be really difficult to add to JRuby).
To specifically answer your question, maybe we could have a
Puppet::JRuby module in which we could put our jruby stuff.
An interesting question: how do we check we run on JRuby?
--
Brice Figureau
My Blog: http://www.masterzen.fr/
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to puppet-...@googlegroups.com.
To unsubscribe from this group, send email to
puppet-dev+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en.
