Cool stuff. I probably should have alerted you to the 2.0 version of
Mina has bugs up the wazoo and would have advocated not to use it. I
also back ported to 1.1.2 for the AyncHttpClient stuff, and that
stabilized everything.
Jeff
Jason Dillon wrote:
> Well... me thinks I've finally gotten over the major hump...
>
> I had been using Mina 2.x (which is in pre-release right now)... I was
> using it because it has some synchronous request/response handling muck
> already there which I kinda needed to implement the remote shell proxy
> muck. And well, I must have spent like a week digging through code,
> running tests, rebuilding my tree, and really... going crazy because no
> matter what I did nothing seemed to behave itself as I needed it to.
>
> So I ported and basically re-wrote all that request/response stuff I
> wanted so I could add debug to it and understand why it was not happy...
> or try to at least. It really seemed like there was some kind of
> threading problem or a sync problem but all of the diag I got from logs
> just made no sense and the results I get from running tests were random
> at best.
>
> Ugh... so anyways, I wanted to use Mina 2.0 for the request/response
> stuff... which I rewrote anyways... so a little before midnight on the
> night before the 20th I decided to move Mina 1.1.2 (latest official
> release). Updated some apis, hacked around a few other bits... and
> about 10 or 10 minutes into the 20th I had the tcp transports all ported
> and... um working!!! for the normally async, req/resp and for the stream
> i/o all at the same time. It was a nice birthday present... such a good
> feeling to finally get this crap working after going mad over it over
> the past few days!!! :-)
>
> And now over the days past I've fixed up the vm:// transport and fixed
> some bugs in the req/resp handling which were causing things to puke up
> insanity in a oh so blissful random form.
>
> Um... ya, so I think I've finally mastered... er well, beaten to
> submission, mina... which I still really like a lot btw... And I've
> implemented some framework to make using it a little easier for me,
> probably going to invent some more as I go... but right now... um... it
> seems to work. Tcp, ssl, vm... all working happy, happy. Message
> passing along side of stream i/o sitting on top of a few
> request/response bits to boot.
>
> Oh ya... and well, you can actually get a remote shell with this stuff
> now. Its not 100% functional, but you can connect to a remote server
> (running a rsh-server command), authenticate (internally it will do some
> rsa key exchange muck too) and then well, you are running commands on
> the remote system. And the output comes back to you, and you can 'exit'
> the remote shell and get back to your local shell. Wee.
>
> Its still not done though, all this batting with mina had kept me from
> finishing up the gshell rsh protocl bits and finishing up the state
> management and such... but I think I'm past that now, so I really hope
> to get these loose ends tied up soonish. Its still a bit POC, I'm still
> learning and adapting to how best to implement this stuff, but I expect
> the first version to be a "just make it work" and then after that a bit
> of refactor to clean things up (which really has how most of the gshell
> implementation has been going... sans the first week of insane code
> drops when I wrote most of it).
>
> * * *
>
> Okay, I'll stop babbling now... my point is... the remoting crapo
> works... and I think that maybe in another week or so there should be
> something functional enough for me to integration back into Geronimo.
> Something like a gsh rsh into a running server (over ssl with
> authentication) and then maybe a few stupid commands to show vm stats,
> shutdown or something else. I guess since some peeps are keen for
> scripting muck I can show how you can use the script command to execute
> a script on a server too ;-)
>
> I'm still working on the 'makecoffee --now' command...
>
> :-P
>
> Oh, and lastly... and then ya... I'll shut up... If anyone is even
> remotely interested in any of this crap it would be nice to have another
> mass of grey matter pondering over some of its present and future...
>
> Cheers,
>
> --jason
>
>
> PS. Sorry, I couldn't resist... I think that the stuff in there now
> can probably support a telnet as well as ssh transport impl too. Though
> I've yet to find an ASL friendly java ssh client+server library to build
> it with...
>
>
>
> On Sep 18, 2007, at 3:09 PM, Jason Dillon wrote:
>
>> Hiya folks... just a quick update on GShell muck...
>>
>> I've been madly hacking on the remote shell support for the past week
>> and well, its really close to being functional. I initially was going
>> to implement a POC over ActiveMQ but that was pissing me off soo much
>> that I decided to peek at what Jeff had done with GShell and
>> discovered how incredibly cool Apache Mina is!! So I've based the
>> remote-shell client/server bits off of Mina and its working out fairly
>> well, though I had to go and learn about it (which was actually
>> fun-ish, sorta, okay not really so much :-P ).
>>
>> Anyways, I've dug through GCache (which really did help me to
>> understand Mina) as well as some other Mina-based projects and I've
>> whipped up a fairly simple system (er I think so), which is extensible
>> supports TCP, SSL and VM (er well almost VM), so you can do something
>> like:
>>
>> ./bin/gsh rsh ssl://someserver:9999
>>
>> or
>>
>> ./bin/gsh rsh tcp://someserver:9999
>>
>> The same is true for the server side, to be able to accept these rsh
>> commands you could:
>>
>> ./bin/gsh rsh-server ssl://localhost:9999
>>
>> or
>>
>> ./bin/gsh rsh-server tcp://localhost:9999
>>
>> etc...
>>
>> I've implemented a RSA-based client-server handshake (similar to what
>> gcache was doing, actually based on what it was doing) to support a
>> decent authentication mechanism. Might eventually hook up some JAAS
>> crapo later, but... ya later.
>>
>> Anyways, right now I'm trying to figure out some small problems with
>> the stream message passing and marshaling, which I hope to get
>> resolved really damn soon, cause this is fricken annoying me. And
>> once that is sorted, a few minor clean up things and the
>> rsh/rsh-server bits should be functional enough to actually use.
>>
>> More to come...
>>
>> Cheers,
>>
>> --jason
>>
>>
>>
>> On Sep 8, 2007, at 12:40 PM, Jason Dillon wrote:
>>
>>> A little bit more insight into what I'm thinking of doing... since
>>> some of you can't read minds to well :-P
>>>
>>> I'd like to convert all of the assemblies to basically look like what
>>> the assemblies/geronimo-jetty6-javaee5-gshell produces.
>>>
>>> And then I'd like to start converting the other cli bits to gshell
>>> command impls, like: deployer, client and shutdown.
>>>
>>> And then (maybe around the same time or before the above), I'd like
>>> to adapt the gshell of target jvm bits to load jars from the
>>> repository, instead of using the lib/* bits.
>>>
>>> A little background for those who haven't looked at
>>> assemblies/geronimo-jetty6-javaee5-gshell and what it produces from a
>>> lib/* perspective. Right now I've set up the assembly to produce:
>>>
>>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib
>>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/boot
>>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/endorsed
>>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/gshell
>>>
>>> Where the bits in lib/* and lib/endorsed/* are the same as they were
>>> before. The bits in lib/boot/* and lib/gshell/* are specific to
>>> gshell. And normally a gshell installation would have everything I
>>> put into lib/gshell/* into lib/*, but I moved them to a sub dir for
>>> now... since the bin/*.jar's load jars from the ../lib/* dirs.
>>>
>>> The lib/boot/* stuff is the very minimal gshell bootstrap classes,
>>> which setup up the other happiness... and let you do things like:
>>>
>>> java -jar
>>> ./geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/boot/gshell-bootstrap.jar
>>>
>>>
>>> And that will give you a nice shell... or
>>>
>>> java -jar
>>> ./geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/boot/gshell-bootstrap.jar
>>> start-server
>>>
>>> That will launch the G server process using all of the right
>>> -Djava.ext.dirs and whatever properties that we currently have hacked
>>> into platform scripts.
>>>
>>> Anyways, so the idea is to move all of the bits which are current in
>>> the lib/* into the repository, and then configure the gshell command
>>> impl to load put the correct dependency artifacts onto the classpath
>>> of the target jvm that is booted up. This will augment the existing
>>> kernel bootstrap from repo stuff, putting evertying except what is
>>> needed from gshell into the repository...
>>>
>>> And really, what I'd like to eventually get to is having the
>>> bootstrap from the repository... so that everything except for what
>>> is now it lib/boot/* and lib/endorsed/* can live in the repository
>>> like happy little communistic jars should be :-P
>>>
>>> * * *
>>>
>>> And then there are longer term things for GShell...
>>>
>>> Remote administration (via, telnet, ssh, or custom ssl protocol...
>>> last is most likely to actually happen soonish)
>>>
>>> Process management, which is great for clusters, or staging ->
>>> production management. A full suite of command-line tools which can
>>> manage the configuration of a server... easily. So, for example,
>>> lets say you've got a configuration that is working really well for
>>> you... but you want to play with something new...
>>>
>>> So you might:
>>>
>>> ./bin/gsh backup-configuration before-mucking
>>> ./bin/gsh start-server
>>>
>>> And then go and change a whole bunch of stuff... and it doesn't
>>> work... yikes... so rollback...
>>>
>>> ./bin/gsh backup-configuration hosed-server
>>> ./bin/gsh restore-configuration before-mucking
>>> ./bin/gsh start-server
>>>
>>> And then maybe you want to play with the "hosed-server" configuration
>>> again...
>>>
>>> ./bin/gsh start-server --configuration hosed-server
>>>
>>> Of course, all of these could have been run from a single ./bin/gsh,
>>> but just for clarity, you can run them one off too.
>>>
>>> Maybe list or mange the configurations
>>>
>>> ./bin/gsh list-configurations
>>> ./bin/gsh remove-configuration some-unwanted-config
>>> ./bin/gsh copy-configuration default some-new-config
>>>
>>> The sky is the limit really... for what kind of management we can do...
>>>
>>> Lets say you wanted to do the above on a remote node?
>>>
>>> ./bin/gsh remote-shell someserver:9443
>>> Connecting to someserver:9447...
>>> Connected
>>>
>>> username: system
>>> password: **** (remember this is all jline, so we can mask
>>> passwords like one woudl expect)
>>>
>>> someserver:9447 > list-configurations
>>> someserver:9447 > remove-configuration some-unwanted-config
>>> someserver:9447 > copy-configuration default some-new-config
>>>
>>> So, all of these operations would happen on the node named
>>> "someserver" listening on 9443 (over ssl of course). Or how about
>>> you want to reboot a server remotely?
>>>
>>> someserver:9447 > restart-server now
>>> Geronimo server shutting down...
>>> ....
>>> Geronimo server shutdown.
>>> Geronimo server starting...
>>> ...
>>> Geronimo server started in ...
>>>
>>> Since GShell manages the processes its really easy to perform a full
>>> restart of a Server w/o needing magical platform scripting muck. And
>>> it will just work the same on each platform too.
>>>
>>> Once we have clustering, then we can do the same kinda thing for an
>>> entire cluster of nodes...
>>>
>>> someserver:9447 > restart-cluster now
>>> Shutting down 2 nodes...
>>> <node1> Geronimo server shutting down...
>>> <node1>....
>>> <node2> Geronimo server shutting down...
>>> <node2>....
>>> <node1>Geronimo server shutdown.
>>> <node2>Geronimo server shutdown.
>>> Starting up 2 nodes...
>>> <node1>Geronimo server starting...
>>> <node1>..
>>> <node2>Geronimo server starting...
>>> <node2>..
>>> <node1>Geronimo server started in ...
>>> <node2>Geronimo server started in ...
>>> Started up 2 nodes.
>>>
>>> And well, if you had some kinda script file which controlled say a
>>> logical grouping of nodes you could easily invoke that script (ya
>>> even on a remote system) and it will go and do it:
>>>
>>> someserver:9447 > script -l groovy
>>> local:file://restart-universe.groovy qa-universe
>>>
>>> The local: bit of the uri siginals the local URL handler to be used,
>>> which will cause the file://restart-universe.groovy to be loaded from
>>> the gsh instance where you are actually logged into (and ran the
>>> remote-shell gshell command) and will pipe its contents securely to
>>> the remote shell running on someserver:9447 and pass it to the script
>>> command to execute.
>>>
>>> The restart-universe.groovy might look something like this:
>>>
>>> <snip>
>>> import universe.Lookup
>>>
>>> assert args.size == 1 : 'Missing universe name'
>>>
>>> def universe = args[0]
>>>
>>> // Look up a list of nodes (for now say they are basically
>>> hostname:port)
>>> def nodes = Lookup.lookup(universe)
>>>
>>> log.info("Stopping universe ${universe}...")
>>> nodes.each { host ->
>>> shell.execute("remove-shell $host stop-server")
>>> }
>>> log.info("Universe ${universe} stopped")
>>>
>>> log.info("Starting universe ${universe}...")
>>> nodes.each { host ->
>>> shell.execute("remove-shell $host start-server")
>>> }
>>> log.info("Universe ${universe} started")
>>> </snip>
>>>
>>> Its kinda crude script, but I think you get the general point...
>>>
>>> * * *
>>>
>>> Anyways... I see... well, *HUGE* potential for this stuff...
>>>
>>> And really, a lot of what I just described above isn't that far into
>>> fantasy, its all relatively easy to implement on top of GShell... as
>>> it is now (or really as it was a year+ ago when I wrote it). Its
>>> really a matter of do others see the same value... and do others see
>>> the vision of using GShell as the core process launcher to allow
>>> things like "restart-server", or a "stop-server; copy-configuration
>>> default known-good; copy-configuration default testing;
>>> start-server", or that uber-fancy remote-shell muck.
>>>
>>> So, I'm gonna give y'all a few days to grok (or try to) what I've
>>> just spit out... please ask questions or comment, as I like to know
>>> I'm not just talking to myself here.
>>>
>>> And then maybe later next week, we might vote or come to some other
>>> consensus that this is the right direction for Geronimo, and well...
>>> then I'll make it become reality.
>>>
>>> Aighty, and now I'll shut up :-P
>>>
>>> --jason
>>>
>>>
>>>
>>> On Sep 8, 2007, at 11:53 AM, Jason Dillon wrote:
>>>
>>>> Aighty, well... I've done some long awaited re-factoring, and while
>>>> its still not _perfect_ its a whole lot better now IMO I think from
>>>> a framework perspective that its probably mature enough to take on
>>>> the task of being the server bootloader.
>>>>
>>>> I'm going to continue to refactor the guts of GShell over time, of
>>>> course... but I think that what is there now is highly usable for a
>>>> simple platform independent launcher, as well as for porting over
>>>> the other cli bits we have.
>>>>
>>>> I've done a lot of work in the past week, incase you didn't see the
>>>> storm of scm messages... pulled out pico, plopped in plexus, pulled
>>>> out commons-logging and commons-lang, which are suck and boated (in
>>>> that order). I've gotten the basic framework and supported classes
>>>> to use GShell down to ~ 1mb (a wee bit under)... though when I
>>>> started to add the layout.xml abstraction stuff, I had to pull in
>>>> xstream which bloated her back up to ~1.4m. I may eventually fix
>>>> that... or not, cause xstream is soooo very handy for xml -> object
>>>> stuff.
>>>>
>>>> I've fallen in love with annotations... they are *ucking great.
>>>> They work really well for handling the cli option and argument muck
>>>> which most every command needs to do. And striping out the
>>>> insano-sucking commons-cli really simplified command implementations
>>>> dramatically IMO.
>>>>
>>>> Anyways... I've make a heck of a lot of progress on cleaning up the
>>>> GShell framework... and more is to come I'm sure... But for now, I
>>>> think its probably ready for use primetime as the Geronimo Server's
>>>> bootloader.
>>>>
>>>> I think this provides a some significant value...
>>>>
>>>> 1) Platform scripts become consistent and relatively simple, easy
>>>> to maintain
>>>>
>>>> 2) Everyone will now have a consist way of launching the server,
>>>> even if you like a .sh, .bat, or java -jar, then end process that is
>>>> launched will be the same for everyone.
>>>>
>>>> 3) Opens up the door for some really nice and fancy fancy
>>>> management muck (like restarting the server from the web console, or
>>>> cloning a server instance or backing up a server instance...)
>>>>
>>>> 4) Lays the ground work for future features, like cluster
>>>> management, remote administration and scripting...
>>>>
>>>> * * *
>>>>
>>>> So, I think its time to decide... are we a go or no go for GShell as
>>>> the core CLI for Geronimo thingys and even more important, are we go
>>>> or no go for using GShell to boot up the server process?
>>>>
>>>> --jason
>>>>
>>>
>>
