Cool stuff. I probably should have alerted you to the 2.0 version of Mina has bugs up the wazoo and would have advocated not to use it. I also back ported to 1.1.2 for the AyncHttpClient stuff, and that stabilized everything.
Jeff Jason Dillon wrote: > Well... me thinks I've finally gotten over the major hump... > > I had been using Mina 2.x (which is in pre-release right now)... I was > using it because it has some synchronous request/response handling muck > already there which I kinda needed to implement the remote shell proxy > muck. And well, I must have spent like a week digging through code, > running tests, rebuilding my tree, and really... going crazy because no > matter what I did nothing seemed to behave itself as I needed it to. > > So I ported and basically re-wrote all that request/response stuff I > wanted so I could add debug to it and understand why it was not happy... > or try to at least. It really seemed like there was some kind of > threading problem or a sync problem but all of the diag I got from logs > just made no sense and the results I get from running tests were random > at best. > > Ugh... so anyways, I wanted to use Mina 2.0 for the request/response > stuff... which I rewrote anyways... so a little before midnight on the > night before the 20th I decided to move Mina 1.1.2 (latest official > release). Updated some apis, hacked around a few other bits... and > about 10 or 10 minutes into the 20th I had the tcp transports all ported > and... um working!!! for the normally async, req/resp and for the stream > i/o all at the same time. It was a nice birthday present... such a good > feeling to finally get this crap working after going mad over it over > the past few days!!! :-) > > And now over the days past I've fixed up the vm:// transport and fixed > some bugs in the req/resp handling which were causing things to puke up > insanity in a oh so blissful random form. > > Um... ya, so I think I've finally mastered... er well, beaten to > submission, mina... which I still really like a lot btw... And I've > implemented some framework to make using it a little easier for me, > probably going to invent some more as I go... but right now... um... it > seems to work. Tcp, ssl, vm... all working happy, happy. Message > passing along side of stream i/o sitting on top of a few > request/response bits to boot. > > Oh ya... and well, you can actually get a remote shell with this stuff > now. Its not 100% functional, but you can connect to a remote server > (running a rsh-server command), authenticate (internally it will do some > rsa key exchange muck too) and then well, you are running commands on > the remote system. And the output comes back to you, and you can 'exit' > the remote shell and get back to your local shell. Wee. > > Its still not done though, all this batting with mina had kept me from > finishing up the gshell rsh protocl bits and finishing up the state > management and such... but I think I'm past that now, so I really hope > to get these loose ends tied up soonish. Its still a bit POC, I'm still > learning and adapting to how best to implement this stuff, but I expect > the first version to be a "just make it work" and then after that a bit > of refactor to clean things up (which really has how most of the gshell > implementation has been going... sans the first week of insane code > drops when I wrote most of it). > > * * * > > Okay, I'll stop babbling now... my point is... the remoting crapo > works... and I think that maybe in another week or so there should be > something functional enough for me to integration back into Geronimo. > Something like a gsh rsh into a running server (over ssl with > authentication) and then maybe a few stupid commands to show vm stats, > shutdown or something else. I guess since some peeps are keen for > scripting muck I can show how you can use the script command to execute > a script on a server too ;-) > > I'm still working on the 'makecoffee --now' command... > > :-P > > Oh, and lastly... and then ya... I'll shut up... If anyone is even > remotely interested in any of this crap it would be nice to have another > mass of grey matter pondering over some of its present and future... > > Cheers, > > --jason > > > PS. Sorry, I couldn't resist... I think that the stuff in there now > can probably support a telnet as well as ssh transport impl too. Though > I've yet to find an ASL friendly java ssh client+server library to build > it with... > > > > On Sep 18, 2007, at 3:09 PM, Jason Dillon wrote: > >> Hiya folks... just a quick update on GShell muck... >> >> I've been madly hacking on the remote shell support for the past week >> and well, its really close to being functional. I initially was going >> to implement a POC over ActiveMQ but that was pissing me off soo much >> that I decided to peek at what Jeff had done with GShell and >> discovered how incredibly cool Apache Mina is!! So I've based the >> remote-shell client/server bits off of Mina and its working out fairly >> well, though I had to go and learn about it (which was actually >> fun-ish, sorta, okay not really so much :-P ). >> >> Anyways, I've dug through GCache (which really did help me to >> understand Mina) as well as some other Mina-based projects and I've >> whipped up a fairly simple system (er I think so), which is extensible >> supports TCP, SSL and VM (er well almost VM), so you can do something >> like: >> >> ./bin/gsh rsh ssl://someserver:9999 >> >> or >> >> ./bin/gsh rsh tcp://someserver:9999 >> >> The same is true for the server side, to be able to accept these rsh >> commands you could: >> >> ./bin/gsh rsh-server ssl://localhost:9999 >> >> or >> >> ./bin/gsh rsh-server tcp://localhost:9999 >> >> etc... >> >> I've implemented a RSA-based client-server handshake (similar to what >> gcache was doing, actually based on what it was doing) to support a >> decent authentication mechanism. Might eventually hook up some JAAS >> crapo later, but... ya later. >> >> Anyways, right now I'm trying to figure out some small problems with >> the stream message passing and marshaling, which I hope to get >> resolved really damn soon, cause this is fricken annoying me. And >> once that is sorted, a few minor clean up things and the >> rsh/rsh-server bits should be functional enough to actually use. >> >> More to come... >> >> Cheers, >> >> --jason >> >> >> >> On Sep 8, 2007, at 12:40 PM, Jason Dillon wrote: >> >>> A little bit more insight into what I'm thinking of doing... since >>> some of you can't read minds to well :-P >>> >>> I'd like to convert all of the assemblies to basically look like what >>> the assemblies/geronimo-jetty6-javaee5-gshell produces. >>> >>> And then I'd like to start converting the other cli bits to gshell >>> command impls, like: deployer, client and shutdown. >>> >>> And then (maybe around the same time or before the above), I'd like >>> to adapt the gshell of target jvm bits to load jars from the >>> repository, instead of using the lib/* bits. >>> >>> A little background for those who haven't looked at >>> assemblies/geronimo-jetty6-javaee5-gshell and what it produces from a >>> lib/* perspective. Right now I've set up the assembly to produce: >>> >>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib >>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/boot >>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/endorsed >>> geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/gshell >>> >>> Where the bits in lib/* and lib/endorsed/* are the same as they were >>> before. The bits in lib/boot/* and lib/gshell/* are specific to >>> gshell. And normally a gshell installation would have everything I >>> put into lib/gshell/* into lib/*, but I moved them to a sub dir for >>> now... since the bin/*.jar's load jars from the ../lib/* dirs. >>> >>> The lib/boot/* stuff is the very minimal gshell bootstrap classes, >>> which setup up the other happiness... and let you do things like: >>> >>> java -jar >>> ./geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/boot/gshell-bootstrap.jar >>> >>> >>> And that will give you a nice shell... or >>> >>> java -jar >>> ./geronimo-jetty6-javaee5-gshell-2.1-SNAPSHOT/lib/boot/gshell-bootstrap.jar >>> start-server >>> >>> That will launch the G server process using all of the right >>> -Djava.ext.dirs and whatever properties that we currently have hacked >>> into platform scripts. >>> >>> Anyways, so the idea is to move all of the bits which are current in >>> the lib/* into the repository, and then configure the gshell command >>> impl to load put the correct dependency artifacts onto the classpath >>> of the target jvm that is booted up. This will augment the existing >>> kernel bootstrap from repo stuff, putting evertying except what is >>> needed from gshell into the repository... >>> >>> And really, what I'd like to eventually get to is having the >>> bootstrap from the repository... so that everything except for what >>> is now it lib/boot/* and lib/endorsed/* can live in the repository >>> like happy little communistic jars should be :-P >>> >>> * * * >>> >>> And then there are longer term things for GShell... >>> >>> Remote administration (via, telnet, ssh, or custom ssl protocol... >>> last is most likely to actually happen soonish) >>> >>> Process management, which is great for clusters, or staging -> >>> production management. A full suite of command-line tools which can >>> manage the configuration of a server... easily. So, for example, >>> lets say you've got a configuration that is working really well for >>> you... but you want to play with something new... >>> >>> So you might: >>> >>> ./bin/gsh backup-configuration before-mucking >>> ./bin/gsh start-server >>> >>> And then go and change a whole bunch of stuff... and it doesn't >>> work... yikes... so rollback... >>> >>> ./bin/gsh backup-configuration hosed-server >>> ./bin/gsh restore-configuration before-mucking >>> ./bin/gsh start-server >>> >>> And then maybe you want to play with the "hosed-server" configuration >>> again... >>> >>> ./bin/gsh start-server --configuration hosed-server >>> >>> Of course, all of these could have been run from a single ./bin/gsh, >>> but just for clarity, you can run them one off too. >>> >>> Maybe list or mange the configurations >>> >>> ./bin/gsh list-configurations >>> ./bin/gsh remove-configuration some-unwanted-config >>> ./bin/gsh copy-configuration default some-new-config >>> >>> The sky is the limit really... for what kind of management we can do... >>> >>> Lets say you wanted to do the above on a remote node? >>> >>> ./bin/gsh remote-shell someserver:9443 >>> Connecting to someserver:9447... >>> Connected >>> >>> username: system >>> password: **** (remember this is all jline, so we can mask >>> passwords like one woudl expect) >>> >>> someserver:9447 > list-configurations >>> someserver:9447 > remove-configuration some-unwanted-config >>> someserver:9447 > copy-configuration default some-new-config >>> >>> So, all of these operations would happen on the node named >>> "someserver" listening on 9443 (over ssl of course). Or how about >>> you want to reboot a server remotely? >>> >>> someserver:9447 > restart-server now >>> Geronimo server shutting down... >>> .... >>> Geronimo server shutdown. >>> Geronimo server starting... >>> ... >>> Geronimo server started in ... >>> >>> Since GShell manages the processes its really easy to perform a full >>> restart of a Server w/o needing magical platform scripting muck. And >>> it will just work the same on each platform too. >>> >>> Once we have clustering, then we can do the same kinda thing for an >>> entire cluster of nodes... >>> >>> someserver:9447 > restart-cluster now >>> Shutting down 2 nodes... >>> <node1> Geronimo server shutting down... >>> <node1>.... >>> <node2> Geronimo server shutting down... >>> <node2>.... >>> <node1>Geronimo server shutdown. >>> <node2>Geronimo server shutdown. >>> Starting up 2 nodes... >>> <node1>Geronimo server starting... >>> <node1>.. >>> <node2>Geronimo server starting... >>> <node2>.. >>> <node1>Geronimo server started in ... >>> <node2>Geronimo server started in ... >>> Started up 2 nodes. >>> >>> And well, if you had some kinda script file which controlled say a >>> logical grouping of nodes you could easily invoke that script (ya >>> even on a remote system) and it will go and do it: >>> >>> someserver:9447 > script -l groovy >>> local:file://restart-universe.groovy qa-universe >>> >>> The local: bit of the uri siginals the local URL handler to be used, >>> which will cause the file://restart-universe.groovy to be loaded from >>> the gsh instance where you are actually logged into (and ran the >>> remote-shell gshell command) and will pipe its contents securely to >>> the remote shell running on someserver:9447 and pass it to the script >>> command to execute. >>> >>> The restart-universe.groovy might look something like this: >>> >>> <snip> >>> import universe.Lookup >>> >>> assert args.size == 1 : 'Missing universe name' >>> >>> def universe = args[0] >>> >>> // Look up a list of nodes (for now say they are basically >>> hostname:port) >>> def nodes = Lookup.lookup(universe) >>> >>> log.info("Stopping universe ${universe}...") >>> nodes.each { host -> >>> shell.execute("remove-shell $host stop-server") >>> } >>> log.info("Universe ${universe} stopped") >>> >>> log.info("Starting universe ${universe}...") >>> nodes.each { host -> >>> shell.execute("remove-shell $host start-server") >>> } >>> log.info("Universe ${universe} started") >>> </snip> >>> >>> Its kinda crude script, but I think you get the general point... >>> >>> * * * >>> >>> Anyways... I see... well, *HUGE* potential for this stuff... >>> >>> And really, a lot of what I just described above isn't that far into >>> fantasy, its all relatively easy to implement on top of GShell... as >>> it is now (or really as it was a year+ ago when I wrote it). Its >>> really a matter of do others see the same value... and do others see >>> the vision of using GShell as the core process launcher to allow >>> things like "restart-server", or a "stop-server; copy-configuration >>> default known-good; copy-configuration default testing; >>> start-server", or that uber-fancy remote-shell muck. >>> >>> So, I'm gonna give y'all a few days to grok (or try to) what I've >>> just spit out... please ask questions or comment, as I like to know >>> I'm not just talking to myself here. >>> >>> And then maybe later next week, we might vote or come to some other >>> consensus that this is the right direction for Geronimo, and well... >>> then I'll make it become reality. >>> >>> Aighty, and now I'll shut up :-P >>> >>> --jason >>> >>> >>> >>> On Sep 8, 2007, at 11:53 AM, Jason Dillon wrote: >>> >>>> Aighty, well... I've done some long awaited re-factoring, and while >>>> its still not _perfect_ its a whole lot better now IMO I think from >>>> a framework perspective that its probably mature enough to take on >>>> the task of being the server bootloader. >>>> >>>> I'm going to continue to refactor the guts of GShell over time, of >>>> course... but I think that what is there now is highly usable for a >>>> simple platform independent launcher, as well as for porting over >>>> the other cli bits we have. >>>> >>>> I've done a lot of work in the past week, incase you didn't see the >>>> storm of scm messages... pulled out pico, plopped in plexus, pulled >>>> out commons-logging and commons-lang, which are suck and boated (in >>>> that order). I've gotten the basic framework and supported classes >>>> to use GShell down to ~ 1mb (a wee bit under)... though when I >>>> started to add the layout.xml abstraction stuff, I had to pull in >>>> xstream which bloated her back up to ~1.4m. I may eventually fix >>>> that... or not, cause xstream is soooo very handy for xml -> object >>>> stuff. >>>> >>>> I've fallen in love with annotations... they are *ucking great. >>>> They work really well for handling the cli option and argument muck >>>> which most every command needs to do. And striping out the >>>> insano-sucking commons-cli really simplified command implementations >>>> dramatically IMO. >>>> >>>> Anyways... I've make a heck of a lot of progress on cleaning up the >>>> GShell framework... and more is to come I'm sure... But for now, I >>>> think its probably ready for use primetime as the Geronimo Server's >>>> bootloader. >>>> >>>> I think this provides a some significant value... >>>> >>>> 1) Platform scripts become consistent and relatively simple, easy >>>> to maintain >>>> >>>> 2) Everyone will now have a consist way of launching the server, >>>> even if you like a .sh, .bat, or java -jar, then end process that is >>>> launched will be the same for everyone. >>>> >>>> 3) Opens up the door for some really nice and fancy fancy >>>> management muck (like restarting the server from the web console, or >>>> cloning a server instance or backing up a server instance...) >>>> >>>> 4) Lays the ground work for future features, like cluster >>>> management, remote administration and scripting... >>>> >>>> * * * >>>> >>>> So, I think its time to decide... are we a go or no go for GShell as >>>> the core CLI for Geronimo thingys and even more important, are we go >>>> or no go for using GShell to boot up the server process? >>>> >>>> --jason >>>> >>> >>