Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 27 May 2011 02:58, David Cantrell wrote: > On Thu, May 26, 2011 at 05:25:01PM +0100, Simon Wistow wrote: > >> Unless things have changed dramatically ActiveMQ has many, many features >> pretty much all poorly documented in the typical ASF/Java project >> fashion (i.e a poorly maintained wiki referencing multiple versions of >> the software implicitly, a collection of hard to navigate JIRA pages and >> masses of badly written Javadoc with no clear entry point). > > You're not talking about ActiveMQ you tease, you're talking about perl! > And Javascript, and exim, and BIND, and Linux, and ... Actually, I reckon a lot of Perl modules have quite good documentation compared to most of the Java modules I've looked into! The Perl community tends to start out with a SYNOPSIS section that includes a simple example of how to use the module, and that can be so helpful to get started. > Of course, with perl (and Javscript, BIND and Linux), if you want to > learn how to use the damned thing and how it all fits together, then > there are some splendid books you can buy and a community of users. > Maybe you can do the same with ActiveMQ. I came across this while looking for comparisons of other queues: http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes After reading through it, my take-away message was: All software sucks. RabbitMQ initially sounded quite good to me, btw, until I saw that message queue replication was still merely planned for future versions. Oh :( But perhaps it's better to have something that has less features but is reliable, than one that promises the moon, but will crash when I least expect it? (As people seem to imply with ActiveMQ) Thanks for all the input so far, guys. -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On Thu, May 26, 2011 at 05:25:01PM +0100, Simon Wistow wrote: > Unless things have changed dramatically ActiveMQ has many, many features > pretty much all poorly documented in the typical ASF/Java project > fashion (i.e a poorly maintained wiki referencing multiple versions of > the software implicitly, a collection of hard to navigate JIRA pages and > masses of badly written Javadoc with no clear entry point). You're not talking about ActiveMQ you tease, you're talking about perl! And Javascript, and exim, and BIND, and Linux, and ... Of course, with perl (and Javscript, BIND and Linux), if you want to learn how to use the damned thing and how it all fits together, then there are some splendid books you can buy and a community of users. Maybe you can do the same with ActiveMQ. -- David Cantrell | Nth greatest programmer in the world Longum iter est per praecepta, breve et efficax per exempla.
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On Thu, May 26, 2011 at 05:25:01PM +0100, me said: > My experience is a few years old and it's very possible things have > changed now Apparently not http://goodstuff.im/activemq-not-ready-for-prime-time Money quote: "I recommended ActiveMQ to a client [...] (and) I am quite frankly tremendously embarrassed. I am writing this so that if you are considering ActiveMQ versus the numerous other open source message queues that you don't make the same mistake that I did... choose something else."
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On Tue, May 24, 2011 at 03:18:27PM +1000, Toby Wintermute said: > By the way.. how are you finding ActiveMQ, especially when interacting > with it from Perl? My experience is a few years old and it's very possible things have changed now but, given a choice between working with ActiveMQ again and sawing off my own limbs with a spork I would choose self-mutilation with the hybrid cutlery every time. Every time. In my experience people's timeline with ActiveMQ goes like this (mine included, obv.) "Oooh, this looks nice. Not too hard to install. Oh, and look how easy it was to get stuff working. This is magic! Awesome. [time passes] Hmm, that's a bit odd, I guess I'll just work around it [more time passes] Wait? What? That's retarded! Why is it doing that? [even more time passes] Holy Shit! Where's my data? Screw this." Unless things have changed dramatically ActiveMQ has many, many features pretty much all poorly documented in the typical ASF/Java project fashion (i.e a poorly maintained wiki referencing multiple versions of the software implicitly, a collection of hard to navigate JIRA pages and masses of badly written Javadoc with no clear entry point). The basics (queues, topics, failover) don't work reliably because it appears like the dev team are hellbent on replicating every last feature of MQSeries and TibCo Rendevous. But doing it badly. ... Ad breathe. You'll have even more trouble with Stomp. It looks like I never released my Net::Stomp::Async module which got round the problem of only ever being able to consume 100 messages/s (because of select()) but you might still run into problems with the fact that Stomp assumes everything is text which can make doing property based routing difficult. Also, (again unless things have changed) ActiveMQ requires that backoff is done by the client. I'm not sure if that's an issue. I've had some issues with RabbitMQ reliability in the past and adminning Erlang can be interesting. More recently I've used ZeroMQ for some stuff, a little Redis PubSub (with custom patches for queues as well as topics), TheSchwartz with specialist workers and written a really simple system in node.js
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On Wed, May 25, 2011 at 02:10, Ruud H.G. van Tol wrote: > On 2011-05-25 09:19, James Laver wrote: >> On 24 May 2011, at 06:31, Daniel Pittman wrote: […] > Anyone here who used Spread recently? I made trivial use of it in both testing and production to serve as a collector for Apache logs. It was very little trouble, and did the job, but was relatively painful to expand the network since it required restarts of the service. The Apache log collector handled that gracefully, but IIRC you need to specifically support it. Performance was reasonable, but then... I was using the 3.1 release at the time, though, so I have no idea if 4.0 or 4.1 improve on that. > It is supposed to be faster, lower level, taking < 1% CPU. ...CPU load on the network was comparable to the load on ActiveMQ systems under similar load. Spread isn't really much different otherwise in terms of either speed, or being "lower level" (in either the sense of being better for, or worse for, that. ;) If I had to pick a "does as little as possible" low level library, 0mq looks like the best available choice, but it provides next to *nothing* in terms of scaling; you have a toolkit for messaging and get to build out the rest of the routing, scaling, persistence, etc yourself. Daniel -- ⎋ Puppet Labs Developer – http://puppetlabs.com ✉ Daniel Pittman ✆ Contact me via gtalk, email, or phone: +1 (503) 893-2285 ♲ Made with 100 percent post-consumer electrons
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On Wed, May 25, 2011 at 02:02, Dean Wilson wrote: > On Wed, May 25, 2011 at 08:19:07AM +0100, James Laver wrote: >> You mean apart from RabbitMQ? Rabbit is also the most full-featured of any >> of the free message queues I've come across. > > Apart from HA, mesh networks and clustering. Rabbit is no where near prime > time for that yet. That is pretty much it: I would absolutely recommend RabbitMQ if you don't need to scale, but ActiveMQ is the only robust choice I found if you do need those things. (For some values of clustering, obviously, given the Erlang clustering that RabbitMQ can do. :) Daniel -- ⎋ Puppet Labs Developer – http://puppetlabs.com ✉ Daniel Pittman ✆ Contact me via gtalk, email, or phone: +1 (503) 893-2285 ♲ Made with 100 percent post-consumer electrons
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 2011-05-25 09:19, James Laver wrote: On 24 May 2011, at 06:31, Daniel Pittman wrote: It is, presently, the best of the options out there in terms of scale You mean apart from RabbitMQ? Rabbit is also the most full-featured of any of the free message queues I've come across. My personal experience of using ActiveMQ is "don't: it'll cause you headache after headache". The same could be said of RabbitMQ, but we were doing some fairly tediously complicated stuff there (stuff that I believe would be impossible under ActiveMQ). Having watched people fight ActiveMQ trying to get it to do simple things, having watched people fighting Net::Stomp to find out why messages were being dropped at random sizes and having seen just how many times people wanted to go to the pub after dealing with ActiveMQ, I've already chosen RabbitMQ again for my current project. Anyone here who used Spread recently? It is supposed to be faster, lower level, taking < 1% CPU. http://www.spread.org/ http://activemq.apache.org/how-does-activemq-compare-to-spread-toolkit.html http://jeremy.zawodny.com/blog/archives/010511.html -- Ruud
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On Wed, May 25, 2011 at 08:19:07AM +0100, James Laver wrote: > You mean apart from RabbitMQ? Rabbit is also the most full-featured of any of > the free message queues I've come across. Apart from HA, mesh networks and clustering. Rabbit is no where near prime time for that yet. > My personal experience of using ActiveMQ is "don't: it'll cause you > headache after headache". I don't do much with Net::STOMP (but it works for my basic cases) but we do quite a lot with multi-dc resilient broker networks with clients in Ruby and Java (and some perl) that RabbitMQ can't do using ActiveMQ. If you have multiple locations or need solid HA modern ActiveMQ versions are pretty much the best solution at the moment. Plus most companies have more in-house JVM tuning skills than they do Erlang runtime. Dean -- Dean Wilson http://www.unixdaemon.net @unixdaemon http://www.puppetcookbook.com @puppetcookbook
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 25 May 2011, at 03:57, Toby Wintermute wrote: > > Quickly updating this -- further investigation shows that ActiveMQ's > behaviour is to take any ACK to mean you're acknowledging everything > sent so far. If you don't ACK anything, or the recent things, then > it'll happily store and resend them. Just like TCP :) All this means is that you need to separate the concern of queue management from the concern of queue processing.
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 24 May 2011, at 06:31, Daniel Pittman wrote: > It is, presently, the best of the options out there in terms of scale You mean apart from RabbitMQ? Rabbit is also the most full-featured of any of the free message queues I've come across. My personal experience of using ActiveMQ is "don't: it'll cause you headache after headache". The same could be said of RabbitMQ, but we were doing some fairly tediously complicated stuff there (stuff that I believe would be impossible under ActiveMQ). Having watched people fight ActiveMQ trying to get it to do simple things, having watched people fighting Net::Stomp to find out why messages were being dropped at random sizes and having seen just how many times people wanted to go to the pub after dealing with ActiveMQ, I've already chosen RabbitMQ again for my current project. /j
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 24 May 2011 19:53, Toby Wintermute wrote: > On 24 May 2011 15:18, Toby Wintermute wrote: >> On 23 May 2011 21:07, Peter Edwards wrote: >>> Back to fighting with ActiveMQ. Feh. >> >> By the way.. how are you finding ActiveMQ, especially when interacting >> with it from Perl? > > Answering my own question a bit here, but I have now used ActiveMQ > with Perl via the STOMP interface to ActiveMQ. > > 1) I've already had to file two bugs with patches to AnyEvent::STOMP, > as it simply didn't work at all. This is possibly due to a stricter > parsing of STOMP by ActiveMQ than others.. I'm not sure. > It doesn't give me much confidence that anyone else is working with > both ActiveMQ and this CPAN module though! > > 2) The ACK system seems to be ignored by ActiveMQ. > I set "ACK"s to be manual, and then I am only sending ACKs for 50% of > the messages I receive. > I am setting the Receipt header in messages I send. > > So, I expect that either > a) I will not receipt RECEIPTs for things I didn't ACK, or > b) The broker will resend messages that I failed to ACK. > > Neither of these happen. > I receive RECEIPTs for every message sent, and un-ACKed messages are > not requeued. > > Is that meant to happen? Quickly updating this -- further investigation shows that ActiveMQ's behaviour is to take any ACK to mean you're acknowledging everything sent so far. If you don't ACK anything, or the recent things, then it'll happily store and resend them. So if I receive 10 messages, and sent ACKs for none except the last, then all are considered good by activemq. Nothing will be requeued. If I receive 10 messages, and send ACKs for, say, #8, but not #9 or #10, then activemq will consider the first 8 as good, and requeue and resend the ninth and tenth. So if you're using ACKs to indicate that you haven't totally crashed while processing a message, then the system will work as you expect.. if you crash and reconnect, you'll receive the last message again. This is a bit annoying if you're writing an asynchronous consumer with AnyEvent::STOMP though, because you could conceivably receive 3 messages and be processing them simultaneously.. finish #3 first and ACK it, then crash out before finishing #1 and #2. But because you ACKed #3, ActiveMQ considers 1 and 2 as fine too :( Note that I am a newbie at AMQ so it's possible there's some sort of option floating around to change this that I just haven't found yet.. -Toby
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
Actually we are running ActiveMQ 5.4.0 in production although latest is 5.4.2 on that branch and 5.5.0 is now also available. http://activemq.apache.org/ https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12311210&version=12315623 Yet more "Java leaking memory" bugfixes by the look of it :-) -Peter
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 24 May 2011, at 06:18, Toby Wintermute wrote: > On 23 May 2011 21:07, Peter Edwards wrote: >> Back to fighting with ActiveMQ. Feh. > By the way.. how are you finding ActiveMQ, especially when interacting > with it from Perl? AFAICS, it seems to work best through the STOMP support. Net::Stomp is a bit funky, but I've wrapped it into something that doesn't have a multi-stage setup process and thus can be glued into Catalyst with Catalyst::Model::Adaptor. It sprouted serialisation and deserialisation of messages and tracing (set STOMP_TRACE in the environment to have the messages dumped). At some point I should tidy up the tests and docs and shove it on CPAN. Any idea what name I should give it? (Net::Stomp::Simple? It uses Moose so it's not terribly ::Lite.) > I've been recommended to use it for a project at work, but I don't > think the recommender has actually used it. Curious to know how it > works in the real world. Being software, it is of course hateful, but I've found it to be the least-worst of all of the message-queue systems I've tried. I made a Debian (therefore also Ubuntu) package for it which took a whole load of arsing around away.
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
> > By the way.. how are you finding ActiveMQ, especially when interacting > > with it from Perl? > > I'm using it from Net::Stomp which had issues but now (version 0.40 on) works fine for us. Historically, ActiveMQ had problems with memory leaks though 5.3 works fine for us. It is also the standard queue platform used more widely within the BBC. One thing to watch out for is if you use non-persistent queues and don't consume the messages eventually the server runs out of memory and at that point (rather bizarrely IMO) still accepts TCP/IP connections but blocks responding to requests. You do see messages in its log, though. IIRC some people have had problems with high volume sites when the message packet is over 2K and occasionally frames get dropped (so use a smaller payload). We're using it in a fairly simplistic way, I write to a queue to notify publishing events (start and finish) and we also wrote a node.js http://nodejs.org/ pub-sub hub which picks those up using https://github.com/benjaminws/stomp-js and re-broadcasts by cometd to web clients (in addition to handling other client-server notifications and messaging). Seems to work okay. The JS code with node turned out to be straightforward and easier to implement a FSA than say writing a Perl POE server. Regards, Peter http://perl.dragonstaff.co.uk
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 24 May 2011 15:18, Toby Wintermute wrote: > On 23 May 2011 21:07, Peter Edwards wrote: >> Back to fighting with ActiveMQ. Feh. > > By the way.. how are you finding ActiveMQ, especially when interacting > with it from Perl? Answering my own question a bit here, but I have now used ActiveMQ with Perl via the STOMP interface to ActiveMQ. 1) I've already had to file two bugs with patches to AnyEvent::STOMP, as it simply didn't work at all. This is possibly due to a stricter parsing of STOMP by ActiveMQ than others.. I'm not sure. It doesn't give me much confidence that anyone else is working with both ActiveMQ and this CPAN module though! 2) The ACK system seems to be ignored by ActiveMQ. I set "ACK"s to be manual, and then I am only sending ACKs for 50% of the messages I receive. I am setting the Receipt header in messages I send. So, I expect that either a) I will not receipt RECEIPTs for things I didn't ACK, or b) The broker will resend messages that I failed to ACK. Neither of these happen. I receive RECEIPTs for every message sent, and un-ACKed messages are not requeued. Is that meant to happen? Cheers, Toby
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On Mon, May 23, 2011 at 22:18, Toby Wintermute wrote: > On 23 May 2011 21:07, Peter Edwards wrote: >> Back to fighting with ActiveMQ. Feh. > > By the way.. how are you finding ActiveMQ, especially when interacting > with it from Perl? FWIW, we find it scales fairly efficiently and works robustly in the field for the use we make in MCollective. It is, presently, the best of the options out there in terms of scale – but if you don't need clustering, or super-efficient multi-site message routing, you might find that some of the simpler competitors are easier to approach. > I've been recommended to use it for a project at work, but I don't > think the recommender has actually used it. Curious to know how it > works in the real world. For anything message-bus you really need to ask "in the real world, for a load like X", for your own X. The number and size of messages, persistence, timeliness, and consumers can make a substantial difference to how well the tool works. For anything less than a few MQ, or a few thousand small (4-128K) messages a second, most of the options work just fine. Daniel -- ⎋ Puppet Labs Developer – http://puppetlabs.com ✉ Daniel Pittman ✆ Contact me via gtalk, email, or phone: +1 (503) 893-2285 ♲ Made with 100 percent post-consumer electrons
ActiveMQ (was Re: Devel::Cover with Moose?)
On 23 May 2011 21:07, Peter Edwards wrote: > Back to fighting with ActiveMQ. Feh. By the way.. how are you finding ActiveMQ, especially when interacting with it from Perl? I've been recommended to use it for a project at work, but I don't think the recommender has actually used it. Curious to know how it works in the real world. Cheers, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Devel::Cover with Moose?
On 24 May 2011 04:26, Paul Johnson wrote: > On Mon, May 23, 2011 at 06:27:22PM +1000, Toby Corkindale wrote: >> Is anyone else using Devel::Cover with Moose, much? >> I seem to get megabytes of warnings displayed even on trivial modules. >> It's trying to create MD5 digests of files containing the materialised >> functions created by __PACKAGE__->meta->make_immutable. >> >> For example: >> >> $ cat Foo.pm >> package Foo; >> use Moose; >> __PACKAGE__->meta->make_immutable; >> 1; >> >> $ perl -MDevel::Cover -e 'use Foo' >> >> I've filed a bug report, but I don't suppose there's just an easy workaround? > > I've read the bug report. I don't suppose there is, unless you count > piping the output through grep -v "Can't" as an easy workaround. It's > quite easy, at least. > > Is the problem just that there is a lot of noise, or are the results > incorrect in some way too? It's mainly the noise that bothers me. > The noise I can probably deal with. The problem is, I suppose, that > Moose is making up filenames and then when Devel::Cover takes it at its > word and goes to look in those files it finds that Moose was actually > telling porky pies, and so it complains about that. I suppose that's fair enough, albeit annoying. > Does Moose consistently format its made up filenames in some way that I > could recognise them? Is there any useful information in the made up > filenames or should I just ignore them completely? They appear to be package+function names, rather than file names - eg: My::Package::Name::new > I realise that this is not the appropriate list for this discussion, but > I'm not on any Moose lists and you didn't ask in perl-qa :-P I'm not on those lists either.. Sounds like I should probably join one of these days. -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Devel::Cover with Moose?
On Mon, May 23, 2011 at 06:27:22PM +1000, Toby Corkindale wrote: > Is anyone else using Devel::Cover with Moose, much? > I seem to get megabytes of warnings displayed even on trivial modules. > It's trying to create MD5 digests of files containing the materialised > functions created by __PACKAGE__->meta->make_immutable. > > For example: > > $ cat Foo.pm > package Foo; > use Moose; > __PACKAGE__->meta->make_immutable; > 1; > > $ perl -MDevel::Cover -e 'use Foo' > > I've filed a bug report, but I don't suppose there's just an easy workaround? I've read the bug report. I don't suppose there is, unless you count piping the output through grep -v "Can't" as an easy workaround. It's quite easy, at least. Is the problem just that there is a lot of noise, or are the results incorrect in some way too? The noise I can probably deal with. The problem is, I suppose, that Moose is making up filenames and then when Devel::Cover takes it at its word and goes to look in those files it finds that Moose was actually telling porky pies, and so it complains about that. Does Moose consistently format its made up filenames in some way that I could recognise them? Is there any useful information in the made up filenames or should I just ignore them completely? I realise that this is not the appropriate list for this discussion, but I'm not on any Moose lists and you didn't ask in perl-qa :-P -- Paul Johnson - p...@pjcj.net http://www.pjcj.net
Re: Devel::Cover with Moose?
After upgrading to latest NYTProf 4.06 I get fewer warnings 82% ... Source for Moose generated method (unknown origin) isn't available (/home/edwarp11/dev/generated method (unknown origin): No such file or directory) Then when I upgrade to latest Moose 2.0007 I get $ perl -d:NYTProf t.pl Couldn't load class (MooseX::StrictConstructor::Trait::Class) because: Undefined subroutine &namespace::autoclean::on_scope_end called at /home/edwarp11/perl_local1/lib/perl5/namespace/autoclean.pm line 57. Upgraded namespace::autoclean from 0.11 to 0.12 and $ perl -d:NYTProf.pl runs cleanly but when I run nytprofhtml I get 0% ... Eval::Closure::__ANON__[(eval 113)[/home/edwarp11/perl_local1/lib/perl5/Eval/Closure.pm:124]:26] has no caller subnames but a call count of 2 about 100 times > [edwarp11@wsfmas26 dev]$ perl -d:NYTProf t.pl > [edwarp11@wsfmas26 dev]$ nytprofhtml > Reading nytprof.out > Processing nytprof.out data > Writing sub reports to nytprof directory > 81% ... Unable to open '/home/edwarp11/dev/generated method (unknown > origin)' for reading: No such file or directory. > 99% ... Unable to open '/home/edwarp11/dev/accessor foo defined at T.pm' > for reading: No such file or directory. > >
Re: Devel::Cover with Moose?
On 23 May 2011 09:27, Toby Corkindale wrote: > Is anyone else using Devel::Cover with Moose, much? > I seem to get megabytes of warnings displayed even on trivial modules. > It's trying to create MD5 digests of files containing the materialised > functions created by __PACKAGE__->meta->make_immutable. > > Similar behaviour with NYTProf [edwarp11@wsfmas26 dev]$ perl -d:NYTProf t.pl [edwarp11@wsfmas26 dev]$ nytprofhtml Reading nytprof.out Processing nytprof.out data Writing sub reports to nytprof directory 81% ... Unable to open '/home/edwarp11/dev/generated method (unknown origin)' for reading: No such file or directory. 99% ... Unable to open '/home/edwarp11/dev/accessor foo defined at T.pm' for reading: No such file or directory. A bit of Googling turns up a comment July 2009 from Tim Bunce http://blog.woobling.org/2009/07/optimizing-mooses-startup.html then http://code.google.com/p/perl-devel-nytprof/source/browse/trunk/t/71-moose.t?spec=svn1374&r=1374 r1374<http://code.google.com/p/perl-devel-nytprof/source/detail?spec=svn1374&r=1374> by tim.bunce on Sep 30, 2010 Diff<http://code.google.com/p/perl-devel-nytprof/source/diff?spec=svn1374&r=1374&format=side&path=/trunk/t/71-moose.t&old_path=/trunk/t/71-moose.t&old=> Add preliminary (incomplete) test files for #line handling and Moose. I'd be interested in hearing if anyone has these tools playing together well. There's not much Moose in the production code here and what we have was added after our last round of NYTProf'ing so I haven't looked in detail at the reports it produces yet. Back to fighting with ActiveMQ. Feh. Regards, Peter http://perl.dragonstaff.co.uk
Devel::Cover with Moose?
Is anyone else using Devel::Cover with Moose, much? I seem to get megabytes of warnings displayed even on trivial modules. It's trying to create MD5 digests of files containing the materialised functions created by __PACKAGE__->meta->make_immutable. For example: $ cat Foo.pm package Foo; use Moose; __PACKAGE__->meta->make_immutable; 1; $ perl -MDevel::Cover -e 'use Foo' I've filed a bug report, but I don't suppose there's just an easy workaround? Cheers, Toby
