Starman and Docker
Hi,
I have a problem with Starman when run inside a Docker container.
Well, rather, it runs just fine - but doesn't want to die neatly.
I can Ctrl-C the running docker container, or send it a SIGTERM or
SIGKILL, and the docker process returns me to a command prompt.. but
the container lives on!
To demonstrate, get a container with perl and starman installed[1],
and put this "hello.psgi" into the root:
my $app = sub {
return [
'200',
[ 'Content-type' => 'text/plain' ],
[ "Hello world" ]
];
};
Then launch it with:
docker run --rm -t -p 5000:5000 my_container starman /mini.psgi
Try requesting http://localhost:5000 and you'll get back "Hello world".
Now Ctrl-C the running docker. Wait a second, and try querying that
URL again -- still hello world!
"docker ps" will show the container is still running.
This is probably *not* a Starman bug, but I did wonder if anyone has
encountered this and come up with a neat solution?
Cheers,
Toby
Note 1 -- to do this, you probably want to run something like:
$ docker run -t -i ubuntu:trusty /bin/bash
# apt-get update
# apt-get -y install starman
(exit with Ctrl-D)
$ docker commit a1b2c3d4e5 my_container (replacing the
hexcode with the container id that was printed when you ran it just
before)
Re: Deploying perl code
On 29 July 2014 06:11, Paul Johnson wrote: > On Mon, Jul 28, 2014 at 07:51:12PM +0100, Robert Rothenberg wrote: >> On Thu, Jul 24, 2014 at 4:25 PM, David Cantrell >> wrote: >> >> > I'm looking for tools that will make it easy to go from a bunch of code >> > in a release branch on github to an updated bunch of servers, with >> > minimal downtime. If it matters we're using Debian. >> >> At $previous, we deployed (mainly bespoke) Catalyst apps using the >> following toolchain at a small shop: > > If I were designing an environment from scratch I would take a serious > look at docker. I've been using it recently for cpancover.com and, > whilst it still has some rough edges, I really like the concept of > compartmentalising functionality. It's probably not the whole solution > (and in this case in particular it may not be any part of the solution) > but I predict a strong future for docker. (Ha! What do I know?) +1 for docker. You can host your own "docker registry" which then allows you to push/pull docker images to it. Images with shared history will only need to send and store the differences. So, build a company base image that includes your usual libraries and configuration. Then for each app, commit a "Dockerfile" to the repo, which is instructions on how it builds itself into a docker image, using the company base image as the starting point. Then setup your continuous integration server to run "docker build" and "docker push" for every revision that passes its tests. Over on your application servers, get them to reboot or restart the services frequently, pulling updates from the docker registry as they go. (Perhaps using different tags for staging and production though) Achievement unlocked: Continuous delivery
Open/Free BSD users -- help needed to fix Test::PostgreSQL
Hi, I took over Test::PostgreSQL a while back. I still get a lot of test failures from OpenBSD and FreeBSD users on CPANTS, and I think it's simply down to the postgres binaries being installed in different locations. If you run either of those systems, would you be able to tell me the path to the executables? On most Linux distros, they are in /usr/lib/postgresql/$VERSION/bin Cheers, Toby
Re: Open/Free BSD users -- help needed to fix Test::PostgreSQL
On 11 August 2014 15:48, Randal L. Schwartz wrote: >>>>>> "Toby" == Toby Wintermute writes: > > Toby> I still get a lot of test failures from OpenBSD and FreeBSD users on > Toby> CPANTS, and I think it's simply down to the postgres binaries being > Toby> installed in different locations. > > Toby> If you run either of those systems, would you be able to tell me the > Toby> path to the executables? > > Toby> On most Linux distros, they are in > Toby> /usr/lib/postgresql/$VERSION/bin > > FreeBSD puts them in a sane place: /usr/local/bin/psql etc. > > As I recall, OpenBSD ditto. > > Why the heck would people put "bin" stuff under a *lib* directory? > That's just... insane. Well, you can have multiple versions installed at once that way. Then /usr/bin/ contains scripts that pass control to the appropriate version binary based upon environment variables.
Re: Open/Free BSD users -- help needed to fix Test::PostgreSQL
On 11 August 2014 16:21, Randal L. Schwartz wrote: >>>>>> "Toby" == Toby Wintermute writes: > >>> Why the heck would people put "bin" stuff under a *lib* directory? >>> That's just... insane. > > Toby> Well, you can have multiple versions installed at once that way. Then > Toby> /usr/bin/ contains scripts that pass control to the appropriate > Toby> version binary based upon environment variables. > > Yeah, but lib is for ... libraries. Not bins. *throws hands in air* I'm not trying to defend the concept! you'd have to ask the Debian developers what they were thinking. > Also, picking bins based on env is about as silly as writing > "#!/usr/bin/env perl". *My* env is not necessarily *yours*. Put the > right path there, please. Maybe, but you have to admit that it is handy to have all the different postgres instances (on varying ports and PG versions) all listed and managed by single commands. (On Debian/Ubuntu boxes, there are meta commands that operate over all the instances, which I imagine is why they are split out of the path and then proxied back in)
DateTime::TimeZone breaking under PAR?
Hmm, I encountered an odd problem with DateTime::TimeZone today. When using it via a PAR archive, it refused to determine the local timezone. (This was on a Debian Etch fresh install, albeit with upgraded PAR libraries, since the normal Etch ones are quite broken, and then confirmed on an Ubuntu Hardy setup as well.) The test script that fails: timetest.pl #!/usr/bin/perl use strict; use warnings; use DateTime; use DateTime::TimeZone; my $tz = DateTime::TimeZone->new( name => 'local' ); print "Success!\n"; Create the PAR with: pp -o datetime.par -M DateTime::Locale::en timetest.pl (That's on Etch.. If I'm on Hardy I need an extra: -M DateTime::TimeZone::Local::Unix ) Then execute it: parl datetime.par timetest.pl I get "Cannot determine local time zone", rather than "Success!", on both Etch and Hardy. DateTime versions: Etch - 0.35 Hardy - 0.42 and then 0.4501 DateTime::TimeZone versions: Etch - 0.42 Hardy - 0.7701 and then 0.8301 Any thoughts if I'm Doing It Wrong, before I submit to RT? Cheers, Toby
Re: DateTime::TimeZone breaking under PAR?
2008/12/23 Johan Lindström : > At 08:18 2008-12-22, Toby Wintermute wrote: >> >> When using it via a PAR archive, it refused to determine the local >> timezone. >> (This was on a Debian Etch fresh install, albeit with upgraded PAR >> libraries, since the normal Etch ones are quite broken, and then >> confirmed on an Ubuntu Hardy setup as well.) > > Usually when things don't work under PAR/PerlApp/etc it's dynamically loaded > modules that weren't packaged properly. > > I see you force DateTime::Locale::en to be packaged, presumably because it > blew up without it. But there may be other modules it needs too. Presumably > related to the time zone stuff. > > A super quick look in DateTime::TimeZone reveals a runtime require of > DateTime::TimeZone::* . > > What if you manually add them all to the PAR file? > > (I usually put use statements in the source for this, with a comment saying > it's for PAR). A couple of other people have said it's a failure of Module::ScanDeps to correctly identify all the required modules.. Some of which are attempted to be loaded via a string-eval.. It makes sense, I suppose, since the name of the module it attempts to load is generated procedurally.. but is a pain. Manually adding all the DateTime::TimeZone::* files (and while one is at it, the TZ::Locale::* files) did indeed fix the problem. However, this seems like a wider problem with using PAR files.. How do you detect undetected runtime requirements? Just install *every* requirement to a private INC, then PAR it in it's entirety? Thanks for the help, Toby
XML::LibXML and HTML (in >=v1.67)
Hi,
I've been using XML::LibXML in the back-end of rea-toys[1] to scrape a
certain website for a while now, but noticed it all broke down when I
upgraded XML::LibXML from 1.66 to 1.69, and after some quick testing I
narrowed the change down to being between version 1.66 and 1.67.
My first instinct was to write a test and create a RT ticket [2], but
it occurs to me now that maybe I'm just Doing It Wrong..
If you have a moment.. Does this look right to you?
# Assuming $html contains a fairly typical HTML or XHTML webpage..
my $parser = XML::LibXML->new;
my $doc = $parser->parse_html_string(
$html => { recover => 1, suppress_errors => 1 }
);
The problem occurs when the html contains (the commonly used) & symbol
within attributes, such as:
I know that really one should escape the ampersand in those
circumstances, however real-world web-pages rarely do this.. And this
behaviour was tolerated in XML::LibXML 1.66, just not subsequent
versions.. but eh, maybe it's just the way I'm calling the parser?
Alternatively.. what do YOU use to parse real-world websites that are
often not totally valid?
If you'd like to see the small standalone test, click through to the
RT ticket and it is attached there.
Cheers,
Toby
[1: git://github.com/TJC/rea-toys.git ]
[2: http://rt.cpan.org/Public/Bug/Display.html?id=44715 ]
Re: XML::LibXML and HTML (in >=v1.67)
2009/4/1 Tatsuhiko Miyagawa :
> On Wed, Apr 1, 2009 at 2:53 AM, Dave Cross wrote:
>>> I know that really one should escape the ampersand in those
>>> circumstances, however real-world web-pages rarely do this.. And this
>>> behaviour was tolerated in XML::LibXML 1.66, just not subsequent
>>> versions.. but eh, maybe it's just the way I'm calling the parser?
>>
>> Sounds like XML::LibXML has fixed a bug. XML parsers are supposed to throw
>> an exception when they encounter invalid XML.
>
> The method we're talking about here is parse_*html*, and libxml2
> continues parsing HTML with errors like this, and XML::LibXML has an
> option (recover=>1) not to choke on that:
>
> perldoc XML::LibXML::Parser
>
> Parsing HTML may cause problems, especially if the ampersand ('&') is
> used. Such links cause the parser to throw errors. In
> such cases libxml2 still parses the entire document as there was no
> error ... Such HTML documents should be parsed using the recover flag.
That is indeed what the POD docs say, and in version 1.66, the
behaviour matched the documentation.
In 1.67 to 1.69_2, the behaviour appears to differ -- ie. Such errors
are fatal, despite using the recover flag.
Re: XML::LibXML and HTML (in >=v1.67)
2009/4/1 Tatsuhiko Miyagawa : > On Tue, Mar 31, 2009 at 10:45 PM, Toby Wintermute wrote: >> The problem occurs when the html contains (the commonly used) & symbol >> within attributes, such as: >> >> >> I know that really one should escape the ampersand in those >> circumstances, however real-world web-pages rarely do this.. And this >> behaviour was tolerated in XML::LibXML 1.66, just not subsequent >> versions.. but eh, maybe it's just the way I'm calling the parser? > > XML::Liberal [1] exactly addresses issues like this, and it also got > broken with XML::LibXML 1.67 with its error format change but works > with 1.69_2 on CPAN. > >> Alternatively.. what do YOU use to parse real-world websites that are >> often not totally valid? > > I use my own Web::Scraper [2,3] to scrape stuff and it uses > HTML::TreeBuilder (and ::XPath) to build a DOM tree and runs XPath or > CSS selector against it. It's definitely slower than LibXML but can > deal with such broken HTML documents very well. If you really care > about performance there's also HTML::TreeBuilder::LibXML on github [4] > that is a drop-in replacement for H::TB::XPath but uses LibXML under > the hood. Thanks, Web::Scraper looks quite neat. However I want to avoid applications breaking on random CPAN module upgrades (as just happened with the XML::LibXML upgrade yesterday), so I might steer clear of it until it loses the big, bold warning about the API still being unstable. I'm sure you understand :) > Another option would be to filter out such XHTML errors with > HTML::Tidy before passing it to LibXML. It would be neat if you do > that cleanup only if libxml parsing fails even with recover_errors > etc. set. Hmm, that is an interesting idea. This particular company's website also tends to get other aspects of their HTML broken too, such as repeated id values, so maybe HTML::Tidy will help with that too. ta, Toby
Re: XML::LibXML and HTML (in >=v1.67)
2009/4/2 mirod : > Tatsuhiko Miyagawa wrote: >> >> On Tue, Mar 31, 2009 at 10:45 PM, Toby Wintermute >> wrote: >>> >>> The problem occurs when the html contains (the commonly used) & symbol >>> within attributes, such as: >>> [snip] > > Indeed when I tested the various ways to get XML from HTML, a couple of > years ago, I found that the best way was to go through HTML::TreeBuilder. It > managed to make sense, without choking, of more random web pages than both > tidy and XML::LibXML. > > The only problem I found was with tags like '' which gets output by > the as_XML method as '', which is not quite well-formed XML. > This doesn't prevent you from using XPath on it with > HTML::TreeBuilder::XPath though. > > So HTML::TreeBuilder::XPath, beyond being a shameless plug, is my preferred > way to process HTML while still being able to use XPath. Ah, I was hoping I wouldn't have to go and re-write the xpath queries as HTML::Tree look_down() queries.. So H-TB-XPath looks great :) Thanks! Toby
Re: How we see CVs
2009/4/4 Bradley Dean : > On Thu, Apr 02, 2009 at 11:25:27AM +0100, Clark, Nicholas wrote: >> > On Thu, Apr 02, 2009 at 10:44:25AM +0100, James Laver wrote: >> > [...] >> > > Provided there's a valid text/plain part, people tend not to have a >> > > problem. >> > >> > Does that include a text/plain whose content is "This is a HTML mail, >> > please >> > bend over and use Outlook to view it"? >> >> That's spam. :-) > > My favourite email was one in which: > 1. text/plain part of the mulipart/alternative email read: > "Your email client cannot read this email" > 2. Once mutt had dumped to text/html part to text the email read: > [email header] > [email footer] > 3. Looking at the source revealed an email made of two images > files. > > Now if that doesn't warrant some slightly grumpy but informative feedback > to the source I don't know what does. :) Especially given that the image > parts had pretty much just textual content. Wow, sounds like something that would have come out of a certain Clerkenwell-based "Digital marketing agency" I worked for some years ago.. Some people just can't be told, although it's sad that it's still so common from so many sources :(
PARing autobox has me stumped
For some reason PAR::Packer isn't remembering to packing autobox's modules, but only when run on the (production) debian 4.5 systems, or Win32 with Strawberry Perl. Has anyone else seen this behaviour? Here's a really minimal test case: - test.pl --- #!/usr/bin/perl use autobox; --- $ pp -p -o test.par test.pl $ unzip -l test.par | grep autobox On Ubuntu 8.10 and 9.04, that returns a list that includes autobox.pm and autobox.so. However on Debian Etch 4.0 & 4.5 and Strawberry Perl it returns nothing. Any thoughts from ones wiser than I? Cheers, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: PARing autobox has me stumped
2009/8/13 Chisel Wright : > On Wed, Aug 12, 2009 at 04:06:56PM +0100, Chisel Wright wrote: >> Just to confuse things further ... not for me: > > Of course I get different results after actually installing autobox: > > chi...@zombie:tmp$ unzip -l test.par > Archive: test.par > Length Date Time Name > > 0 08-12-09 17:40 lib/ > 0 08-12-09 17:40 script/ > 542 08-12-09 17:40 MANIFEST > 215 08-12-09 17:40 META.yml > 15452 08-12-09 17:40 lib/Cwd.pm > 629 08-12-09 17:40 lib/File/Spec.pm > 7749 08-12-09 17:40 lib/File/Spec/Unix.pm > 675 08-12-09 17:40 lib/Scope/Guard.pm > 0 11-05-08 03:36 lib/auto/Cwd/Cwd.bs > 13660 11-05-08 03:36 lib/auto/Cwd/Cwd.so > 272 08-12-09 17:40 script/main.pl > 29 08-12-09 17:40 script/test.pl > --- > 39223 12 files So if I understand that correctly, your Ubuntu 9.04 box returns different results to my Ubuntu 9.04 box. In fact, you're getting the same behaviour I only get on Debian 4 and Windows. That makes it sound like it's a module-version problem.. but damnit, I have tried hard to make sure all four boxes involved are all running the latest versions of the relevant modules :(
Re: PARing autobox has me stumped
2009/8/13 : > Quoting Chisel Wright : >> >> Of course I get different results after actually installing autobox: > > Ack! Is this normal for PAR? I've always avoided it due to suspicions of > Evil .so library problems and so on. It does do some black magic in order to get XS modules working (the .so files you mention), but in general it does seem to work.. mostly. The lack of reliability puts me off it a bit, but I have to admit it's very handy when I want to take an app quickly from a dev box to some servers, and I don't want to spend all wee building debian packages for the dependencies. -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Anyone hiring at the moment?
2009/9/22 Matt Follett : > On Mon, Sep 21, 2009 at 5:25 PM, Ask Bjørn Hansen wrote: >> On Sep 21, 2009, at 15:16, Abigail wrote: >>> Well, we (booking.com) are. But that unfortunally requires a relocation >>> to (or near) Amsterdam, and I can't imagine anyone willing to do that. >> >> At solfo.com we hire skilled Perl people once in a while too, but we're >> even farther away! (Los Angeles; although only half of us are there) :-) > > We are hiring a Perl programmer and aren't as far as LA, but sadly, are as > far as St. Louis, MO. > Hah, I call your ridiculously long international commute, and raise you! We're looking to hire a Perl developer, in Melbourne, AU. (Hey, come on, it's almost summer; doesn't anyone want to come and play in the sun?) -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Sending $US to the US
2009/10/7 Ovid : > So, in order to get married, I need my birth certificate. Texas requires, > since I live outside the US, that I mail all of my paperwork to them, along > with $27 US (not surprisingly, they don't accept pounds). That's about £17. > My bank charges £20 for the bank draft, plus the £17 face value. > > Anyone happen to know a less expensive way to send, via mail, US dollars to > the US, short of sending cash? If you sign up to XE.com's FX trading site, http://www.xe.com/fx/ then you can send them UK pounds via a regular bank transfer (BACS, it's like paying a bill via internet banking), and then they will post you a bank draft in the denominated currency of your choice. (They also allow you to have the money sent via electronic means too) I use these guys to xfer cash from the UK to Australia, have done so for the past few years, and it's always been fine.. Good exchange rates too. (I searched around a fair bit when I was emigrating) (PS. I'm not affiliated with them in any way, and I don't receive any benefits by mentioning them) Cheers, Toby
Sanity check - DateTime on Perl 5.12.1
Hi, I wondered if anyone else running Perl 5.12.1 (and 12.0 would be interesting too) could quickly check if they can build DateTime 0.5x and pass the unit tests? I have them failing on two machines, but they're very similar and I am worried I might have screwed something up elsewhere, since I have been playing with a lot of versions of Perl on said machines. My Perl: This is perl 5, version 12, subversion 1 (v5.12.1) built for x86_64-linux-thread-multi Thanks! Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Sanity check - DateTime on Perl 5.12.1
On 9 June 2010 18:27, Colin Campbell wrote: > On 09/06/10 06:00, Toby Wintermute wrote: >> >> Hi, >> I wondered if anyone else running Perl 5.12.1 (and 12.0 would be >> interesting too) could quickly check if they can build DateTime 0.5x >> and pass the unit tests? >> >> I have them failing on two machines, but they're very similar and I am >> worried I might have screwed something up elsewhere, since I have been >> playing with a lot of versions of Perl on said machines. >> >> My Perl: >> This is perl 5, version 12, subversion 1 (v5.12.1) built for >> x86_64-linux-thread-multi >> >> >> Thanks! >> Toby >> > No problems building or running tests DateTime 0.55 > My Perl: > This is perl 5, version 12, subversion 1 (v5.12.1) built for x86_64-linux Cheers. Do you know if it still works if you recompile Perl with -Dusethreads?
Re: Sanity check - DateTime on Perl 5.12.1
On 9 June 2010 17:31, Daniel Pittman wrote: > Toby Wintermute writes: > >> I wondered if anyone else running Perl 5.12.1 (and 12.0 would be interesting >> too) could quickly check if they can build DateTime 0.5x and pass the unit >> tests? > > I can't, but the CPAN Testers can and do succeed: > > http://matrix.cpantesters.org/?dist=DateTime%200.55;reports=1;os=linux;perl=5.12.1 Mmm, I had noticed they seemed to think it worked too. Next question is: Why is DateTime failing *for me* ;) The failing tests seem to have, as their underlying failure: Can't locate object method "_normalize_tai_seconds" via package "DateTime::Infinite::Future" Which is an XS method from DateTime.so.. Oddly, I now see that if I force install DateTime, I can then run the code from the tests successfully.. but running the actual tests fails.. seems like they aren't picking up the blib/arch/ bits. Odd. Thanks for the confirmation that it was something at my end :) -Toby
Re: Relocatable @INC puzzle
On 14 July 2010 03:39, Paul Makepeace wrote: > I might be misunderstanding the purpose of this... > > I just built perl 5.12.1 with relocatable @INC but when I relocate (mv > ~/perl5 ~/perl) the site_perl directories haven't moved, so CPAN > breaks. > > Compiled at Jul 12 2010 23:52:59 > �...@inc: > /Users/paulmakepeace/perl5/lib/site_perl/5.12.1/darwin-thread-multi-2level > /Users/paulmakepeace/perl5/lib/site_perl/5.12.1 > /Users/paulmakepeace/perl/lib/5.12.1/darwin-thread-multi-2level > /Users/paulmakepeace/perl/lib/5.12.1 > > Rationale: I'm hoping to be able to build a perl that's completely > self-contained that I can then bundle as part of an app/git checkout. > The various developers will all be working with this on in their own > home directories so this perl needs an @INC that's entirely relative. > > I can sort of see that there's an argument that moving the perl > shouldn't move the CPAN but wondering if there's an official line, or > this is a bug, or currently under discussion, and what the preferred > way to solve this is (local::lib?), ... I have been / still am working on a similar affair for the same purpose. I've hit similar weirdness with @INC not doing what I expected, but didn't resolve the problem. However I note that "perlbrew" http://search.cpan.org/dist/App-perlbrew/ has come along recently and can help. It doesn't build you a relocateable Perl at all, however it makes building a local Perl very easy. So instead of giving your devs an all-in-one Perl tarball, you can say "Run perlbrew and go and make a coffee", and by the time they come back they have their own local perl instance. It'll have hardcoded @INC still, but at least it's in their own directory. Cheers, Toby
SystemTap hooks in Perl
Hey all, I've been playing around with Perl's SystemTap integration (in core since 5.12.0), but I've been struggling to find any documentation on it. I've deduced enough to get info on subroutine entry/exit, but wondered what else was out there? Also - I wondered if it was possible to create user-defined points -- I gather it's possible for the DTrace implementation (see http://search.cpan.org/~chrisa/Devel-DTrace-Provider-0.02/lib/Devel/DTrace/Provider.pm) but that code seems to be too tightly bound to DTrace to work with SystemTap. Cheers, Toby PS. Have posted up my notes on getting it going under Ubuntu at: http://blog.dryft.net/2011/03/profiling-perl-with-systemtap-on-linux.html -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Flickr CPAN module
On 10 March 2011 08:27, Dave Hodgkinson wrote: > Which is preferred? > > There are a few by people I would trust so I'm confused! I'm rather dissatisfied by the current Flickr::API, so have been working on a new/forked version of it. I'm not quite ready to release yet, but perhaps you'd like to help? https://github.com/TJC/Flickr-API2 -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: SystemTap hooks in Perl
On 9 March 2011 01:01, Steve Mynott wrote: > On Tue, Mar 08, 2011 at 04:23:04PM +1100, Toby Wintermute typed: > >> I've been playing around with Perl's SystemTap integration (in core >> since 5.12.0), but I've been struggling to find any documentation on >> it. > > Maybe this is of some interest? > > <https://dgl.cx/2011/01/dtrace-and-perl> Ah, I'd seen that; it helped me get as far as I did with the blog post. I really feel like there should be *some* POD distributed with Perl about what dtrace/systemtap hooks it supports! It's funny to have the feature mentioned in INSTALL, but then nothing else, anywhere, about it.. thanks, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Flickr CPAN module
On 22 March 2011 05:59, Dave Hodgkinson wrote: > On 21 Mar 2011, at 02:17, Toby Wintermute wrote: >> On 10 March 2011 08:27, Dave Hodgkinson wrote: >>> Which is preferred? >>> >>> There are a few by people I would trust so I'm confused! >> >> I'm rather dissatisfied by the current Flickr::API, so have been >> working on a new/forked version of it. >> I'm not quite ready to release yet, but perhaps you'd like to help? >> >> https://github.com/TJC/Flickr-API2 > > I was toying with a use-case driven Flickr::API::Useful based around > things you actually want from it... I'd love to have something like that in Flickr::API2 :) ie. A way to ask for photos, and to be given back an iterator of Flickr::...::Photo objects, rather than the current method which is just a big hash of keys that vary depending on what objects you've been given. (Hey, but at least it's an improvement on the old CPAN module, which returned an XML::Parser::Lite::Tree document and expected you to know how to query Yet Another XML Module.) Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Flickr CPAN module
On 22 March 2011 19:20, Dave Hodgkinson wrote:
> On 22 Mar 2011, at 02:04, Toby Wintermute wrote:
>> On 22 March 2011 05:59, Dave Hodgkinson wrote:
>>> On 21 Mar 2011, at 02:17, Toby Wintermute wrote:
>>>> On 10 March 2011 08:27, Dave Hodgkinson wrote:
>>>>> Which is preferred?
>>>>>
>>>>> There are a few by people I would trust so I'm confused!
>>>>
>>>> I'm rather dissatisfied by the current Flickr::API, so have been
>>>> working on a new/forked version of it.
>>>> I'm not quite ready to release yet, but perhaps you'd like to help?
>>>>
>>>> https://github.com/TJC/Flickr-API2
>>>
>>> I was toying with a use-case driven Flickr::API::Useful based around
>>> things you actually want from it...
>>
>> I'd love to have something like that in Flickr::API2 :)
>
> As a subclass. or uses ::API2.
I was thinking in the API2:: namespace at least; something so that you
could do something like:
use Flickr::API2 ':foo';
my $f = Flickr::API2->new(...auth stuff...);
$f->getPhotos(tag => "cats")->foreach(sub { my $photo = shift; say
$photo->title });
>> ie. A way to ask for photos, and to be given back an iterator of
>> Flickr::...::Photo objects, rather than the current method which is
>> just a big hash of keys that vary depending on what objects you've
>> been given.
>>
>> (Hey, but at least it's an improvement on the old CPAN module, which
>> returned an XML::Parser::Lite::Tree document and expected you to know
>> how to query Yet Another XML Module.)
>
> What? That's how you do it?
The Flickr.com API used to just return XML data; the old Flickr::API
module basically just parsed it back to you verbatim, albeit via an
XML parsing module the author also wrote.
It didn't really make it easy to adopt the module for use.
The current F::API2 module uses the JSON version of the Flickr API,
and so can pass back a reasonably sane Perl structure.
Toby
--
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world
Re: Flickr CPAN module
On 22 March 2011 20:42, Kieren Diment wrote:
> On 22/03/2011, at 1:04 PM, Toby Wintermute wrote:
>> A way to ask for photos, and to be given back an iterator of
>> Flickr::...::Photo objects, rather than the current method which is
>> just a big hash of keys that vary depending on what objects you've
>> been given.
>
> Every time I've looked at evern $web->{'big name service provider's'} API the
> cognitive load has been horrendous, and because I haven't had a day or two
> for someone to pay me to get my head around it I've just given up. An
> iterator or array of Flickr::Photo objects sounds like the solution for this
> particular problem. Next on your list can you write up an Ebay::Simple
> module for us?
>
This has taken valuable time out of my Civilisation 5 playing tonight,
but I've updated Flickr::API2 with some helper methods and pushed them
to the github repo now.
I'll just have to conquer the world tomorrow instead. :)
The API with helper methods looks like this:
my $f = Flickr::API2->new({ ... });
$f->test->echo(foo => 'bar');
my @pics = $f->photos->search(
tags => 'kitten',
per_page => 10,
);
foreach my $pic (@pics) {
say "Title: " . $pic->title;
}
Not complete yet mind, but what do you think of stuff so far?
cheers,
Toby
--
Turning and turning in the widening gyre
The falcon cannot hear the falconer
Things fall apart; the center cannot hold
Mere anarchy is loosed upon the world
Re: Perl on a smartphone?
On 23 March 2011 10:56, gvim wrote: > Anyone running Perl on a smartphone? Android must be close by now but I > haven't ventured into this market yet. Would like bash 4 too. Make that Perl > + bash + vim and I'm happy to pay. There were some official Android packages to give you Perl and stuff on Android.. Can't remember what the thing was called, but search around for it. I think PJF demoed it a while back. T -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Flickr CPAN module
Just a quick mention, since a few people were interested earlier.. I've uploaded the first version to CPAN today, as Flickr::API2 It could use some more, many more Perl helper classes, but the most important few are done, and more will follow. Maybe in the current state it will attract more patches, as people fill in the blanks for the particular areas they want to see fleshed out more? Feedback and patches appreciated. Cheers, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Best practice for unit tests that rely on internet access?
What is the best practice for writing unit tests that rely on internet access? ie. You have a CPAN module which is all about talking to a web service.. So you have unit tests that do just that. What if the system testing it doesn't allow HTTP connections out? The tests will fail.. Should one a) Prompt the user for whether to skip the tests? (I hate this option.. I dislike interactivity in installers) b) Only run the tests if specifically set by AUTHOR_TEST or something, otherwise mock the server? (I dislike this.. means few good tests run for users) c) Try to test if there's a working connection, and silently skip the tests if not? (Risks skipping tests if the connect fails for other reasons than no outbound HTTP allowed) d) ??? Cheers, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Best practice for unit tests that rely on internet access?
On 1 May 2011 08:34, David Precious wrote: > On Friday 29 April 2011 08:33:47 Leon Brocard wrote: >> If the module is all about testing a live service then by all means test >> it. Unless it takes too long, or costs money, or might change in the >> future when you don't have time to update the module... > > Arguably, if whatever service the module talks to changes in the future in a > way which stops the module working, failing test reports helps to highlight > that; even if the developer doesn't have time to fix it, it highlights to > potential users that the module "doesn't work" any more. > > That, to me, is a good reason to offer the ability to test against the real > service. I agree with this; if the web service changes enough to break compatibility with my module, then a slew of FAIL reports will make this clear to myself, and users. > On the other hand, you could end up with incorrect failure reports when the > service is temporarily unavailable, or cannot be reached due to a restrictive > firewall, for instance. Having to "force install" because the tests fail > because the firewall hasn't been updated to allow connections yet isn't too > pretty. Also true. Hmm :/ -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Best practice for unit tests that rely on internet access?
On 1 May 2011 19:14, Dave Hodgkinson wrote: > Isn't is traditional in an interactive CPAN shell session to ask if people > want to connect to the internet and run the real tests? It seems to be traditional, but I don't like it and hope it isn't best practice. I hate it when I ask CPANPLUS to install a module, and it goes off to install the hundred-odd sub-dependencies.. and then I go to lunch, and come back to find CPANPLUS paused after 5 modules on one that is asking a question I don't care about. > And as long as it installs quickly and cleanly under cpanm then the rest > of the world is happy too... :)
Re: Best practice for unit tests that rely on internet access?
Hmm, so using LWP::Online, and putting a sensible minimum version of Perl in my Makefile (5.8.8) have removed most of the failures.. but I'm still getting occasional automated test failure reports that confuse me.. The tests have passed some API calls, so must be online, but then fail on a specific call. Always the same bit fails, when it fails - but mostly it passes for other people. Annoying - I suspect the keys used for the unit tests might be getting throttled at the service or something, so if enough automated tests are run close together, it just fails some things? Damn :/ On 3 May 2011 23:09, David Cantrell wrote: > On Fri, Apr 29, 2011 at 11:15:58AM +1000, Toby Wintermute wrote: >> What is the best practice for writing unit tests that rely on internet >> access? >> >> ie. You have a CPAN module which is all about talking to a web >> service.. So you have unit tests that do just that. >> What if the system testing it doesn't allow HTTP connections out? The >> tests will fail.. > > Skip 'em if you can't even connect - or if the user doesn't provide an > API key or whatever that the tests need. > >> a) Prompt the user for whether to skip the tests? (I hate this >> option.. I dislike interactivity in installers) > > Interactivity is fine, provided it has sensible defaults. > >> b) Only run the tests if specifically set by AUTHOR_TEST or something, >> otherwise mock the server? (I dislike this.. means few good tests run >> for users) >> c) Try to test if there's a working connection, and silently skip the >> tests if not? (Risks skipping tests if the connect fails for other >> reasons than no outbound HTTP allowed) > > You could always check whether outbound HTTP is allowed by connecting to > somewhere entirely different. Try penthouse or thepiratebay - > somewhere which is highly unlikely to have been explicitly whitelisted, > so is a good test of whether Generic Web Stuff can be expected to work. > > -- > David Cantrell | A machine for turning tea into grumpiness > > It wouldn't hurt to think like a serial killer every so often. > Purely for purposes of prevention, of course. > -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Devel::Cover with Moose?
On 24 May 2011 04:26, Paul Johnson wrote: > On Mon, May 23, 2011 at 06:27:22PM +1000, Toby Corkindale wrote: >> Is anyone else using Devel::Cover with Moose, much? >> I seem to get megabytes of warnings displayed even on trivial modules. >> It's trying to create MD5 digests of files containing the materialised >> functions created by __PACKAGE__->meta->make_immutable. >> >> For example: >> >> $ cat Foo.pm >> package Foo; >> use Moose; >> __PACKAGE__->meta->make_immutable; >> 1; >> >> $ perl -MDevel::Cover -e 'use Foo' >> >> I've filed a bug report, but I don't suppose there's just an easy workaround? > > I've read the bug report. I don't suppose there is, unless you count > piping the output through grep -v "Can't" as an easy workaround. It's > quite easy, at least. > > Is the problem just that there is a lot of noise, or are the results > incorrect in some way too? It's mainly the noise that bothers me. > The noise I can probably deal with. The problem is, I suppose, that > Moose is making up filenames and then when Devel::Cover takes it at its > word and goes to look in those files it finds that Moose was actually > telling porky pies, and so it complains about that. I suppose that's fair enough, albeit annoying. > Does Moose consistently format its made up filenames in some way that I > could recognise them? Is there any useful information in the made up > filenames or should I just ignore them completely? They appear to be package+function names, rather than file names - eg: My::Package::Name::new > I realise that this is not the appropriate list for this discussion, but > I'm not on any Moose lists and you didn't ask in perl-qa :-P I'm not on those lists either.. Sounds like I should probably join one of these days. -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
ActiveMQ (was Re: Devel::Cover with Moose?)
On 23 May 2011 21:07, Peter Edwards wrote: > Back to fighting with ActiveMQ. Feh. By the way.. how are you finding ActiveMQ, especially when interacting with it from Perl? I've been recommended to use it for a project at work, but I don't think the recommender has actually used it. Curious to know how it works in the real world. Cheers, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: GTUG's
On 7 May 2011 22:15, Dave Hodgkinson wrote: > Anyone know anything about this? > > http://www.gtugs.org/ I joined up and went to a few GTUG meetings, but then they seemed to run out of content/presenters.. Or maybe the guy running it in Melbourne just lost interest or was too busy.. I don't know. What did you want to know about? -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 24 May 2011 15:18, Toby Wintermute wrote: > On 23 May 2011 21:07, Peter Edwards wrote: >> Back to fighting with ActiveMQ. Feh. > > By the way.. how are you finding ActiveMQ, especially when interacting > with it from Perl? Answering my own question a bit here, but I have now used ActiveMQ with Perl via the STOMP interface to ActiveMQ. 1) I've already had to file two bugs with patches to AnyEvent::STOMP, as it simply didn't work at all. This is possibly due to a stricter parsing of STOMP by ActiveMQ than others.. I'm not sure. It doesn't give me much confidence that anyone else is working with both ActiveMQ and this CPAN module though! 2) The ACK system seems to be ignored by ActiveMQ. I set "ACK"s to be manual, and then I am only sending ACKs for 50% of the messages I receive. I am setting the Receipt header in messages I send. So, I expect that either a) I will not receipt RECEIPTs for things I didn't ACK, or b) The broker will resend messages that I failed to ACK. Neither of these happen. I receive RECEIPTs for every message sent, and un-ACKed messages are not requeued. Is that meant to happen? Cheers, Toby
Re: GTUG's
On 25 May 2011 00:24, Dave Hodgkinson wrote: > On 24 May 2011, at 06:22, Toby Wintermute wrote: >> On 7 May 2011 22:15, Dave Hodgkinson wrote: >>> Anyone know anything about this? >>> >>> http://www.gtugs.org/ >> >> I joined up and went to a few GTUG meetings, but then they seemed to >> run out of content/presenters.. Or maybe the guy running it in >> Melbourne just lost interest or was too busy.. I don't know. >> >> What did you want to know about? > > More about networking. And ultimately having a source of talent to > draw on :) Yeah, I'd say it would be useful for networking. Although there can be a lot of people who are all about the "google reselling & support" rather than development.. but also people who are into development. See how you go? Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 24 May 2011 19:53, Toby Wintermute wrote: > On 24 May 2011 15:18, Toby Wintermute wrote: >> On 23 May 2011 21:07, Peter Edwards wrote: >>> Back to fighting with ActiveMQ. Feh. >> >> By the way.. how are you finding ActiveMQ, especially when interacting >> with it from Perl? > > Answering my own question a bit here, but I have now used ActiveMQ > with Perl via the STOMP interface to ActiveMQ. > > 1) I've already had to file two bugs with patches to AnyEvent::STOMP, > as it simply didn't work at all. This is possibly due to a stricter > parsing of STOMP by ActiveMQ than others.. I'm not sure. > It doesn't give me much confidence that anyone else is working with > both ActiveMQ and this CPAN module though! > > 2) The ACK system seems to be ignored by ActiveMQ. > I set "ACK"s to be manual, and then I am only sending ACKs for 50% of > the messages I receive. > I am setting the Receipt header in messages I send. > > So, I expect that either > a) I will not receipt RECEIPTs for things I didn't ACK, or > b) The broker will resend messages that I failed to ACK. > > Neither of these happen. > I receive RECEIPTs for every message sent, and un-ACKed messages are > not requeued. > > Is that meant to happen? Quickly updating this -- further investigation shows that ActiveMQ's behaviour is to take any ACK to mean you're acknowledging everything sent so far. If you don't ACK anything, or the recent things, then it'll happily store and resend them. So if I receive 10 messages, and sent ACKs for none except the last, then all are considered good by activemq. Nothing will be requeued. If I receive 10 messages, and send ACKs for, say, #8, but not #9 or #10, then activemq will consider the first 8 as good, and requeue and resend the ninth and tenth. So if you're using ACKs to indicate that you haven't totally crashed while processing a message, then the system will work as you expect.. if you crash and reconnect, you'll receive the last message again. This is a bit annoying if you're writing an asynchronous consumer with AnyEvent::STOMP though, because you could conceivably receive 3 messages and be processing them simultaneously.. finish #3 first and ACK it, then crash out before finishing #1 and #2. But because you ACKed #3, ActiveMQ considers 1 and 2 as fine too :( Note that I am a newbie at AMQ so it's possible there's some sort of option floating around to change this that I just haven't found yet.. -Toby
Re: ActiveMQ (was Re: Devel::Cover with Moose?)
On 27 May 2011 02:58, David Cantrell wrote: > On Thu, May 26, 2011 at 05:25:01PM +0100, Simon Wistow wrote: > >> Unless things have changed dramatically ActiveMQ has many, many features >> pretty much all poorly documented in the typical ASF/Java project >> fashion (i.e a poorly maintained wiki referencing multiple versions of >> the software implicitly, a collection of hard to navigate JIRA pages and >> masses of badly written Javadoc with no clear entry point). > > You're not talking about ActiveMQ you tease, you're talking about perl! > And Javascript, and exim, and BIND, and Linux, and ... Actually, I reckon a lot of Perl modules have quite good documentation compared to most of the Java modules I've looked into! The Perl community tends to start out with a SYNOPSIS section that includes a simple example of how to use the module, and that can be so helpful to get started. > Of course, with perl (and Javscript, BIND and Linux), if you want to > learn how to use the damned thing and how it all fits together, then > there are some splendid books you can buy and a community of users. > Maybe you can do the same with ActiveMQ. I came across this while looking for comparisons of other queues: http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes After reading through it, my take-away message was: All software sucks. RabbitMQ initially sounded quite good to me, btw, until I saw that message queue replication was still merely planned for future versions. Oh :( But perhaps it's better to have something that has less features but is reliable, than one that promises the moon, but will crash when I least expect it? (As people seem to imply with ActiveMQ) Thanks for all the input so far, guys. -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Speed v Version
On 1 June 2011 18:46, Dirk Koopman wrote: > I contemplating providing "encouragement" to a customer to upgrade from > 5.8.7 to something more modern. One of the overriding issues is "speed". The > customer is fixated with "speed". > > Unfortunately one of the major things the customer's clients do is > "replicate" their ISAM data into databases, usually MS-SQL via DBI and > DBD::ODBC. The ISAM data is always on Linux (and a few Unix) boxes. The > replication is a batch process that must complete overnight. Currently it is > a fairly close run thing. > > I don't suppose anyone has done any speed benchmarking on the various perls > to date? Still less on newer DBIs etc? All the customer's modules are circa > 2005. 1) Are they running old Red Hat or CentOS versions? I ask because the Perl shipped on those was, for quite a long time, very, very broken due to a vendor patch that made bless() take 1000x longer than it should. In that case, just using a non-broken Perl will be extremely faster. 2) Perl 5.10.x onwards are significantly faster when operating on modules which make heavy use certain types of inheritance. I seem to remember around ~20% performance improvements moving from 5.8.8 to 5.10.0, on a particular application. But that was probably a best-case (or worst case) scenario. 3) I've had a fair bit of experience at getting brought in to fix people's really-long-running Perl DB imports, and every time it comes down to optimising the database interactions, or *maybe* optimising some particularly stupid Perl code. As others have said - profile the code! If 90% of your time is spent in DBI::db::execute() then start looking at the SQL they're running.. Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Should I work in the US or the UK? - which pays best?
Just wanted to chip in and say -- why are you only considering the US or the UK? If you're looking for a nice English-speaking city which employs Perlmongers, have you considered Melbourne or Sydney in Australia? Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Perl threads and libwww wierdness
Hi, I'm hitting some really odd behaviour, infrequently, with libwww and mechanize under a highly-threaded Perl. Can I get a quick check to see if what I'm doing is known to work reliably for you? I have encountered a situation where I see unusual 404 errors - in between 0.03% to 0.10% of requests. Errors are randomly spaced on random pages, but over time the average amounts are quite consistent. Error rates initially increase with the number of simultaneous threads, but seem to top off at .1%. (ie. One in a thousand requests) The 404 errors are reported on the distant webserver as well, for URLs that are definitely not 404. (as the identical URL is being requested successfully many times in the same period). Scale: This is typically running around 40 threads, all going flat-out on an 8-core system; issues show up whenever you get over ~6 threads though. The only reason I don't think this is a problem with the network or webserver is that the problems don't show up if I use fork() instead of threads. (On otherwise identical code; and the same overall throughput rates are reached. However the fork() version is just for that bit of code for testing this; it misses some functionality.) It's also being a pain to try and replicate the threaded issue with a standalone server away from our code though, which isn't a good sign. Hence, looking for some confirmation of whether this might just be a known-bug with Perl or libwww before I go chasing down this rabbit hole for miles. :( This was running on Perl 5.14.1 and current versions of the above modules. (I don't think there are any bugfixes listed in 5.14.2 that would affect this issue? [1]) Any thoughts? Thanks! Toby [1 http://search.cpan.org/~flora/perl-5.14.2/pod/perldelta.pod ] -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Should I work in the US or the UK? - which pays best?
On 14 December 2011 12:17, Simon Wistow wrote: > On Wed, Dec 14, 2011 at 12:07:12PM +1100, Toby Wintermute said: >> Just wanted to chip in and say -- why are you only considering the US or the >> UK? >> If you're looking for a nice English-speaking city which employs >> Perlmongers, have you considered Melbourne or Sydney in Australia? > > Well, the original reason this thread kicked off, was I think, "Is it > more expensive to run a dev team in the US than in the UK?" > > My one comment on Oz is ... whilst I don't mind being 5500 miles away > from my family it's comforting to know I can be on a plane and back in > about 15 hours if something happens. > > There are far fewer flights back to the UK from Australia and they take > much longer. I think when I worked it out last time the average time for > me to get back to the UK if the flag went up, so to speak, was about 48 > hours. I've never tried looking at that, let's see: It's currently 1pm in Melbourne - if we said a flag went up right now then according to skyscanner.com.au, I could hop on a flight at 10:20pm this evening for $1800 and get to Heathrow at 11:35am tomorrow - 24 hours of travel, or 33 hours since the flag went up. Or for $4000 there's a flight that'd get me there at quarter past 5 am tomorrow, 28 hours after the 1pm flag. Yeah.. not great if you need a faster response time. > Not that that should stop you - the quality of life in Sydney and > Melbourne are astonishingly high - but it's a point to mention. Having lived in both London for a while and Melbourne, I have to say that Melbourne's quality of life is very nice, and the pay for a good Perl developer is higher than what I think I'd be getting in London at the moment. (At least, if I was working somewhere with comparable working conditions; I dare say there's more to be made in some of the crazy financial jobs.. but we have those here too if you're that way inclined.) -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Perl threads and libwww wierdness
On 14 December 2011 12:16, Toby Wintermute wrote: > Hi, > I'm hitting some really odd behaviour, infrequently, with libwww and > mechanize under a highly-threaded Perl. Worth noting - I'm pretty sure LWP is all pure-perl, but mechanize calls some XS libraries - HTML::Parser, PullParser and TokeParser. TC
Re: Perl threads and libwww wierdness
On 14 December 2011 19:44, Leo Lapworth wrote: > I still think it's your webserver See below.. > On 14 December 2011 01:16, Toby Wintermute wrote: >> I'm hitting some really odd behaviour, infrequently, with libwww and >> mechanize under a highly-threaded Perl. > >> I have encountered a situation where I see unusual 404 errors - in >> between 0.03% to 0.10% of requests. >> Errors are randomly spaced on random pages >> >> The 404 errors are reported on the distant webserver as well, for URLs >> that are definitely not 404. (as the identical URL is being requested >> successfully many times in the same period). > > My logic would be, if the webserver is reporting intermittent 404's for a > specific URL, then it's the webserver that's generating it. That would be my logic too, but (a) I can't work out why it would drop one in a thousand requests only, and (b) the non-threaded Perl scripts don't see ANY 404 errors. >> Scale: This is typically running around 40 threads, all going flat-out >> on an 8-core system; issues show up whenever you get over ~6 threads >> though. > > So the webserver can't cope with the traffic generated when you use > more than 6 threads, so too many requests. I start seeing errors appear before the webserver is getting fully loaded up though. The machine serving the requests is quite powerful, and there's only one machine making all the requests. >> The only reason I don't think this is a problem with the network or >> webserver is that the problems don't show up if I use fork() instead >> of threads. (On otherwise identical code; and the same overall >> throughput rates are reached. However the fork() version is just for >> that bit of code for testing this; it misses some functionality.) > > Does the fork() submit the same number of requests in the same > time period? - if it's less than your thread version then that > would point to the webserver, not the Perl code. That's the thing -- the fork-based version will generate just as many hits, but with zero 404 errors. So the webserver can handle that level of traffic. Eg. I can pull more than 260/sec off the webserver with either fork or thread based app. But the thread-based app starts seeing errors creep in once it gets over around 180-200/sec, and by 260/sec, it's up to 0.1% of all requests. (ie. once every few seconds) > I'd look into if your webserver (you don't mention what software it > is, apache/starman/IIS?) has some sort of "a, I can't cope, > throw a 404" setting or bug. We've now replicated the issue on both Apache 2.2 and nginx, and also against an older machine which coped with less overall hitrate, but still similar error rates. I'm just annoyed I haven't managed to replicate the issue on stand-alone code I can demonstrate on a desktop class machine just yet :/ Thanks, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Should I work in the US or the UK? - which pays best? (slight diversion)
On 15 December 2011 08:01, Kieren Diment wrote: > On 15/12/2011, at 2:52, Travis Basevi wrote: > >> On 14/12/2011 09:39, Kieren Diment wrote: >> >>> (I've got to concur with Toby btw. Australia is a pretty good option >>> in a lot of respects. Outside of Melbourne, Perl jobs (as opposed to >>> jobs with some perl) are a bit thin on the ground... >> >> Not that I'm necessarily in a hurry to move back just yet, but by "outside >> of Melbourne" do you mean in the areas just outside of Melbourne, or are you >> including the rest of Australia in that and implying that Melbourne is some >> sort of IT hot-bed? >> > > To my mind there seems to be much more Perl going on in Melbourne than > elsewhere (possibly the influence of Monash University). Although if you do a > search for perl on com.au a fair bit of stuff usually comes up. Melbourne.pm is the most active of the Australian perlmonger lists, and I believe the only one in Australia to manage regular meetings (almost) every month. Although maybe I'm just biassed because I've been managing those for the last couple of years :) Historically (ie. before my time) the Melbourne Perl mongers spawned the Open Source Developers Club, which has the associated yearly OSDC conference, which moves around Australia. So it does seem like Melbourne has been a hotbed of open-source development for some time now. It does feel like first-class Perl jobs (as opposed to sysadmin-with-perl or other-language-plus-perl) has dropped in popularity in Melbourne over time though; I think I work at one of the few development houses that still uses it exclusively for big projects. Most others that I know have migrated to Ruby or Java. :( Attendance at Perlmonger meetings is down a bit compared to four years ago, and it seems harder to rustle up people to give technical talks. What's the feeling in London and the US? It sounds like there's still some big places doing pure-Perl work? (net-a-porter? photobox?) Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Should I work in the US or the UK? - which pays best? (slight diversion)
On 15 December 2011 11:36, Toby Wintermute wrote: > On 15 December 2011 08:01, Kieren Diment wrote: >> On 15/12/2011, at 2:52, Travis Basevi wrote: >> >>> On 14/12/2011 09:39, Kieren Diment wrote: >>> >>>> (I've got to concur with Toby btw. Australia is a pretty good option >>>> in a lot of respects. Outside of Melbourne, Perl jobs (as opposed to >>>> jobs with some perl) are a bit thin on the ground... >>> >>> Not that I'm necessarily in a hurry to move back just yet, but by "outside >>> of Melbourne" do you mean in the areas just outside of Melbourne, or are >>> you including the rest of Australia in that and implying that Melbourne is >>> some sort of IT hot-bed? >>> >> >> To my mind there seems to be much more Perl going on in Melbourne than >> elsewhere (possibly the influence of Monash University). Although if you do >> a search for perl on com.au a fair bit of stuff usually comes up. > > > Melbourne.pm is the most active of the Australian perlmonger lists, > and I believe the only one in Australia to manage regular meetings > (almost) every month. Although maybe I'm just biassed because I've > been managing those for the last couple of years :) > > Historically (ie. before my time) the Melbourne Perl mongers spawned > the Open Source Developers Club, which has the associated yearly OSDC > conference, which moves around Australia. So it does seem like > Melbourne has been a hotbed of open-source development for some time > now. > > It does feel like first-class Perl jobs (as opposed to > sysadmin-with-perl or other-language-plus-perl) has dropped in > popularity in Melbourne over time though; I think I work at one of the > few development houses that still uses it exclusively for big > projects. Most others that I know have migrated to Ruby or Java. :( > Attendance at Perlmonger meetings is down a bit compared to four years > ago, and it seems harder to rustle up people to give technical talks. I thought about this some more, and feel I need to correct myself. There's at least three or four other places I know of that have been hiring for Perl projects in recent memory. It's just not everywhere is involved with the community, so they don't come to mind immediately.. and hiring has been at a low rate due to the general global economic situation.
Re: Should I work in the US or the UK? - which pays best? (slight diversion)
On 15 December 2011 17:25, Toby Wintermute wrote: > On 15 December 2011 11:36, Toby Wintermute wrote: >> On 15 December 2011 08:01, Kieren Diment wrote: >>> On 15/12/2011, at 2:52, Travis Basevi wrote: >>> >>>> On 14/12/2011 09:39, Kieren Diment wrote: >>>> >>>>> (I've got to concur with Toby btw. Australia is a pretty good option >>>>> in a lot of respects. Outside of Melbourne, Perl jobs (as opposed to >>>>> jobs with some perl) are a bit thin on the ground... >>>> >>>> Not that I'm necessarily in a hurry to move back just yet, but by "outside >>>> of Melbourne" do you mean in the areas just outside of Melbourne, or are >>>> you including the rest of Australia in that and implying that Melbourne is >>>> some sort of IT hot-bed? >>>> >>> >>> To my mind there seems to be much more Perl going on in Melbourne than >>> elsewhere (possibly the influence of Monash University). Although if you do >>> a search for perl on com.au a fair bit of stuff usually comes up. >> >> >> Melbourne.pm is the most active of the Australian perlmonger lists, >> and I believe the only one in Australia to manage regular meetings >> (almost) every month. Although maybe I'm just biassed because I've >> been managing those for the last couple of years :) >> >> Historically (ie. before my time) the Melbourne Perl mongers spawned >> the Open Source Developers Club, which has the associated yearly OSDC >> conference, which moves around Australia. So it does seem like >> Melbourne has been a hotbed of open-source development for some time >> now. >> >> It does feel like first-class Perl jobs (as opposed to >> sysadmin-with-perl or other-language-plus-perl) has dropped in >> popularity in Melbourne over time though; I think I work at one of the >> few development houses that still uses it exclusively for big >> projects. Most others that I know have migrated to Ruby or Java. :( >> Attendance at Perlmonger meetings is down a bit compared to four years >> ago, and it seems harder to rustle up people to give technical talks. > > I thought about this some more, and feel I need to correct myself. > There's at least three or four other places I know of that have been > hiring for Perl projects in recent memory. It's just not everywhere is > involved with the community, so they don't come to mind immediately.. > and hiring has been at a low rate due to the general global economic > situation. And after asking around at the PM meeting last night, it seems we have quite a lot of big places using Perl here. At least two banks, two telcos, and assorted bigger companies of which you'd recognise the name, plus bunches of start-ups and smaller places. I was glad to hear it :) -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Perl threads and libwww wierdness
2011/12/16 Peter Vereshagin : > Hello. > > 2011/12/14 12:16:04 +1100 Toby Wintermute => To > London.pm Perl M[ou]ngers : > TW> I'm hitting some really odd behaviour, infrequently, with libwww and > TW> mechanize under a highly-threaded Perl. > > TW> Error rates initially increase with the number of simultaneous > TW> threads, but seem to top off at .1%. (ie. One in a thousand requests) > > (i)Threaded perl5 ( 'use threads' ) doesn't seem to be recommended for > production environments. I know it certainly wasn't recommended back in the days of 5.6 or 5.8, but I thought things had improved since then.. > Forks are the standard IPC method for perl5. Event-driven stuff (AE/EV) is a > more advanced way, but for 40 parallel requests the forks are just ok. I'm not sure IPC means what you think it means - or I'm misunderstanding you - It's Inter-process communication, right? It'd say that forking, threading, and event-driven, are all methods of doing simultaneous processing - but don't specify anything about IPC. Which is the reason I was using threads - it's easy to do IPC between them by sharing access to some data structures. Whereas with forking you need to set up pipes, posix shared memory, sockets, or some other means. In the end, I bit the bullet yesterday and did rewrite the code to use fork(), and then used Net::STOMP::Client with RabbitMQ to perform the IPC. Total throughput is actually slightly faster than the threaded version, it uses quite a bit less memory, and critically, doesn't get any spurious errors. -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Perl threads and libwww wierdness
2011/12/16 Peter Vereshagin : > Hello. > > 2011/12/16 12:38:16 +1100 Toby Wintermute => To > London.pm Perl M[ou]ngers : > TW> > (i)Threaded perl5 ( 'use threads' ) doesn't seem to be recommended for > TW> > production environments. > TW> > TW> I know it certainly wasn't recommended back in the days of 5.6 or 5.8, > TW> but I thought things had improved since then.. > > They did. Perl6 was released and it seems to have threads those can be > recommended. > Perl5 have fork() that 'just works' and seems to be enough. So, you're saying that threads under perl5 is forever going to be considered broken and not worth touching then? :( Why do all the main distros ship a threading-enabled Perl? I note that Padre won't build without threads enabled either. > TW> > Forks are the standard IPC method for perl5. Event-driven stuff (AE/EV) > is a > TW> > more advanced way, but for 40 parallel requests the forks are just ok. > TW> > TW> I'm not sure IPC means what you think it means - or I'm > TW> misunderstanding you - It's Inter-process communication, right? > TW> It'd say that forking, threading, and event-driven, are all methods of > TW> doing simultaneous processing - but don't specify anything about IPC. > > > Right. I just use to avoid telling many words like 'method of doing > simultaneous processing' in favor of 'perlipc' in this case because: > > $ man perlipc | grep fork | wc -l > 37 > $ man perlipc | grep thread | wc -l > 4 > > and having several perl processes to communicate between is made just easier > by mean of fork(). How is fork() making it easier to communicate between processes? On the contrary, I say that using threads::shared makes it far more easier than fork() and rolling your own IPC. > TW> Which is the reason I was using threads - it's easy to do IPC between > > ... processes? (= > > TW> them by sharing access to some data structures. > TW> Whereas with forking you need to set up pipes, posix shared memory, > TW> sockets, or some other means. > > Memory-mapped file doesn't seem to be bad: > > https://metacpan.org/module/IPC::MMA As far as I can tell, that doesn't offer any way to perform synchronisation/condvars or suchlike - ie. where you can control the flow of one thread from another. > TW> In the end, I bit the bullet yesterday and did rewrite the code to use > TW> fork(), and then used Net::STOMP::Client with RabbitMQ to perform the > TW> IPC. > > Ouch. Isn't *MQ about to 'lose data under some circumstances' ? No, not really. What makes you think that? It depends entirely on how you configure and use it - you can have messages you want it to make sure you really, really don't lose - or you can have messages that are more transient. -Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: CRUDdy DBIC question
On 20 January 2012 01:00, Bob MacCallum wrote: > Sorry about the Perl question. > > We have a database model where the master copy of the data is file based. > > Is there some DBIx::Class magic which does some kind of nested > update_or_create_or_delete? For example, an object might initially be > written to the db along with its three children, but then someone > edits the file and removes one child, adds another, and edits an > existing child. > > I've seen http://search.cpan.org/~scain/DBIx-DBStag-0.12/DBIx/DBStag.pm > and stag-storenode.pl - if we convert our files into Stag format > temporarily, maybe this could work. Are there any other options I've > missed? Hi Bob, Have you looked at this module? It's a nested update, maybe you could extend it to cope with create+delete as well? (I note that DBIC already copes with nested create, at least) http://search.cpan.org/~jjnapiork/DBIx-Class-ResultSet-RecursiveUpdate-0.24/ tjc -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
[OT] Prepaid mobile plans with data, possibly roaming
Hello all, I'm making a whirlwind visit to London and bits of Europe soon, although with awfully poor timing, just missing YAPC:EU :( Hopefully I'll manage to run into a few of you while I'm in London though. The last time I visited, I picked up an Orange SIM that had "Unlimited data" on the plan.. only to discover they meant "unlimited, but at <5kb/sec". So this time around I thought I should get some local knowledge. Can you suggest a mobile carrier who has: * 3G support for prepaid SIMs * Has reasonably priced data plans * For bonus points -- will work while roaming in Germany. Although I suspect I'll just have to buy yet another SIM over there.. Alternatively.. I'd be happy to just get a data-only SIM and use Skype or similar the whole time. I hope this isn't too off-topic! Thanks, Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: [OT] Prepaid mobile plans with data, possibly roaming
On 21 August 2012 01:17, James Laver wrote: > On Mon, Aug 20, 2012 at 04:03:43PM +0100, Peter Corlett wrote: >> [0] The unlimit appears to be of the order of a few gigabytes. > > They're claiming truly unlimited actually. For an extra fiver they even > officially permit tethering. Are they likely to even notice if I'm tethering it, sans that fiver? (I've always wondered how on earth they could tell) (I'm on a Samsung Note w/Cyanogen 9, so there's not going to be any firmware tethering lockout going on here) Cheers for the advice, everyone! Now to see if I can get a 3 SIM dispatched to where I'll be staying first up in London.. :) Toby -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: [OT] Prepaid mobile plans with data, possibly roaming
On 22 August 2012 02:15, Anthony Lucas wrote: > 3 can be quite good with the traffic detection if it's obvious. > > In my experience, 5 or so hours of usage (unauthorised tethering) and they're > on to you. > Pop it back in your phone, reboot or lose the tower, and you're back in > business as far as internet access on the phone, > but you're now on some kind of watch-list and they're a lot faster to catch > on the second time. Ah, really? That's irritating that they bother.. I mean, I'm just using the same bandwidth quota regardless of method :/ Do Three block outbound ports? (ie. Can I just tunnel everything through an ssh or openvpn tunnel so that no traffic inspection can be done..) Toby
Re: [OT] Prepaid mobile plans with data, possibly roaming
On 23 August 2012 00:41, Anthony Lucas wrote: > They just want you to buy their tethering add-on and mifi products. It's probably simpler for me to just pay for the damn tethering add-on, but part of me is annoyed at the restriction and wishes to try to bypass it :) I'll try VPNing back to my shell account and see what happens.. can't be any slower than 3G already is.. TC
Re: [OT] Prepaid mobile plans with data, possibly roaming
On 23 August 2012 19:43, Peter Corlett wrote: > On 23 Aug 2012, at 07:22, Toby Wintermute wrote: > [...] >> I'll try VPNing back to my shell account and see what happens.. can't >> be any slower than 3G already is.. > > IME, this will make HTTP *faster* because it bypasses the bloody awful > transparent proxies that mobile telcos insist on using to mangle web traffic. > > It'll also stop said proxies from popping up the "we think you're trying to > visit a porn site, so we've blocked it as an excuse to try and sell you our > own soft porn service" page when you're visiting vile corrupting sites such > as camra.org.uk. Sounds like it's worth setting up OpenVPN on my mobile as well then! (I was thinking "tether mobile, then run vpn from laptop" but maybe I should go "openvpn from mobile, then tether it") Thanks for the all the advice so far :)
Re: CVE-2013-1667: important rehashing flaw
I note that while 5.16.3 is visible on CPAN, no-one seems to have updated perl.org yet - it still offers 5.16.2 as the latest release for download. On 5 March 2013 02:26, Nicholas Clark wrote: > Technically this is off topic: > > - Forwarded message from Ricardo Signes - > > Date: Mon, 4 Mar 2013 10:20:11 -0500 > From: Ricardo Signes > To: perl5-port...@perl.org > Subject: CVE-2013-1667: important rehashing flaw > User-Agent: Mutt/1.5.21 (2010-09-15) > > > The following message concerns a hash-related flaw in perl 5, which has been > assigned CVE-2013-1667. > > In order to prevent an algorithmic complexity attack against its hashing > mechanism, perl will sometimes recalculate keys and redistribute the contents > of a hash. This mechanism has made perl robust against attacks that have > been demonstrated against other systems. > > Research by Yves Orton has recently uncovered a flaw in the rehashing code > which can result in pathological behavior. This flaw could be exploited to > carry out a denial of service attack against code that uses arbitrary user > input as hash keys. > > Because using user-provided strings as hash keys is a very common operation, > we > urge users of perl to update their perl executable as soon as possible. > Updates to address this issue have bene pushed to main-5.8, maint-5.10, > maint-5.12, maint-5.14, and maint-5.16 branches today. Vendors* were informed > of this problem two weeks ago and are expected to be shipping updates today > (or > otherwise very soon). > > bleadperl is not affected. > > This issues affects all production versions of perl from 5.8.2 to 5.16.x. It > does not affect the upcoming perl 5.18. > > This issue has been assigned the identifier CVE-2013-1667. > > In the next few weeks, expect to see a more detailed post from researcher Yves > Orton or me. > > -- > rjbs > > > > - End forwarded message - > > > You will be wanting to be sure that this one is patched, either by your > vendor, or locally if you maintain your own build. The fix is under 40 lines, > most of which is *deleting* code and comments. > > If you know how to attack it, the results are pretty ugly, and pretty much > impossible to mitigate in user code. Right now, we don't think that anyone > *else* knows how to do it. You're only safe from DOS as long as this remains > the case. > > Nicholas Clark -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: New pet keeping rules in the Netherlands
That would be a Spider Wasp then. http://en.wikipedia.org/wiki/Spider_wasp Note that they attack and paralyse large spiders then drag them back to their nests. If you don't look closely (and why would you!) then you may think this is one creature, and come to the conclusion that giant spiders have now grown wings. This is truly the stuff nightmares are born from. So, umm, come to sunny Australia! On 21 June 2013 08:31, Kieren Diment wrote: > I did see a very large wasp dragging a huntsman corpse along around the side > of my house a few months ago. > > > On 21/06/2013, at 8:03, James Laver wrote: > >> On 20 Jun 2013, at 22:50, Kieren Diment wrote: >> >>> I particularly like it when they have babies and there are hundreds of tiny >>> cute huntsman spiders all over the ceiling. For a bonus, they eat >>> cockroaches. >> >> There was an old lady who swallowed a fly. >> >> Wait, what eats huntsmans again? >> >> James > -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: OT: Cheapo vps hosting
On 21 June 2013 22:54, Ben Tisdall wrote: > I'm moving to Germany and would like to maintain a UK IP address while > there, primarily to run a web proxy. I'd like to spend no more than 10 > GBP/mo; I don't care too much about uptime and not at all about the > data on the server, any of this lot set alarm bells ringing? I've been using Evorack to maintain a UK IP, and it sets me back a huge four quid a month for an Ubuntu LTS VPS. I haven't had any problems with them. -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: Living with smart match breakage
On 15 June 2013 09:03, Gordon Banner wrote: > Putting my hand up as one of the idiots... > > I've been using given/when for ages. My impression was that it was > announced as a new feature, "yay perl has a case statement (only better) at > last", and I piled in. Maybe I passed danger signals on the way, but not > consciously. Lots of books/presentations promoted it, and I don't remember > the small print. > > Marking it as experimental now seems to me to be rewriting history, what we > wish we'd said at the time. The 5.10 perlsyn page for example does not give > any warnings at all. > > I defend the option to change the language and remove broken stuff, but I'm > slightly peeved to be labelled a bleeding-edge cowboy for using something > that has been promoted widely for years. +1 I've appreciated using given/when with the saner end of smartmatching and after it's been around for six major versions of Perl, felt like it was something one could keep relying upon. By all means introduce a new smartERmatch, but it is annoying to have the old operator deprecated without even having the replacement ready yet. > Finally, the OP mentioned "convoluted boilerplate" and was offered > > no warnings 'experimental::smartmatch'; > # or > use experimental 'smartmatch'; > > If you think that's memorable and quick to type, your brain and fingers must > move faster than mine, or you program in an IDE. By the time you've knocked > that lot out you might as well type a bunch of elsifs anyway. Try putting either of those in a Perl version prior to 5.17! It just fails out at compile time. I reckon the convoluted boilerplate mentioned involves putting tests for perl versions around those, which is indeed convoluted. -T -- Turning and turning in the widening gyre The falcon cannot hear the falconer Things fall apart; the center cannot hold Mere anarchy is loosed upon the world
Re: OT: Cheapo vps hosting
On 4 July 2013 22:29, Dirk Koopman wrote: > On 04/07/13 11:24, Stanislaw Pusep wrote: >> >> There's a free VPN server hosted in UK: http://www.vpnbook.com/ >> Quite responsive; sponsored by NSA, maybe?! >> > > More likely by one of their local agents In all seriousness - that is quite possible. A hacker friend of mine swears there are some VPN providers who are fronts for such agencies; I guess the cost of running the VPN service is far outweighed by the amount of info you get to sniff. A similar trick worked for wikileaks early on, according to some rumours. (http://www.theregister.co.uk/2010/06/02/wikileaks_tor_snooping_denial/) T
