We hit problems with the newer Metabase code. It seems the latest version of Metabase has radically different internals, and isn't pulling back the reports as expected. However, it doesn't help that Amazon's SimpleDB has the most useless ordering mechanism I've ever had the misfortune to deal with. Apparently the following timestamps are in ascending order:
GUID [3140779c-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:44:48Z][2013-09-26T14:44:48Z].. stored.. cached GUID [cc212043-6e0e-1014-82b0-0d755be8bcae].. time [2013-09-27T13:37:20Z][2013-09-27T13:37:20Z].. stored.. cached GUID [320ef5ae-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:44:49Z][2013-09-26T14:44:49Z].. stored.. cached GUID [2fd58d66-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:44:45Z][2013-09-26T14:44:45Z].. stored.. cached GUID [ddc56792-2779-11e3-8a37-e2882d12fa71].. time [2013-09-27T13:36:51Z][2013-09-27T13:36:51Z].. stored.. cached GUID [336112a2-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:44:51Z][2013-09-26T14:44:51Z].. stored.. cached GUID [3fe51316-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:45:12Z][2013-09-26T14:45:12Z].. stored.. cached GUID [35d86012-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:44:56Z][2013-09-26T14:44:56Z].. stored.. cached GUID [38255816-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:44:59Z][2013-09-26T14:44:59Z].. stored.. cached GUID [3d2ccc72-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:45:08Z][2013-09-26T14:45:08Z].. stored.. cached GUID [45137f76-26ba-11e3-b8b2-96b61dda783b].. time [2013-09-26T14:45:21Z][2013-09-26T14:45:21Z].. stored.. cached GUID [de87e77c-2779-11e3-b8b2-96b61dda783b].. time [2013-09-27T13:36:52Z][2013-09-27T13:36:52Z].. stored.. cached We've been hit by this numerous times, and it is why there is so much code inside the Generator to try and cope with this. I look forward to when we can finally move to MongoDB :) Nearly all the cronjobs are switched back on, and I hope to have the Reports and Statistics websites running off the new server sometime today. Thanks, Barbie. -- Birmingham.pm - http://birmingham.pm.org CPAN Testers - http://cpantesters.org YAPC Surveys - http://yapc-surveys.org Perl Jam - http://perljam.info On Tue, Oct 1, 2013 at 8:06 AM, Barbie <bar...@missbarbell.co.uk> wrote: > After a bit of a hiccup this morning, it appears the pull process from the > Metabase is now working. I'm going to let it run unhindered this morning to > give it a good run at catching up. It's currently 4 days behind, but its > shooting through at the moment, so I expect it to be up to date in a few > days. I'll switch the builders on this afternoon or this evening, and > hopefully I can switch the websites over tonight too. > > Cheers, > Barbie. > > -- > Birmingham.pm - http://birmingham.pm.org > CPAN Testers - http://cpantesters.org > YAPC Surveys - http://yapc-surveys.org > Perl Jam - http://perljam.info > > > On Mon, Sep 30, 2013 at 8:58 PM, Barbie <bar...@missbarbell.co.uk> wrote: > >> Okay the field list on the cpanstats db completed, so I killed the >> repair, and recopied the metabase table. Now waiting for the field list to >> complete on the metabase db. Seems its not supposed to lock, but there is a >> bug in the current releases of MySQL where it does :( >> >> The reports pull is now queued, so as soon as the field list completes >> it'll be off and running. I'll be switching other cronjobs on once it >> starts processing. >> >> Cheers, >> Barbie. >> >> >> -- >> Birmingham.pm - http://birmingham.pm.org >> CPAN Testers - http://cpantesters.org >> YAPC Surveys - http://yapc-surveys.org >> Perl Jam - http://perljam.info >> >> >> On Mon, Sep 30, 2013 at 2:30 PM, Barbie <bar...@missbarbell.co.uk> wrote: >> >>> Not sure of the best way forward, so if anyone has suggestions, please >>> shout. >>> >>> The index broke because I started a mysql shell, without the -A option. >>> As a consequence it started a process to get the field list of the tables >>> in the metabase db. As it was taking over an hour I CTRL-C'ed it and killed >>> the process. Bad idea. I've now been running the repair table for over 14 >>> hours, as well as running a field list query on the cpanstats db that was >>> started by another mysql shell. >>> >>> The question is, is there a quick way to tell the mysql shell what the >>> field list is? Or is it not getting a field list but do something else, as >>> the state says 'checking table'. I'm loathed to stop anything at the >>> moment, as I don't know how long it will take, and whether it is near the >>> end anyway. With the split of the disks, I'm confident the reports pull and >>> builder scripts will catch up very quickly, so I'm willing to wait another >>> day or two. Failing that I'm going to have to write an alias for the >>> command to also add the -A. >>> >>> Does this field list actually lock any tables, as it appeared to when I >>> first started it, but some of the cronjobs I've restarted appear to be >>> fine. If it doesn't lock tables, I'll stop the repair, recopy the backup >>> metabase db, then try another mysql shell, and just let it run. >>> >>> All advice gratefully received. >>> >>> Thanks, >>> Barbie. >>> >>> >>> -- >>> Birmingham.pm - http://birmingham.pm.org >>> CPAN Testers - http://cpantesters.org >>> YAPC Surveys - http://yapc-surveys.org >>> Perl Jam - http://perljam.info >>> >>> >>> On Sun, Sep 29, 2013 at 10:05 PM, Barbie <bar...@missbarbell.co.uk>wrote: >>> >>>> I knew it was fateful to say it had gone well. I accidentally managed >>>> to break the indexing during a field list check :( Repairing table now, and >>>> also re-copying a fresh copy, just in case that would be quicker. >>>> >>>> More progress tomorrow. >>>> >>>> Barbie. >>>> >>>> -- >>>> Birmingham.pm - http://birmingham.pm.org >>>> CPAN Testers - http://cpantesters.org >>>> YAPC Surveys - http://yapc-surveys.org >>>> Perl Jam - http://perljam.info >>>> >>>> >>>> On Sun, Sep 29, 2013 at 6:54 PM, Barbie <bar...@missbarbell.co.uk>wrote: >>>> >>>>> Databases copied. Libraries (Perl & C) installed. Websites prepped. >>>>> Now starting to switch cronjobs back on. >>>>> >>>>> Might have some hiccups getting all the mailers working, as I'm sure I >>>>> had problems with them last time with reverse DNS. >>>>> >>>>> Prefs site may need to wait until later in the week, as I need to sort >>>>> out some new SSL certificates, both for the prefs.cpantesters.org and >>>>> admin.cpantesters.org. The current one is due to expire shortly, and >>>>> would rather use someone else than have to go through the hoops with our >>>>> current supplier. >>>>> >>>>> Thankfully so far all has gone well. However, please let me know if >>>>> you see any issues arising during the week. >>>>> >>>>> Thanks, >>>>> Barbie. >>>>> >>>>> >>>>> -- >>>>> Birmingham.pm - http://birmingham.pm.org >>>>> CPAN Testers - http://cpantesters.org >>>>> YAPC Surveys - http://yapc-surveys.org >>>>> Perl Jam - http://perljam.info >>>>> >>>>> >>>>> On Sun, Sep 29, 2013 at 1:03 AM, Barbie <bar...@missbarbell.co.uk>wrote: >>>>> >>>>>> ftp, rsync and web now running for: >>>>>> >>>>>> cpan.cpantesters.org >>>>>> backpan.cpantesters.org >>>>>> >>>>>> Databases still copying and not due to finish until the morning. >>>>>> >>>>>> Thanks, >>>>>> Barbie. >>>>>> >>>>>> -- >>>>>> Birmingham.pm - http://birmingham.pm.org >>>>>> CPAN Testers - http://cpantesters.org >>>>>> YAPC Surveys - http://yapc-surveys.org >>>>>> Perl Jam - http://perljam.info >>>>>> >>>>>> >>>>>> On Sat, Sep 28, 2013 at 9:21 AM, Barbie <bar...@missbarbell.co.uk>wrote: >>>>>> >>>>>>> Minor set back, the CPAN and BACKPAN mirrors were too big to fit on >>>>>>> the SSDs with the web pages, so having to copy them over to the HDDs. >>>>>>> Builders are up to date, only page requests getting added, rather than >>>>>>> reports, so they're ready to go. Once mirrors have been copied and >>>>>>> configured for the web/ftp/rsync, the sync from master will be >>>>>>> restarted. >>>>>>> Then the database will be next. >>>>>>> >>>>>>> Thanks, >>>>>>> Barbie. >>>>>>> >>>>>>> -- >>>>>>> Birmingham.pm - http://birmingham.pm.org >>>>>>> CPAN Testers - http://cpantesters.org >>>>>>> YAPC Surveys - http://yapc-surveys.org >>>>>>> Perl Jam - http://perljam.info >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 27, 2013 at 8:11 AM, Barbie <bar...@missbarbell.co.uk>wrote: >>>>>>> >>>>>>>> Just to let you know the reports pull has now stopped, with the >>>>>>>> last report pulled from 06:11 UTC. The builder will now catch up, and >>>>>>>> as >>>>>>>> soon as the DB is as complete as possible, that will be moved over. I'm >>>>>>>> trying to avoid downtime, but you won't see any new reports on the site >>>>>>>> until the processes are all switched on, on the new server. >>>>>>>> >>>>>>>> Likewise anyone trying to get database updates via the APIs or >>>>>>>> archived data will only get updates up to today. The same applies to >>>>>>>> the >>>>>>>> CPAN/BACKPAN mirrors, which are being copied over at the moment, and I >>>>>>>> aim >>>>>>>> to get these up and running on the new server first. Fingers crossed >>>>>>>> this >>>>>>>> could be this evening or tomorrow morning. >>>>>>>> >>>>>>>> I'll keep you updated of progress and let you know when processes >>>>>>>> are switched back on. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Barbie. >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Birmingham.pm - http://birmingham.pm.org >>>>>>>> CPAN Testers - http://cpantesters.org >>>>>>>> YAPC Surveys - http://yapc-surveys.org >>>>>>>> Perl Jam - http://perljam.info >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Sep 25, 2013 at 7:01 PM, Barbie >>>>>>>> <bar...@missbarbell.co.uk>wrote: >>>>>>>> >>>>>>>>> Hi folks, >>>>>>>>> >>>>>>>>> sorry for the sort notice, but the server upgrade will be >>>>>>>>> happening from Friday, and will continue over the weekend. I hadn't >>>>>>>>> quite >>>>>>>>> expected Bytemark to be so quick getting everything ready for us :) >>>>>>>>> >>>>>>>>> Further details on the blog - >>>>>>>>> http://blog.cpantesters.org/diary/167 >>>>>>>>> >>>>>>>>> Please pass on to anyone you know who might be affected. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Barbie. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Birmingham.pm - http://birmingham.pm.org >>>>>>>>> CPAN Testers - http://cpantesters.org >>>>>>>>> YAPC Surveys - http://yapc-surveys.org >>>>>>>>> Perl Jam - http://perljam.info >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
