Hi, can you maybe release this as a patch? I like to inegrate this in github. But I fear I might miss something if I try to fiddle out the changes by hand.
Thanks David Reyes Samblas Martinez wrote: > Sorry for the wait Thomas, > I was working to solve the broken pipe issue that stops the parser > when it finds an error. I have applied a quick and dirty workaround > using try-catch technique and now the process will not stop and just > skip the faulty article and keeps going :) it logs the faulty ones in > a text file (title and position) for posterior forensics, but my first > guesses in that is not a codification issue with utf8 is more an > unexpected formating tag the php parser don't know how to deal with > Actually parsing the german wikipedia with more than 1.3 million articles > > Count: 1043000 > Failing count: 2 > > and keeps going I supose we can sacrificate two articles for having > one milion available now :) > > as you requested I uploaded my working compiled tools[1] but without > any xml sources it's about 113Mb, but if you have a working tools on > your system you just have to change > host-tools/offline-renderer/ArticleParser.py by the attached on this > mail and you can forget to cry like a child that his ice cream has > fall to the floor when after more than 24h parsing hundred of thousand > articles pased the process you see this ugly python error backtrace > blablabla and not your desired file :) > > by the way the faultyarticles.txt is saved at same > host-tools/offline-renderer directory, (i'm too lazy to put a > parameter for change that and I hardcoded the name of the file , > yes... don't waste typing on correct that bad habit, I know) > > If you have curiosity of what articles on the german wiki are causing > troubles > on dewiki-latest-pages-articles.xml (date 2009-11-20) > > ~Storck Bicycle > 832673 > ~Musculus serratus posterior inferior > 857334 > > Regards I hope I will upload the German wikipedia on Sunday... and > will be available on Monday, sorry for the wait but my Asymmetric DSL > is very asymmetric and upload 1.5-2 Gb (expected file size) will take > a bunch of hours. > > For those than wants to compile his own , go for it :) the > Quickreference in the doc directory on the souce is all you need to > start working, just remember than if you have a 64 bit system you > will have to follow the 64 bits method to compile the tools, > > Regards > [1]http://tuxbrain.org/downloads/wikireader/wikireaderbinaries20091127_dsamblas_modified_trycatch.tar.bz2 > David Reyes Samblas Martinez > http://www.tuxbrain.com > Open ultraportable & embedded solutions > Openmoko, Openpandora, Arduino > Hey, watch out!!! There's a linux in your pocket!!! > > > > > 2009/11/27 Thomas HOCEDEZ <thomas.hoce...@free.fr>: >> Thomas HOCEDEZ a écrit : >>> >>> Hi DAvid, >>> >>> Can you share your scripts & configs to do the same in French (and >>> other >>> languages) ? >>> Thanks >>> >>> Thomas >>> >>> >> >> As the Mailing list seems to be broken (or users started hibernating for >> winter...) I find by myself the way to compile things step by step. >> I'm for now rendering the French Wikipedia. As it started a few minutes >> ago, >> the result will be availabel during the weekend (I hope). >> >> I'll also post the way I managed to do so ! (I'm at the office for now, >> and >> I'm leaving...) >> >> Regards to you all ! >> >> Thomas >> > _______________________________________________ > Openmoko community mailing list > community@lists.openmoko.org > http://lists.openmoko.org/mailman/listinfo/community > -- _______________________________________________ Openmoko community mailing list community@lists.openmoko.org http://lists.openmoko.org/mailman/listinfo/community
