The "absolute" was the char[] size, which I had set to ~1MB back in the day. Subsequent use of STL string type does support any memory-fitting string.
On Fri, Nov 27, 2015 at 3:24 PM Markus Krötzsch < mar...@semantic-mediawiki.org> wrote: > On 27.11.2015 15:22, Magnus Manske wrote: > > It was the "absolute terms" problem here ;-) > > But 3MB uncompressed string data does not seem to be so big in absolute > terms, or are you referring to something else (I got this number from > the long pages special)? Parsing a 3MB string may need some extra > memory, but the data you get in the end should not be much bigger than > the original string, or should it? > > Markus > > > > > On Fri, Nov 27, 2015 at 2:12 PM Markus Krötzsch > > <mar...@semantic-mediawiki.org <mailto:mar...@semantic-mediawiki.org>> > > wrote: > > > > On 25.11.2015 16:05, Lydia Pintscher wrote: > > > On Mon, Nov 23, 2015 at 10:54 PM, Magnus Manske > > > <magnusman...@googlemail.com > > <mailto:magnusman...@googlemail.com>> wrote: > > >> Well, my import code chokes on the last two JSON dumps (16th and > > 23rd). As > > >> it fails about half an hour or so in, debugging is ... > > inefficient. Unless > > >> there is something that has changed with the dump itself (new > > data type or > > >> so), and someone tells me, it will be quite some time (days, > > weeks) until I > > >> figure it out. > > > > > > To update everyone here as well: Magnus has been able to pinpoint > the > > > problem and fix the tools. They're catching up again. The issue > was > > > one the extremely big pages that have have recently been created > for > > > research papers: https://www.wikidata.org/wiki/Special:LongPages > > > > Thanks for explaining. This explains why we did not see any problems > or > > unusual behaviour in Wikidata Toolkit. I guess Java simply does not > care > > about how long pages are, as long as they are not very big in > absolute > > terms. > > > > Markus > > > > > > _______________________________________________ > > Wikidata mailing list > > Wikidata@lists.wikimedia.org <mailto:Wikidata@lists.wikimedia.org> > > https://lists.wikimedia.org/mailman/listinfo/wikidata > > > > > > > > _______________________________________________ > > Wikidata mailing list > > Wikidata@lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikidata > > > > > _______________________________________________ > Wikidata mailing list > Wikidata@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikidata >
_______________________________________________ Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
