Hi, Dominique, Just took a quick look at the function that generates the min/max in the file -part.txt, it appears that we have neglected to deal with ibis::LONG and ibis::ULONG properly. The automatic generated description field for categories can be of unlimited length, which is the problem you have observed. Note that it automatically generate a new description if the existing one is an empty string or same as the name of the column. We are now limiting the automatically generated description to be no more than 200 characters long - there is some wiggle room in this limit, but it should not be off by a few characters.
The updated code has been checked in as SVN revision 472 if you are interested in trying it out. Thanks. John On 2/9/12 6:49 PM, Dominique Prunier wrote: > Hey John, > > Thanks for the answer. Indeed, my question was more about the impact on perf > than anything else. Knowing that everything can work without it makes me > wonder when it is necessary to have them. > The fact that the metadata is not updated is actually fortunate in my case > because it tends to generate extremely long descriptions for category column > which make very verbose warnings about line >2048 chars appearing each time > the table is opened. I'll look at this tomorrow. > > Thanks, > ________________________________________ > From: K. John Wu [[email protected]] > Sent: February-09-12 6:53 PM > To: FastBit Users > Cc: Dominique Prunier > Subject: Re: [FastBit-users] Min/max in -part.txt > > Hi, Dominique, > > The program ibis.cpp does go through the data one extra round to > explicit compute the min and max. The operation of updating the min > and max should be part of the normal index building procedure and I > have modified the code to reflect this intention. The modified > version should appear in the SVN shortly. > > In the mean time, this particular change will not affect the query > evaluation process. The queries should get the correct answers, and > the query answering time should not be any longer than if the min and > max are explicitly set in the metadata file. > > Thanks. > > John > > > > On 2/9/12 1:00 PM, Dominique Prunier wrote: >> Hi John, >> >> >> >> I’m currently finishing my tool to import data into FastBit and >> noticed a difference between running ibis –b and calling >> *int**fastbit_build_index*(*const**char**indexLocation, *const**char** >> cname,*const**char**indexOptions) from the C API. It seems that this >> method doesn’t rewrite the –part.txt file to insert min/max values in >> the column data (CATEGORY and LONG). Is it important ? Similarily, it >> doesn’t update partition header to set State and Timestamp, could it >> be a problem. >> >> >> >> My goal is to create a partition that will be read-only and accessed >> by multiple process, so i have to make sure that nothing is going to >> modify it afterwards. >> >> >> >> Thanks, >> >> >> >> */Dominique Prunier/**//* >> >> APG Lead Developper >> >> Logo-W4N-100dpi >> >> 4388, rue Saint-Denis >> >> Bureau 309 >> >> Montreal (Quebec) H2J 2L1 >> >> Tel. +1 514-842-6767 x310 >> >> Fax +1 514-842-3989 >> >> [email protected] <mailto:[email protected]> >> >> www.watch4net.com <http://www.watch4net.com/> >> >> / / >> >> /This message is for the designated recipient only and may contain >> privileged, proprietary, or otherwise private information. If you have >> received it in error, please notify the sender immediately and delete >> the original. Any other use of this electronic mail by you is prohibited. >> >> //Ce message est pour le récipiendaire désigné seulement et peut >> contenir des informations privilégiées, propriétaires ou autrement >> privées. Si vous l'avez reçu par erreur, S.V.P. avisez l'expéditeur >> immédiatement et effacez l'original. Toute autre utilisation de ce >> courrier électronique par vous est prohibée./// >> >> >> >> >> >> _______________________________________________ >> FastBit-users mailing list >> [email protected] >> https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users _______________________________________________ FastBit-users mailing list [email protected] https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/fastbit-users
