here is the link to a forum which states why i have to set the limit.
http://markmail.org/message/km7mlmj46jgfs3rx#query:+page:1+mid:5f7q3wj2htwajjof+state:results
excerpt from the link
"The problem is that CodedInputStream has internal counter of how many
bytes are read so far with the same object.
In my case, there are a lot of small messages saved in the same file.
I do not read them at once and therefore do not care about large
messages, limits. I am safe.
So, the problem can be easily solved by calling:
CodedInputStream input_stream(...);
input_stream.SetTotalBytesLimit(1e9, 9e8);
My use-case is really about storing extremely large number (up to 1e9)
of small messages ~ 10K each. "
My problem is same as above, so i will have to set the limits on coded
input object.
Regards,
Alok
On Jan 16, 10:26 am, alok <alok.jad...@gmail.com> wrote:
> I was actually doing that initially, but I kept getting error on
> "Maximum length for a message is reached" ( I dont have exact error
> string at the moment). This was because my input binary file is large
> and it reaches the limit for coded input very fast.
>
> I saw a post on the forum (or maybe on Stack Exchange) which suggested
> that i should create a new coded_input object for each message. I have
> to reset the limits for coded input object. user on that thread
> suggested that its easy to create and destroy coded_input object.
> These objects are not big.
>
> Anyways, I will try it again by resetting the limits on this object.
> But then, would this be casuing the slowness? I will try and let you
> know the results.
>
> Regards,
> Alok
>
> On Jan 16, 9:46 am, Daniel Wright <dwri...@google.com> wrote:
>
>
>
>
>
>
>
> > You're making a new CodedInputStream for each message -- I think that gives
> > very poor buffering behavior. You should just pass coded_input to
> > ReadAllMessages and keep reusing it.
>
> > Cheers
> > Daniel
>
> > On Sun, Jan 15, 2012 at 4:41 PM, alok <alok.jad...@gmail.com> wrote:
> > > Daniel,
>
> > > i am hoping that my code is incorrect but i am not sure what is wrong
> > > or what is really causing this slowness.
>
> > > @ Henner Zeller, sorry i forgot to include the object length in above
> > > example. I do store object length for each object. I dont have issues
> > > in reading all the objects. Code is working fine. I just want to make
> > > sure to be able to make the code run faster now.
>
> > > attaching my code here...
>
> > > File format is
>
> > > File header
> > > Record1, Record2, Record3
>
> > > Each record contains n objects of type defined in proto file. 1st
> > > object has header which contains the number of objects in each record.
>
> > > <code>
> > > proto file
>
> > > message HeaderMessage {
> > > required double timestamp = 1;
> > > required string ric_code = 2;
> > > required int32 count = 3;
> > > required int32 total_message_size = 4;
> > > }
>
> > > message QuoteMessage {
> > > enum Side {
> > > ASK = 0;
> > > BID = 1;
> > > }
> > > required Side type = 1;
> > > required int32 level = 2;
> > > optional double price = 3;
> > > optional int64 size = 4;
> > > optional int32 count = 5;
> > > optional HeaderMessage header = 6;
> > > }
>
> > > message CustomMessage {
> > > required string field_name = 1;
> > > required double value = 2;
> > > optional HeaderMessage header = 3;
> > > }
>
> > > message TradeMessage {
> > > optional double price = 1;
> > > optional int64 size = 2;
> > > optional int64 AccumulatedVolume = 3;
> > > optional HeaderMessage header = 4;
> > > }
>
> > > message AlphaMessage {
> > > required int32 level = 1;
> > > required double alpha = 2;
> > > optional double stddev = 3;
> > > optional HeaderMessage header = 4;
> > > }
>
> > > </code>
>
> > > <code>
> > > Reading records from binary file
>
> > > bool ReadNextRecord(CodedInputStream *coded_input,
> > > stdext::hash_set<std::string> instruments)
> > > {
> > > uint32 count, objtype, objlen;
> > > int i;
>
> > > int objectsread = 0;
> > > HeaderMessage *hMsg = NULL;
> > > TradeMessage tMsg;
> > > QuoteMessage qMsg;
> > > CustomMessage cMsg;
> > > AlphaMessage aMsg;
>
> > > while(1)
> > > {
> > > if(!coded_input->ReadLittleEndian32(&objtype)) {
> > > return false;
> > > }
> > > if(!coded_input->ReadLittleEndian32(&objlen)) {
> > > return false;
> > > }
> > > CodedInputStream::Limit lim =
> > > coded_input->PushLimit(objlen);
>
> > > switch(objtype)
> > > {
> > > case 2:
> > > qMsg.ParseFromCodedStream(coded_input);
> > > if(qMsg.has_header())
> > > {
> > > //hMsg =
> > > hMsg = new HeaderMessage();
> > > hMsg->Clear();
> > > hMsg->Swap(qMsg.mutable_header());
> > > }
> > > objectsread++;
> > > break;
>
> > > case 3:
> > > tMsg.ParseFromCodedStream(coded_input);
> > > if(tMsg.has_header())
> > > {
> > > //hMsg = tMsg.mutable_header();
> > > hMsg = new HeaderMessage();
> > > hMsg->Clear();
> > > hMsg->Swap(tMsg.mutable_header());
> > > }
> > > objectsread++;
> > > break;
>
> > > case 4:
> > > aMsg.ParseFromCodedStream(coded_input);
> > > if(aMsg.has_header())
> > > {
> > > //hMsg = aMsg.mutable_header();
> > > hMsg = new HeaderMessage();
> > > hMsg->Clear();
> > > hMsg->Swap(aMsg.mutable_header());
> > > }
> > > objectsread++;
> > > break;
>
> > > case 5:
> > > cMsg.ParseFromCodedStream(coded_input);
> > > if(cMsg.has_header())
> > > {
> > > //hMsg = cMsg.mutable_header();
> > > hMsg = new HeaderMessage();
> > > hMsg->Clear();
> > > hMsg->Swap(cMsg.mutable_header());
> > > }
> > > objectsread++;
> > > break;
>
> > > default:
> > > cout << "Invalid object type "<< objtype <<
> > > endl;
> > > return false;
> > > break;
> > > }
> > > coded_input->PopLimit(lim);
> > > if(objectsread == hMsg->count()) break;
> > > }
> > > return true;
> > > }
>
> > > void ReadAllMessages(ZeroCopyInputStream *raw_input,
> > > stdext::hash_set<std::string> instruments)
> > > {
> > > int item_count = 0;
> > > while(1)
> > > {
> > > CodedInputStream in(raw_input);
> > > if(!ReadNextRecord(&in, instruments))
> > > break;
> > > item_count++;
> > > }
> > > cout << "Finished reading file. Total "<<item_count<<" items
> > > read."<<endl;
> > > }
>
> > > int _tmain(int argc, _TCHAR* argv[])
> > > {
> > > GOOGLE_PROTOBUF_VERIFY_VERSION;
>
> > > ZeroCopyInputStream *raw_input;
> > > CodedInputStream *coded_input;
> > > stdext::hash_set<std::string> instruments;
>
> > > string filename = "S:/users/aaj/sandbox/tickdata/bin/hk/
> > > 2011/2011.01.04.bin";
> > > int fd = _open(filename.c_str(), _O_BINARY | O_RDONLY);
>
> > > if( fd == -1 )
> > > {
> > > printf( "Error opening the file. \n" );
> > > exit( 1 );
> > > }
>
> > > raw_input = new FileInputStream(fd);
> > > coded_input = new CodedInputStream(raw_input);
>
> > > uint32 magic_no;
>
> > > coded_input->ReadLittleEndian32(&magic_no);
>
> > > cout << "HEADER: " << "\t" << magic_no<<endl;
> > > cout << "Reading data objects.." << endl;
> > > delete coded_input;
> > > cout << td << '\n';
>
> > > ReadAllMessages(raw_input, instruments);
>
> > > cout << td << '\n';
>
> > > delete raw_input;
> > > _close(fd);
> > > google::protobuf::ShutdownProtobufLibrary();
>
> > > return 0;
> > > }
>
> > > </code>
>
> > > On Jan 14, 3:37 am, Henner Zeller <henner.zel...@googlemail.com>
> > > wrote:
> > > > On Fri, Jan 13, 2012 at 11:22, Daniel Wright <dwri...@google.com> wrote:
> > > > > It's extremely unlikely that text parsing is faster than binary
> > > parsing on
> > > > > pretty much any message. My guess is that there's something wrong in
> > > the
> > > > > way you're reading the binary file -- e.g. no buffering, or possibly a
> > > bug
> > > > > where you hand the protobuf library multiple messages concatenated
> > > together.
>
> > > > In particular, the
> > > > object type, object, object type object ..
> > > > doesn't seem to include headers that describe the length of the
> > > > following message, but such a separator is needed.
> > > > (http://code.google.com/apis/protocolbuffers/docs/techniques.html#stre..
> > > .)
>
> > > > > It'd be easier to comment if you post the code.
>
> > > > > Cheers
> > > > > Daniel
>
> > > > > On Fri, Jan 13, 2012 at 1:22 AM, alok
>
> ...
>
> read more »
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
