Re: reading one message at a time
thanks & regards On Oct 29, 2:40 am, "Kenton Varda" <[EMAIL PROTECTED]> wrote: > On Tue, Oct 28, 2008 at 6:35 AM, Moonstruck <[EMAIL PROTECTED]> wrote: > > > you mean we should write the file like this? > > (sizeof a message) | (serialized message) | (sizeof another message) > > | (another serialized message) || so on and so forth > > > while reading, we'd first read the message size, then read data with > > the specified size to a stream, after that I can get it parsed; > > > is that so ? > > Yes. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: reading one message at a time
On Tue, Oct 28, 2008 at 6:35 AM, Moonstruck <[EMAIL PROTECTED]> wrote: > > you mean we should write the file like this? > (sizeof a message) | (serialized message) | (sizeof another message) > | (another serialized message) || so on and so forth > > while reading, we'd first read the message size, then read data with > the specified size to a stream, after that I can get it parsed; > > is that so ? Yes. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: reading one message at a time
you mean we should write the file like this?
(sizeof a message) | (serialized message) | (sizeof another message)
| (another serialized message) || so on and so forth
while reading, we'd first read the message size, then read data with
the specified size to a stream, after that I can get it parsed;
is that so ?
On Oct 28, 5:36 am, "Kenton Varda" <[EMAIL PROTECTED]> wrote:
> The protocol buffer format expects you to remember where the message ends;
> it cannot figure out for itself. So, you need to write the size of each
> message to your file before you write the message itself.
>
>
>
> On Mon, Oct 27, 2008 at 11:42 AM, Amit Gupta <[EMAIL PROTECTED]> wrote:
>
> > I have message defined as
> > message Person
> > {
> > required int32 id;
> > }
>
> > and than, after protoc-compiling I dump 500 Million of such buffers
> > using a c++ application into a file.
>
> > Now, when I read it back using a different C++ application (and
> > ParseFromIoStream, I get an error "message too long and look at the
> > file ..."
>
> > My intended behavior is to read the pb-message one at a time, analyze
> > its content and than read next message. For some reason. when I
> > deseralize from iostream, pb is reading and deseralizing full content
> > of the file.
>
> > How can I make the deserialization to work online, one message at a
> > time from the IoStream.
>
> > My code is attached below.
>
> > Many Thanks, Amit
>
> > #include
> > #include
> > #include
> > #include
> > #include
> > #include "temp.pb.h"
>
> > using namespace std;
>
> > #define MAX 5
>
> > void write()
> > {
> > fstream output("out.db", ios::out | ios::trunc | ios::binary);
>
> > for(int num = 0; num < MAX; ++num)
> > {
> > Person p;
> > p.set_id(num);
> > p.SerializeToOstream(&output);
> > }
> > }
>
> > void read()
> > {
> > fstream input("out.db", ios::in | ios::binary);
>
> > int num = 0;
> > for(; num < MAX; ++num)
> > {
> > Person p;
> > p.ParseFromIstream(&input);
> > }
> > cout << endl << num << endl;
> > }
>
> > int main()
> > {
> > write();
> > //cout << "Done Writing";
> > read();
> > cout << "Done reading" << endl;
> > return 0;
> > }- Hide quoted text -
>
> - Show quoted text -
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: reading one message at a time
The protocol buffer format expects you to remember where the message ends;
it cannot figure out for itself. So, you need to write the size of each
message to your file before you write the message itself.
On Mon, Oct 27, 2008 at 11:42 AM, Amit Gupta <[EMAIL PROTECTED]> wrote:
>
> I have message defined as
> message Person
> {
> required int32 id;
> }
>
> and than, after protoc-compiling I dump 500 Million of such buffers
> using a c++ application into a file.
>
> Now, when I read it back using a different C++ application (and
> ParseFromIoStream, I get an error "message too long and look at the
> file ..."
>
> My intended behavior is to read the pb-message one at a time, analyze
> its content and than read next message. For some reason. when I
> deseralize from iostream, pb is reading and deseralizing full content
> of the file.
>
> How can I make the deserialization to work online, one message at a
> time from the IoStream.
>
> My code is attached below.
>
> Many Thanks, Amit
>
>
> #include
> #include
> #include
> #include
> #include
> #include "temp.pb.h"
>
> using namespace std;
>
> #define MAX 5
>
> void write()
> {
> fstream output("out.db", ios::out | ios::trunc | ios::binary);
>
> for(int num = 0; num < MAX; ++num)
> {
>Person p;
>p.set_id(num);
>p.SerializeToOstream(&output);
> }
> }
>
> void read()
> {
> fstream input("out.db", ios::in | ios::binary);
>
> int num = 0;
> for(; num < MAX; ++num)
> {
>Person p;
>p.ParseFromIstream(&input);
> }
> cout << endl << num << endl;
> }
>
> int main()
> {
> write();
> //cout << "Done Writing";
> read();
> cout << "Done reading" << endl;
> return 0;
> }
>
>
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
reading one message at a time
I have message defined as
message Person
{
required int32 id;
}
and than, after protoc-compiling I dump 500 Million of such buffers
using a c++ application into a file.
Now, when I read it back using a different C++ application (and
ParseFromIoStream, I get an error "message too long and look at the
file ..."
My intended behavior is to read the pb-message one at a time, analyze
its content and than read next message. For some reason. when I
deseralize from iostream, pb is reading and deseralizing full content
of the file.
How can I make the deserialization to work online, one message at a
time from the IoStream.
My code is attached below.
Many Thanks, Amit
#include
#include
#include
#include
#include
#include "temp.pb.h"
using namespace std;
#define MAX 5
void write()
{
fstream output("out.db", ios::out | ios::trunc | ios::binary);
for(int num = 0; num < MAX; ++num)
{
Person p;
p.set_id(num);
p.SerializeToOstream(&output);
}
}
void read()
{
fstream input("out.db", ios::in | ios::binary);
int num = 0;
for(; num < MAX; ++num)
{
Person p;
p.ParseFromIstream(&input);
}
cout << endl << num << endl;
}
int main()
{
write();
//cout << "Done Writing";
read();
cout << "Done reading" << endl;
return 0;
}
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
