Hello everyone,
I hope I can find some advise here.
I have C++ code that writes a number of protobuf messages to a compressed
size delimited stream like this (simplified):
FILE *ofile = fopen("myfile.bin.gz", "wb");
google::protobuf::io::FileOutputStream ostream(_fileno(ofile));
google::protobuf::io::GzipOutputStream zipstream(&ostream);
while (loop) {
google::protobuf::util::SerializeDelimitedToZeroCopyStream(my_msg,
zipstream);
}
This works fine. The files are written and I can read them back in in C++
with no issues.
Now I am trying to read them in Python and I'm having difficulties to
understand the structure of the files. Here's what I'm trying:
def read_messages(raw_data: bytes):
offset = 0
while offset < len(raw_data):
# Read the size (4 bytes, little-endian) and decode
size_bytes = raw_data[offset : offset + 4]
offset += 4
size, _ = _DecodeVarint(size_bytes, 0)
# This reads the correct size of the message (verified in C++)
message_data = raw_data[offset : offset + size]
offset += size
# This causes an "Error parsing message" exception at the first
message
msg = my_messages_protobuf.MyMessage()
msg.ParseFromString(message_data)
... and ...
with gzip.open( "myfile.bin.gz", "r") as f:
while True:
chunk = f.read(chunk_size)
if not chunk:
break;
read_messages(chunk)
Now, to clarify a bit, I have worked with protobuf for very long, although
not in Python. Yet much Python code already deserializes such messages that
come in elsewhere, so I assume the whole "setup Protobuf in Python" thing
is not an issue here. It should work.
Given the fact that _DecodeVarint() correctly reads the message size leads
me to believe the reading of the gzipped file is okay too.
Yet when I look at the raw buffer "message_data" it looks very different
than the raw message data looks in C++ when I use the debugger there. I
have no idea what could cause this difference.
Can anybody give me a hint on what could be wrong here?
Much appreciated,
Moose
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/ee182f66-c495-4a05-b6c5-6272151f46d1n%40googlegroups.com.
