error when using nested and packed in python

2009-07-02 Thread Dan

Hello protobuf gurus-
I'm just getting started with the python version of protobuf, using
v2.10. I want to store some lists of floats in a database's blob
field. When using a nested Message structure with the packed=true
option, I'm getting errors that look like this:

Traceback (most recent call last):
  File stdin, line 1, in module
  File build/bdist.linux-i686/egg/google/protobuf/message.py, line
160, in ParseFromString
  File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
line 1215, in MergeFromString
  File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
line 1075, in _DeserializeOneEntity
  File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
line 899, in _RecursivelyMerge
  File build/bdist.linux-i686/egg/google/protobuf/internal/
decoder.py, line 181, in ReadMessageInto
google.protobuf.message.DecodeError: Submessage told to deserialize
from 14-byte encoding, but used only 12 bytes

If I use a flat (not nested) message structure, I do not get these
errors. If I use a nested, packed=false approach, I do not get these
errors. The error happens only when both nested and packed.

Here is some code to reproduce the problem. Note that the
deserialization seems to succeed in putting the correct values into
the list (at least in this test), in spite of the error message.

my.proto file contains:

message TestNest {
optional ListData list_data = 1;
message ListData {
repeated float float_packed = 1 [packed=true];
repeated float float_notpacked = 2 [packed=false];
repeated int32 int_packed = 3 [packed=true];
repeated int32 int_notpacked = 4 [packed=false];
}
}

and then I run this set of commands for each of the fields
[float_packed, float_notpacked, int_notpacked, int_packed] and see
that the error happens only for the two fields that use packed=true.

import my_pb2
a = my_pb2.TestNest()
a.list_data.float_notpacked.extend([1,2,3])
a_string = a.SerializeToString()
a_deserialized = my_pb2.TestNest()
a_deserialized.ParseFromString(a_string)
str(a_deserialized)

I can think of three options to proceed, 2 of which I'm able to do
myself:
1) don't use the packed=true option
2) catch the errors and ignore them
3) if this is actually a bug, ask you gurus to fix the protobuf code

What do you recommend?
Dan

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---



Re: error when using nested and packed in python

2009-07-02 Thread Kenton Varda
Yeah, I think it's broken.
In reflection.py there is a line:
  return decoder.Position() - content_start

I think (untested) that it should actually be:
  return decoder.Position() - initial_position

I have no idea how this got past testing and I have asked the people
responsible to fix it.  Surprisingly you are the first to report this, so
maybe there's a mitigating factor that makes it unusual for the bug to harm
anything.  Sorry for the trouble.

On Thu, Jul 2, 2009 at 5:14 PM, Dan danbr...@gmail.com wrote:


 Hello protobuf gurus-
 I'm just getting started with the python version of protobuf, using
 v2.10. I want to store some lists of floats in a database's blob
 field. When using a nested Message structure with the packed=true
 option, I'm getting errors that look like this:

 Traceback (most recent call last):
  File stdin, line 1, in module
  File build/bdist.linux-i686/egg/google/protobuf/message.py, line
 160, in ParseFromString
  File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
 line 1215, in MergeFromString
  File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
 line 1075, in _DeserializeOneEntity
  File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
 line 899, in _RecursivelyMerge
  File build/bdist.linux-i686/egg/google/protobuf/internal/
 decoder.py, line 181, in ReadMessageInto
 google.protobuf.message.DecodeError: Submessage told to deserialize
 from 14-byte encoding, but used only 12 bytes

 If I use a flat (not nested) message structure, I do not get these
 errors. If I use a nested, packed=false approach, I do not get these
 errors. The error happens only when both nested and packed.

 Here is some code to reproduce the problem. Note that the
 deserialization seems to succeed in putting the correct values into
 the list (at least in this test), in spite of the error message.

 my.proto file contains:

 message TestNest {
optional ListData list_data = 1;
message ListData {
repeated float float_packed = 1 [packed=true];
repeated float float_notpacked = 2 [packed=false];
repeated int32 int_packed = 3 [packed=true];
repeated int32 int_notpacked = 4 [packed=false];
}
 }

 and then I run this set of commands for each of the fields
 [float_packed, float_notpacked, int_notpacked, int_packed] and see
 that the error happens only for the two fields that use packed=true.

 import my_pb2
 a = my_pb2.TestNest()
 a.list_data.float_notpacked.extend([1,2,3])
 a_string = a.SerializeToString()
 a_deserialized = my_pb2.TestNest()
 a_deserialized.ParseFromString(a_string)
 str(a_deserialized)

 I can think of three options to proceed, 2 of which I'm able to do
 myself:
 1) don't use the packed=true option
 2) catch the errors and ignore them
 3) if this is actually a bug, ask you gurus to fix the protobuf code

 What do you recommend?
 Dan

 


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---