Re: [protobuf] Using Protocol Buffers without generating code

[Jason Hsueh]

Protobufs reflection support allows you to do this. We don't have a
walkthrough, but you can create types dynamically using the Descriptors
classes along with DynamicMessage. You can build the definitions you want
into a FileDescriptorProto (along with any dependencies it may have) and
call Descriptors.FileDescriptor.buildFrom(). This will give you descriptor
objects that you can pass to DynamicMessage.

On Thu, Apr 14, 2011 at 10:50 AM, Robert wrote:

> Hi,
>
> I am new to the protocol buffers and I am really impressed of the
> encoding technique.
> I am just wondering if there is a way to use them without having to
> generate code from the .proto files.
>
> I would appreciate a way where you import or even generate the .proto
> files at runtime and then pass a java.util.Map where the library picks
> the properties from.
>
> Maybe there is already a way to do this, but I did not find it neither
> in the tutorials nor the faq.
>
> Thanks & regards,
> Robert
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Cannot deserialize protobuf data from C++ in Java

[Jason Hsueh]

On Fri, Apr 15, 2011 at 9:48 AM, platzhirsch
wrote:

> Right! I think I refactored the whole writing and reading, still I am
> stuck. However by this I get this Exception:
> com.google.protobuf.UninitializedMessageException: Message missing
> required fields: name
>
> In C++:
>
> Name name;
> name.set_name("platzhirsch");
>
> boost::asio::streambuf b;
> std::ostream os(&b);
>
> ZeroCopyOutputStream *raw_output = new OstreamOutputStream(&os);
> CodedOutputStream *coded_output = new CodedOutputStream(raw_output);
>
> coded_output->WriteLittleEndian32(name.ByteSize());
> name.SerializeToCodedStream(coded_output);
>

You'll need to destroy the CodedOutputStream, which will flush data that's
been buffered internally. We should probably make this clearer in the docs.

socket.send(b);
>
> In Java:
>
> NameProtos.Name name =
> NameProtos.Name.parseDelimitedFrom(socket.getInputStream());
> System.out.println(name.newBuilder().build().toString());
>
>
> ---
>
> I can't find out where the problem might be.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] abstract

[Jason Hsueh]

No, this is not possible with protobuf. The stock code generators provide
the basic fields and generated serialization code. If you don't want any of
that implemented, it sounds like you want a custom code generator to
translate the schema describe in the .proto file to the code you do want.

If you want type hierarchies like XSD schema gives you, we don't do that
either: search the mailing list archives for threads about inheritance for
some of the reasoning.

On Fri, Apr 15, 2011 at 8:47 AM, Lars Schouw  wrote:

> Can I specify that the message is abstract so I can implemented it at
> will on my client?
> This is possible in XSD schemas.
> Lars
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Cannot deserialize protobuf data from C++ in Java

[Jason Hsueh]

On Fri, Apr 15, 2011 at 2:06 AM, platzhirsch
wrote:

> My problem is to serialize protobuf data in C++ and deserialize the
> data in Java probably. In fact I get **InvalidProtobufferException**
> in Java.
>
> Here is my minimal example:
>
> My protobuf file looks like this:
>
>option java_outer_classname = "NameProtos";
>
>message Name {
>
>   required string name = 1;
>}
>
> Which I compile with:
>
>protoc -I=. --cpp_out=../cpp/ name.proto
>
>  and
>
>protoc -I=. --java_out=../java/ name.proto
>
>
> In C++ I create the name object this way:
>
>Name name;
>name.set_name("platzhirsch");
>socket.send(name.SerializeAsString);
>
> In Java I read from the socket, until the socket is closed
> (socket.send closes the connection, after it entirely wrote the string
> passed, so I guess here is no when stop reading issue, isn't there?).
>
> In Java I make the following call in order to deserialize:
>
> `NameProtos.Name name =
> NameProtos.Name.parseFrom(ByteString.copyFromUtf8(received))`;
>

The serialized data is binary and not valid UTF8. The conversion here
corrupts the data - make sure to parse the exact bytes that are serialized.


>
> However I always get **InvalidProtocolBufferException**
>
> I have no idea, the received strings are not empty though, they look
> like this:
> [SPACE]platzhirsch[A Character which cannot be displayed probably]
>
> These character which I cannot read, I also get, when using other
> fields like int32. I guess there are just encoded into the string.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: UnmodifiableLazyStringList is not Serializable

[Jason Hsueh]

Also don't have enough Java-fu, but I'd thought this was basically how this
was solved: the messages override some function that cause the JavaVM to
serialize using protobuf serialization. I was under the impression that the
message internals would then no longer needed to implement Serializable.

(Specifically I think the magic is in
http://code.google.com/apis/protocolbuffers/docs/reference/java/serialized-form.html#com.google.protobuf.GeneratedMessageLite
)

On Wed, Apr 13, 2011 at 5:11 PM, Henner Zeller  wrote:

> On Wed, Apr 13, 2011 at 16:01, Ben Wright  wrote:
> > I agree with J.S. on this one - there are many situations in Java EE
> > environments where "Serializable" is checked or java serialization
> > used when it's not simple or feasible to leverage protobuf
> > serialization.  Most of these situations are "invm / in memory"
> > transfers.  Sometimes java serialization is used to clone objects or
> > to safely pass them between ClassLoader scopes.
> >
> > I've run into this limitation with JBoss ESB and passing protocol
> > buffer objects between services through the "InVM" message transport.
>
> It is a couple of years that I've worked with Java, so my
> serialization fu might not be up-to-par. But would the problem be
> solved if the ProtocolBuffer object implements Externalizable ? That
> way it can be serialized via the Java VM, but it would use the
> protobuf specific serialization.
>
> -h
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] required field in extend

[Jason Hsueh]

The proto compiler actually allows you to do this, but it perhaps shouldn't
as it semantically does not make sense. You don't have any guarantees that
each application is aware of the required extension; therefore you can't
really say that a message is invalid if it doesn't have that extension set.
The generated code ends up treating the extension as optional;
person.IsInitialized() does not iterate over the known extensions to search
for required extensions and verify that those are set.

On Mon, Apr 4, 2011 at 2:31 PM, Aaron Rich  wrote:

> Newbie question:
>
> Can an extended message have a required field? For example, If I
> wanted to make any application using detailperson.proto's definition
> of a person have to have a middle name?
>
> person.proto:
> message person
> {
>  required string first_name =1;
>  required string last_name =2;
>
>  extensions 100 to 199;
> }
> -
> detailedperson.proto:
> extend person
> {
>  required string middle_name = 100;
> }
>
> Thanks.
>
> -Aaron
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: self including message array

[Jason Hsueh]

There is no explicit test for this, but the official c++ implementation
emits code that successfully compiles for this message:

message TestRepeatedRecursiveMessage {
  repeated TestRepeatedRecursiveMessage a = 1;
  optional int32 b = 2;
}

On Fri, Apr 1, 2011 at 7:52 AM, zad  wrote:

> Ok I'll do, I  asked to be sure if a recursive definition was
> "syntactically" correct.
> Can you please check if there is a test case or an example?
>
> On Apr 1, 4:44 pm, Jason Hsueh  wrote:
> > You appear to be using a third party implementation for pure C. You'll
> have
> > to check with the author of the implementation.
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Apr 1, 2011 at 7:25 AM, zad  wrote:
> > >  message resource_info {
> > >optional string device_name=18;
> > >optional Res_type res_type=19 [default=RAW];
> > >repeated resource_info resources=2;
> > > }
> >
> > > the above message generates the following .h
> >
> > > /*! \file deviceManager_hint.h
> > >  *  \brief message hints
> > >  */
> >
> > > /*--
> > >  * Generated 2011-04-01 15:14:14 +0100
> > >  *   by protoc99 version 0.17
> > >  * DO NOT EDIT! See "man protoc99" for details.
> > >  *
> > >  *--
> > >  */
> >
> > > #ifndef _CIDL_DEVICEMANAGER_HINT_H
> > > #define _CIDL_DEVICEMANAGER_HINT_H
> >
> > > #include 
> >
> > > enum {
> > >  cidl_resource_info_size_hint = 0
> > >+ PROTOC99_TAG_TYPE_SIZE(1) + PROTOC99_MAX_VAR32_SIZE +
> > > PROTOC99_DEFAULT_STRING_LENGTH /* resource_info::device_name */
> > >+ PROTOC99_TAG_TYPE_SIZE(2) + PROTOC99_MAX_VAR32_SIZE /*
> > > resource_info::res_type */
> > >+ (PROTOC99_TAG_TYPE_SIZE(3) + PROTOC99_MAX_VAR32_SIZE +
> > > cidl_resource_info_size_hint) * PROTOC99_DEFAULT_ARRAY_COUNT /*
> > > resource_info::resu
> > >  cidl_device_list_size_hint = 0
> > >+ PROTOC99_TAG_TYPE_SIZE(1) + PROTOC99_MAX_VAR32_SIZE /*
> > > device_list::found_dev */
> > >+ (PROTOC99_TAG_TYPE_SIZE(2) + PROTOC99_MAX_VAR32_SIZE +
> > > cidl_resource_info_size_hint) * PROTOC99_DEFAULT_ARRAY_COUNT /*
> > > device_list::resour
> > >  cidl_get_device_info_size_hint = 0
> > >+ PROTOC99_TAG_TYPE_SIZE(1) + PROTOC99_MAX_BOOL_SIZE /*
> > > get_device_info::result */,
> > >  cidl_device_info_registered_size_hint = 0,
> > >  cidl_device_info_unregistered_size_hint = 0,
> >
> > > };
> >
> > > #endif /* _CIDL_DEVICEMANAGER_HINT_H */
> >
> > > And during gcc build I got an "/cidl_deviceManager_hint.h:25: error:
> > > 'cidl_resource_info_size_hint' undeclared here (not in a function)"
> > > error from the compiler.
> >
> > > On Apr 1, 2:05 am, Jason Hsueh  wrote:
> > > > Recursive definitions should work, though I'm not sure we have a test
> > > case
> > > > to verify use in repeated fields. I know there is one for optional
> > > messages.
> > > > What result did you get?
> >
> > > > On Tue, Mar 29, 2011 at 5:58 AM, zad 
> wrote:
> > > > > Is it possible to have in a message declaration an array of the
> > > > > declaring message?
> > > > > Let me explain better:
> > > > > I'm trying to achieve the following:
> >
> > > > > message resource_info {
> > > > >optional string device_name=18;
> > > > >optional Res_type res_type=19 [default=RAW];
> > > > >repeated resource_info resources=2;
> > > > > }
> >
> > > > > I tried also with this:
> > > > > message resource_info {
> > > > >optional string device_name=18;
> > > > >optional Res_type res_type=19 [default=RAW];
> > > > >extensions 2 to 3;
> > > > > }
> > > > > extend resource_info{
> > > > >repeated resource_info resources=2;
> > > > > }
> >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > Groups
> > > > > "Protocol Buffers" group.
> > > > > To post to this group, send email to protobuf@googlegroups.com.
&g

Re: [protobuf] Protocol Buffers for Matlab

[Jason Hsueh]

I've added this to the ThirdPartyAddOns wiki page. Thanks for providing this
implementation!

On Tue, Apr 5, 2011 at 8:35 AM, Evan Lapisky  wrote:

> Hi,
>
> We've recently open sourced our Matlab compiler for Protocol Buffers.
> It has been in use internally for a couple of years now to provide our
> Matlab users access to messages created with the official C++ and
> Python protobuf bindings.
>
> It is hosted at http://code.google.com/p/protobuf-matlab/
>
> Some basic setup information is available at
> http://code.google.com/p/protobuf-matlab/source/browse/README.txt
>
> -Evan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Repeat custom message extensions not working in Java on deserialisation

[Jason Hsueh]

The most common issue with Java and extensions is not passing an
ExtensionRegistry containing the extensions you want to the deserialization
method. However, you mention that the parse is failing: if you didn't
register the extensions, or didn't use an extension registry, then the data
would simply be ignored. If you're getting an
InvalidProtocolBufferException, the parse failure indicates either you are
parsing bad data, or one of the required fields has not been set.

If that doesn't get you going in the right direction, can you send a short
code snippet?

On Fri, Apr 1, 2011 at 9:06 AM, Ahab  wrote:

> I've been using protocols for a while, and just moved to extensions
> lasts week.  Deserialising out of a byte[] back into my Batch object
> is now failing - protos below.  I am looking in Eclipse and it is
> struggling with the repeat batchData objects - and my test is
> failing.  They end up not being retrievable after deserialisation.  I
> can see the elements are present before serialisation, but are there
> once they come back out.
>
> Is this all supported, am I doing anything wrong?
>
> Thanks in advance.
>
> message GenericData {
>// Should key be an enum - I think so - part of the contract
>required string idKey = 1;
>enum DataType {
>INTEGER = 1 ;
>STRING = 2 ;
>START = 3;
>END = 4 ;
>TIMING = 5;
>}
>required DataType dataType = 2;
>
>optional int32 intType = 3 [default = 0];
>optional string strType = 4 [default = ""];
>optional double startTime = 5 [default = 0];
>optional double stopTime = 6 [default = 0];
> }
>
> message Batch {
>required string idRequest = 1;
>required string idClient = 2;
>enum AppType {
>A = 0;
>B = 1;
>}
>required AppType idApplication = 3 [ default = A ];
>optional string appInstance = 4;
>
>extensions 100 to 199;
> }
>
> extend Batch {
>repeated GenericData batchData = 100;
>repeated TaskList taskList = 101;
> }
>
> I can send code too - but hoping to get quick steer on whether I am
> doing something badly wrong.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: self including message array

[Jason Hsueh]

You appear to be using a third party implementation for pure C. You'll have
to check with the author of the implementation.

On Fri, Apr 1, 2011 at 7:25 AM, zad  wrote:

>  message resource_info {
>optional string device_name=18;
>optional Res_type res_type=19 [default=RAW];
>repeated resource_info resources=2;
> }
>
> the above message generates the following .h
>
> /*! \file deviceManager_hint.h
>  *  \brief message hints
>  */
>
> /*--
>  * Generated 2011-04-01 15:14:14 +0100
>  *   by protoc99 version 0.17
>  * DO NOT EDIT! See "man protoc99" for details.
>  *
>  *--
>  */
>
> #ifndef _CIDL_DEVICEMANAGER_HINT_H
> #define _CIDL_DEVICEMANAGER_HINT_H
>
> #include 
>
> enum {
>  cidl_resource_info_size_hint = 0
>+ PROTOC99_TAG_TYPE_SIZE(1) + PROTOC99_MAX_VAR32_SIZE +
> PROTOC99_DEFAULT_STRING_LENGTH /* resource_info::device_name */
>+ PROTOC99_TAG_TYPE_SIZE(2) + PROTOC99_MAX_VAR32_SIZE /*
> resource_info::res_type */
>+ (PROTOC99_TAG_TYPE_SIZE(3) + PROTOC99_MAX_VAR32_SIZE +
> cidl_resource_info_size_hint) * PROTOC99_DEFAULT_ARRAY_COUNT /*
> resource_info::resu
>  cidl_device_list_size_hint = 0
>+ PROTOC99_TAG_TYPE_SIZE(1) + PROTOC99_MAX_VAR32_SIZE /*
> device_list::found_dev */
>+ (PROTOC99_TAG_TYPE_SIZE(2) + PROTOC99_MAX_VAR32_SIZE +
> cidl_resource_info_size_hint) * PROTOC99_DEFAULT_ARRAY_COUNT /*
> device_list::resour
>  cidl_get_device_info_size_hint = 0
>+ PROTOC99_TAG_TYPE_SIZE(1) + PROTOC99_MAX_BOOL_SIZE /*
> get_device_info::result */,
>  cidl_device_info_registered_size_hint = 0,
>  cidl_device_info_unregistered_size_hint = 0,
>
> };
>
> #endif /* _CIDL_DEVICEMANAGER_HINT_H */
>
> And during gcc build I got an "/cidl_deviceManager_hint.h:25: error:
> 'cidl_resource_info_size_hint' undeclared here (not in a function)"
> error from the compiler.
>
> On Apr 1, 2:05 am, Jason Hsueh  wrote:
> > Recursive definitions should work, though I'm not sure we have a test
> case
> > to verify use in repeated fields. I know there is one for optional
> messages.
> > What result did you get?
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 29, 2011 at 5:58 AM, zad  wrote:
> > > Is it possible to have in a message declaration an array of the
> > > declaring message?
> > > Let me explain better:
> > > I'm trying to achieve the following:
> >
> > > message resource_info {
> > >optional string device_name=18;
> > >optional Res_type res_type=19 [default=RAW];
> > >repeated resource_info resources=2;
> > > }
> >
> > > I tried also with this:
> > > message resource_info {
> > >optional string device_name=18;
> > >optional Res_type res_type=19 [default=RAW];
> > >extensions 2 to 3;
> > > }
> > > extend resource_info{
> > >repeated resource_info resources=2;
> > > }
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to protobuf@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] self including message array

[Jason Hsueh]

Recursive definitions should work, though I'm not sure we have a test case
to verify use in repeated fields. I know there is one for optional messages.
What result did you get?

On Tue, Mar 29, 2011 at 5:58 AM, zad  wrote:

> Is it possible to have in a message declaration an array of the
> declaring message?
> Let me explain better:
> I'm trying to achieve the following:
>
> message resource_info {
>optional string device_name=18;
>optional Res_type res_type=19 [default=RAW];
>repeated resource_info resources=2;
> }
>
> I tried also with this:
> message resource_info {
>optional string device_name=18;
>optional Res_type res_type=19 [default=RAW];
>extensions 2 to 3;
> }
> extend resource_info{
>repeated resource_info resources=2;
> }
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Unexpected memory usage when using RepeatedPtrField

[Jason Hsueh]

Does your memory allocator immediately release memory to the system?
Google's tcmalloc, for instance, does not, for performance reasons.

On Wed, Mar 30, 2011 at 3:22 AM, martin  wrote:

> Hello everybody,
>
> I've got a strange behaviour when using a message with a repeated
> attribute inside.
>
> This is a very simplified example:
>
> message Input
> {
>   required bytes content = 1;
>   required bytes sender = 2;
>   required bytes rcpt = 3;
> }
>
> message Response
> {
>  repeated Input entries = 1;
> }
>
>
> TEST(CreateResponse)
> {
> {
> Response response;
> google::protobuf::RepeatedPtrField* entries =
> response.mutable_entries();
> //entries->Reserve(1);
>
> for(int i=0;i<1;++i)
> {
>std::auto_ptr in = createInputEntry();
>std::string serialized = in->SerializeAsString();
>
>Input* entry = response.add_entries();
>entry->ParseFromArray(serialized.data(), serialized.length());
> }
>
> std::string serializedResponse = response.SerializeAsString();
>
> }
> }
>
> The createInputEntry and serialization does not make in this example
> any sense, but it is a simplified scenario what happens in the real
> application.
>
> I was expecting that when Response is out of scope the memory usage
> should get back to the initial memory size, but unfortunately this
> does not happen. In my test case (it is not the initial size is around
> 2MB, after inserting some entries it increases up to 9MB and the
> memory is not released.
>
> The behaviour changes in case I resize the pointer array to lets say
> 1 in this example. After Response is out of scope the memory usage
> is the same as at the beginning. But If I reserve more than entries
> are created the inital behaviour comes back.
>
> I am not totally sure what happens at that point. Maybe someone could
> explain this strange behaviour.
>
> Thanks in advance.
> Martin
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Enum changes compatiblities

[Jason Hsueh]

It won't be a decode error as long as the enum field is not required. If an
application without the new values receive a message containing one of the
new enum values, it will treat the value as an unknown field. In the C++ and
Java implementations (Python doesn't propagate unknown fields), the value
will be stored in the UnknownFieldSet so that it will be included if the
message is reserialized. However, the field will not be set, and accessing
the field will give you the default value.

On Thu, Mar 31, 2011 at 1:25 PM, Aaron  wrote:

> Hey,
>
> I wanted to know that a enum values can be changed like optional/
> repeated fields in a protobuf. The new values would only be available
> to applications using the new .proto. Or, will it cause a decode error
> for an application receiving the new enum value but doesn't have the
> updated .proto.
>
> Thanks.
>
> -Aaron
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] cout seems to be redirected in protoc, why?

[Jason Hsueh]

Apologies for the late response. Your plugin must write a valid
CodeGeneratorResponse to stdout: protoc reads that response to modify the
generated code. Your print statements are getting interpreted as part of the
CodeGeneratorResponse and causing parse failures. For details, see:
http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/plugin.proto

On Sat, Mar 5, 2011 at 2:53 AM, Seref Arikan  wrote:

> Greetings,
> I was experimenting with the test plugin source code that arrives with
> protocol buffers, and I attempted to use cout to print out some values
> from within the mock plugin.
> The compiled plugin leads to an error from protoc, saying that plugin
> output is unparsable, due to my use of cout.
> cerr works fine, and so does printf. I've tried to find how cout is
> redirected (and also why) to something other than the console.
>
> Any feedback would be much appreciated
>
> Regards
> Seref
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] enumerated types

[Jason Hsueh]

The protocol compiler just generates C++ code. How the enum is represented
in memory is entirely up to your C++ compiler. The standard allows the
compiler to choose the size of the enum (with the restriction that it should
be not more than sizeof(int)). I would guess that a typical compiler would
represent both enums below as a single byte.

On Thu, Mar 17, 2011 at 3:40 AM, AdrianPilko wrote:

> Are google protocol enumerated types (when built for C++)  held in
> memory from least significant bit zero. In other words does the
> following enum fit in the least significant 2 bits of say an int:
>
> enum my2BitEnum {
>a = 0;
>b = 1;
>c = 2;
>d = 3;
>  }
>
> ...and does my8BitEnumQuestion only fit in a byte sized type (eg
> char), or is it packed somehow to fit in the same memory space as
> my2BitEnum:
>
> enum my8BitEnumQuestion {
>a = 252;
>b = 253;
>c = 254;
>d = 255;
>  }
>
> Thankyou
> Adrian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Protocol buffers set bool default value =true

[Jason Hsueh]

I'm not familiar with that error. Can you provide the protoc command you are
using, as well as the output of protoc --version? And is your .proto file
simply:

message Foo {
  optional bool getA = 1 [default = true];
}

On Wed, Mar 16, 2011 at 12:28 PM, steph  wrote:

> The error text i get is:
>
> Error the .proto-file is invalid, content:   optional bool
> getA = 1 [default = true];, line number: 2
>
>
> On Mar 16, 3:25 pm, Jason Hsueh  wrote:
> > On Wed, Mar 16, 2011 at 12:17 PM, steph  wrote:
> > > I want to have a bool value default to true but I cant seem to make it
> > > do that. As stated in the protocol buffer documentation bools default
> > > to false. However since i'm able to set default values on types i want
> > > to set the defautl value of my bool types to true.
> >
> > > Here is what i've tried:
> >
> > > optional bool getA = 1 [default = true];
> >
> > This first one should work. What did you observe when using this?
> >
> >
> >
> >
> >
> > > optional bool getA = 1 [default = TRUE];
> >
> > > optional bool getA = 1 [default = 1];
> >
> > > none of these worked for me. How can i set the bool to default to
> > > true?
> >
> > > thanks,
> > > steph
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to protobuf@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.- Hide quoted text -
> >
> > - Show quoted text -
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Protocol buffers set bool default value =true

[Jason Hsueh]

On Wed, Mar 16, 2011 at 12:17 PM, steph  wrote:

> I want to have a bool value default to true but I cant seem to make it
> do that. As stated in the protocol buffer documentation bools default
> to false. However since i'm able to set default values on types i want
> to set the defautl value of my bool types to true.
>
> Here is what i've tried:
>
> optional bool getA = 1 [default = true];
>

This first one should work. What did you observe when using this?


>
> optional bool getA = 1 [default = TRUE];
>
> optional bool getA = 1 [default = 1];
>
> none of these worked for me. How can i set the bool to default to
> true?
>
> thanks,
> steph
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Copyright license

[Jason Hsueh]

It's on the left bar of the home page: http://code.google.com/p/protobuf/ -
it's just a link to the New BSD
License
.

On Wed, Mar 16, 2011 at 10:02 AM, Kelly  wrote:

> Hi, I can't find the Google Protocol Buffer license anywhere on the
> site. Can someone point me to it.
>
> Thanks
> K
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Default value for custom message types

[Jason Hsueh]

On Mon, Mar 14, 2011 at 1:12 PM, steph  wrote:

> given the following:
>
> message ClassA {
>required int value1 = 1;
>required int value2 = 2;
>optional ClassB value3 = 3;
> }
>
> message ClassB {
>required int value1 = 1;
>required int value2 = 2;
> }
>
>
> It states in the googleprotocol guide that an optional parameter is
> given a default value if it is not set, if it is a primative type it
> is that primitive types default state. However it does not specify
> what is the default state of a custom message.
>
> So my question is in my code I want to be able to tell if the my
> ClassA object had value3 set or not. Will it be set to null?
>

No, getting value3 will return an instance of ClassB in its default state.
To determine if value3 has been explicitly set, use has_value3().


>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] GZip Stream examples

[Jason Hsueh]

I don't think there are any problems with having a raw prefix, and just
wrapping GZipInputStream around the raw input stream after it's been read,
but that sounds pretty messy. It seems easier to just make the whole thing a
gzip stream - it can then also be easily read by other tools that use zlib.

On Tue, Mar 8, 2011 at 8:09 PM, ksamdev  wrote:

> Cool, it worked great.
>
> Can I mix Raw out and Gzip out in the file?
>
> Say, I'd like to write a raw number (4 bytes) at the beginning of the file
> and then add the message through the Gzip stream. Visually, my file would
> look like:
>
> .
>
> where first  - 4 bytes written with raw_out and the rest:
> GG - with Gzip Stream.
>
> Of course, the reading sequence would be:
>
> 1. read 
> 2. keep reading the rest G through Gzip Stream.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] GZip Stream examples

[Jason Hsueh]

You wrap GZipOutputStream around your underlying output stream, and use the
GZipOutputStream with CodedOutputStream instead:

On Tue, Mar 8, 2011 at 10:10 AM, ksamdev  wrote:

> Hi,
>
> Are there any examples on how to use GzipOUtputStream in ProtoBuf?
> I've manages so far combo:
>
>  _raw_out.reset(new
> ::google::protobuf::io::OstreamOutputStream(&_output));
>
_compressed_out.reset(new
::google::protobuf::io::GZipOutputStream(_raw_out.get()));

>  _coded_out.reset(new
> ::google::protobuf::io::CodedOutputStream(_compressed_out.get()));
>
> (both objects are boost::shared_pointer's).
>
> How am I supposed to use the GzipOutputStream here?
>
> thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Store number of messages

[Jason Hsueh]

You want WriteLittleEndian* for fixed width numbers.

On Sat, Mar 5, 2011 at 7:24 AM, ksamdev  wrote:

> Hmm, it makes sense now and explains everything. Unfortunately, I didn't
> see the way to write fixed width number with CodedOutputStream. Is there a
> way to do this?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] RPC Methods

[Jason Hsueh]

http://code.google.com/apis/protocolbuffers/docs/proto.html#services describes
how to define services. However, protobuf does not provide an RPC
implementation. You may set the file option {lang}_generic_services = true
to have the protocol compiler generate a stub that can be used to integrate
with an RPC implementation. The new recommendation is to write a compiler
plugin to generate implementation-specific code using the service
descriptions provided by the protocol compiler:
http://code.google.com/apis/protocolbuffers/docs/reference/other.html

On Fri, Mar 4, 2011 at 4:38 PM, Joshua Partogi wrote:

> Hi there,
>
> Is it possible to define RPC methods in protobuf? From what I am
> reading it seems that protobuf is only limited to data structured data
> serializer. Is there any documentation I read about RPC methods in
> protobuf if any is available?
>
> Thank you for the assistance.
>
> Kind regards,
> Joshua.
>
> --
> http://twitter.com/jpartogi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Can a message derive from another message?

[Jason Hsueh]

No, message inheritance is not supported. Have a look at
http://code.google.com/apis/protocolbuffers/docs/techniques.html#union and
see if that solves your use case.

On Wed, Mar 2, 2011 at 7:04 AM, ZHOU Xiaobo  wrote:

> Hi:
>   My problem is that when I receive a packet I have to check one field( say
> 'int32 COMMAND' ) to decide
> the type of the inner packet. For example:
>
> message OutterPkt {
>required fiexed32 Length = 1;
>required int32 COMMAND = 2;
>required string Content = 3;
> }
>
> message InnerPktA {
> required int32 UserID;
> required int16 UserGender;
> }
>
> message InnerPktB {
>required string UserName = 1;
>reruired string UserDescripton = 2;
> }
>
> my code to process this packet will be:
>
> ..
> OuuterPkt outp;
> outp.ParseFromString(inputbuf);
> if (outp.COMMAND == 1) {
>InnerPktA pkta;
>pkta.ParseFromString(outp.Content());
>..
> } else if (outp.COMMAND == 2) {
>InnerPktB pktb;
>pktb.ParseFromString(outp.Content());
>..
> } else {
>
> }
>
>
> My question is: I think the pseudo code above is not efficient because it
> parses the buffer twice and there are too many memcpy.
> Is there a better way to deal with this situation?
> thanks.X
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] make check

[Jason Hsueh]

Can you try patching r380? This fixed the issue for the previous
reportof
this error.

On Fri, Feb 25, 2011 at 2:13 PM, Shiv  wrote:

> Hi,
> I'm trying to install protobuf on linux 64bit, but stuck with error
> while doing
> "make check"
> I would like to install this in my local path
> so I did this ./configure --prefix=(my path)
>
> here is what I get when I do
> $make check
> /bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.
> -I..-pthread -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-
> compare -O2 -g -DNDEBUG -MT descriptor.lo -MD -MP -MF .deps/
> descriptor.Tpo -c -o descriptor.lo `test -f 'google/protobuf/
> descriptor.cc' || echo './'`google/protobuf/descriptor.cc
> libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -pthread -Wall -Wwrite-
> strings -Woverloaded-virtual -Wno-sign-compare -O2 -g -DNDEBUG -MT
> descriptor.lo -MD -MP -MF .deps/descriptor.Tpo -c google/protobuf/
> descriptor.cc  -fPIC -DPIC -o .libs/descriptor.o
> google/protobuf/descriptor.cc: In member function `virtual const
> google::protobuf::FieldDescriptor*
>
> google::protobuf::DescriptorBuilder::OptionInterpreter::AggregateOptionFinder::FindExtension(google::protobuf::Message*,
> const std::string&) const':
> ./google/protobuf/descriptor.h:1152: error:
>
> `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_'
> is private
> google/protobuf/descriptor.cc:4341: error: within this context
> ./google/protobuf/descriptor.h:1152: error:
>
> `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_'
> is private
> google/protobuf/descriptor.cc:4342: error: within this context
> make[1]: *** [descriptor.lo] Error 1
>
>
> Could someone please help, is there a way of installing this in my
> local path without being a superuser?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Google Protocol buffers and Global Keys

[Jason Hsueh]

Extensionsare
commonly used to allow other parties to independently add fields to a
message. To avoid tag number collisions, you can assign each company a
separate extension range.

On Wed, Feb 23, 2011 at 5:06 AM, clayton  wrote:

> I am trying to convince management that protocol buffers are a good
> idea for a standard messaging scheme we want to put out.  We would
> create a proto file that describes a message for interoperability
> between different systems and even between different companies.  We
> also want the messaging scheme to be  flexible and expandable (i.e.
> different groups can add optional messages that are unique to their
> program without breaking the messaging scheme).  Protocol buffers seem
> like a wonderful fit.  Our problem is this.  Consider this scenario,
> Company A and B get the protobuf file that defines the new standard.
> Company A wants to add optional fields to their message,
> (understanding that nothing should break) without publishing the
> optional fields because they are specific to some internal message.
> Company B also wants to add optional fields to their message for the
> same reason, again without publishing the optional fields.  Now in
> this situation, my understanding is, that because the variant key
> value is between 1 -  2^29-1, none of the keys are actually global and
> so theoretically Company A and B could be stomping on each other
> without knowing it (if both optional fields are using the same key).
> If this is incorrect, please let me know.   What would be the protobuf
> solution to the problem above?  i.e. how can company A keep for
> stomping on company B?  Is it possible to somehow optionally define
> global unique keys (i.e. 16 byte) at the highest level?
>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Error with Netty and extensions

[Jason Hsueh]

On Sun, Feb 6, 2011 at 12:50 PM, Marco@worldcorp wrote:

> 

Do i need to somehow declare extensions whenever i register my
> ProtocolDecoder / ProtocolEncoder?
>

I don't know what ProtocolDecoder/ProtocolEncoder are, but yes, you need to
provide an ExtensionRegistry to the parsing method. See
http://code.google.com/apis/protocolbuffers/docs/reference/java/com/google/protobuf/MessageLite.Builder.html#mergeFrom(com.google.protobuf.CodedInputStream,
com.google.protobuf.ExtensionRegistryLite)


>
> If i don't set any extensions, the client-server communication works
> fine
>
> could anyone help?
>
> w/kindest regards
>  marco
>
>
>
>
>
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Compilation error for ppc/440 target

[Jason Hsueh]

Yes, this code was added in 2.4.0.

On Fri, Feb 4, 2011 at 7:06 AM, dear chap  wrote:

> Unfortunately we have to use the particular compiler in question and
> cannot upgrade. Is the
> DescriptorBuilder::OptionInterpreter::AggregateOptionFinder new code ?
> I dont see this problem with protobuf-2.3.0.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Generate .proto file from FileDescriptor?

[Jason Hsueh]

I'm not aware of anything in the Java library that provides this
functionality.

On Fri, Feb 4, 2011 at 6:45 AM, Ben Wright  wrote:

> Anyone know if this can be accomplished without resorting to JNI?
>
> On Feb 3, 4:01 pm, Jason Hsueh  wrote:
> > C++'s FileDescriptor::DebugString() produces text that is reparsable as a
> > .proto file
> >
> > On Thu, Feb 3, 2011 at 12:52 PM, Ben Wright 
> wrote:
> > > I've started a project generating new FileDescriptors at runtime in
> > > Java using FileDescriptorProto in order to dynamically extend a base
> > > type.
> >
> > > I have a class that generates .proto files (StringBuilder) and one
> > > that generates FileDescriptorProto.
> >
> > > What I was wondering was if there was a class (in java or c++ that can
> > > turn a FileDescriptor / FileDescriptorProto into a .proto text file.
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to protobuf@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com.
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: New protobuf feature proposal: Generated classes for streaming / visitors

[Jason Hsueh]

I'm starting to look at the patch (meant to start end of last week but got
caught up in other stuff)

On Tue, Feb 1, 2011 at 9:30 PM, Kenton Varda  wrote:

> On Tue, Feb 1, 2011 at 3:17 PM, Jason Hsueh  wrote:
>
>> Conceptually this sounds great, the big question to me is whether this
>> should be implemented as an option in the compiler or as a separate plugin.
>> I haven't taken a thorough look at the patch, but I'd guess it adds a decent
>> amount to the core code generator. I have a preference for the plugin
>> approach, but of course I'm primarily an internal protobuf user, so I'm
>> willing to be convinced otherwise :-) Would using a plugin, possibly even
>> shipped with the standard implementation, make this feature too inconvenient
>> to use? Or is there enough demand for this that it warrants implementing as
>> an option?
>
>
> First of all, note that this feature is off by default.  You have to turn
> it on with the generate_visitors message-level option.  The only new code
> added to the base library is a couple templates in WireFormatLite, which are
> of course never instantiated if you don't generate visitor code.
>
> There are a few reasons I prefer to make this part of the base code
> generator:
>
> - If you look at the patch, you'll see that the code generation for the two
> Guide classes actually shares a lot with the code generation for
> MergeFromCodedStream and SerializeWithCachedSizes.  To make this a plugin,
> either we'd have to expose parts of the C++ code generator internals
> publicly (eww) or we'd have to reproduce a lot of code (also eww).
>
> - The Reader and Writer classes directly use WireFormatLite, which is a
> private interface.
>

> - It seems clear that this feature is widely desired by open source users.
>  We're not talking about a niche use case here.
>
>
>> Regarding the proposed interfaces: I can imagine some applications where
>> the const refs passed to the visitor methods may be too restrictive - the
>> user may instead want to take ownership of the object. e.g., suppose the
>> stream is a series of requests, and each of the visitor handlers needs to
>> start some asynchronous work. It would be good to hear if users have use
>> cases that don't quite fit into this model (or at least if the existing use
>> cases will work).
>>
>
> Interesting point.  In the Reader case, it's creating new objects, so in
> theory it ought to be able to hand off ownership to the Visitor it calls.
>  But, the Walker is walking an existing object and thus clearly cannot give
> up ownership.  It seems clear that some use cases need const references,
> which means that the only way we could support ownership passing is by
> adding another parallel set of methods.  I suppose they could have default
> implementations that delegate to the const reference versions, in which case
> only people who wanted to optimize for them would need to override them.
>  But I'd like to see that this is really desired first -- it's easy enough
> to add later.
>

Yeah, there's definitely a need for the const ref versions. It sounds like
nobody is clamoring for mutable access/ownership-passing so let's proceed as
is.


> Also note that my code currently doesn't reuse message objects, but
> improving it to do so would be straightforward.  A Reader could allocate one
> object of each sub-message type for reuse.  But, it seems like that wouldn't
> play well with ownership-passing.
>

 Perhaps instead of ownership-passing the methods could provide mutable
access so people could Swap() etc. It would defeat the optimization, but at
least be less messy. Anyway, all of this can be revisited later should the
need arise.


>

>

>
>>
>> On Tue, Feb 1, 2011 at 10:45 AM, Kenton Varda  wrote:
>>
>>> Hello open source protobuf users,
>>>
>>> *Background*
>>>
>>> Probably the biggest deficiency in the open source protocol buffers
>>> libraries today is a lack of built-in support for handling streams of
>>> messages.  True, it's not too hard for users to support it manually, by
>>> prefixing each message with its size as described here:
>>>
>>>
>>> http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
>>>
>>> However, this is awkward, and typically requires users to reach into the
>>> low-level CodedInputStream/CodedOutputStream classes and do a lot of work
>>> manually.  Furthermore, many users want to handle streams
>>> of heterogeneous message types.  We tell them to wrap their messages in an
>&g

Re: [protobuf] where is input_stream.py?

[Jason Hsueh]

This file was deleted as part of the python serialization optimizations that
were released in 2.3.0. A lot of the functionality now exists in decoder.py.
You shouldn't be relying on internal parts of the package though, so it
should only serve as a reference point for you.

On Fri, Feb 4, 2011 at 3:05 PM, maxw  wrote:

> Hello,
>
> I'm a relatively new user of protocol buffers in python.  Recently I
> have used them successfully to deserialize whole-messages, but am now
> interested in a more granular streaming method (read field by field).
>
> I found several signs that a file named input_stream.py is or was part
> of some distribution (e.g.
>
> http://code.google.com/p/python-twitter/source/browse/google/protobuf/internal/input_stream.py?r=ea4e8dcae50dcee70d2668b5dce62766ec0620dd
> ).
>
> Yet, the 2.3.0 and 2.4.0a distributions don't seem to contain this
> file.
>
> Any hint would be greatly appreciated.
> - Max
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Generate .proto file from FileDescriptor?

[Jason Hsueh]

C++'s FileDescriptor::DebugString() produces text that is reparsable as a
.proto file

On Thu, Feb 3, 2011 at 12:52 PM, Ben Wright  wrote:

> I've started a project generating new FileDescriptors at runtime in
> Java using FileDescriptorProto in order to dynamically extend a base
> type.
>
> I have a class that generates .proto files (StringBuilder) and one
> that generates FileDescriptorProto.
>
> What I was wondering was if there was a class (in java or c++ that can
> turn a FileDescriptor / FileDescriptorProto into a .proto text file.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: New protobuf feature proposal: Generated classes for streaming / visitors

[Jason Hsueh]

Conceptually this sounds great, the big question to me is whether this
should be implemented as an option in the compiler or as a separate plugin.
I haven't taken a thorough look at the patch, but I'd guess it adds a decent
amount to the core code generator. I have a preference for the plugin
approach, but of course I'm primarily an internal protobuf user, so I'm
willing to be convinced otherwise :-) Would using a plugin, possibly even
shipped with the standard implementation, make this feature too inconvenient
to use? Or is there enough demand for this that it warrants implementing as
an option?

Regarding the proposed interfaces: I can imagine some applications where the
const refs passed to the visitor methods may be too restrictive - the user
may instead want to take ownership of the object. e.g., suppose the stream
is a series of requests, and each of the visitor handlers needs to start
some asynchronous work. It would be good to hear if users have use cases
that don't quite fit into this model (or at least if the existing use cases
will work).

On Tue, Feb 1, 2011 at 10:45 AM, Kenton Varda  wrote:

> Hello open source protobuf users,
>
> *Background*
>
> Probably the biggest deficiency in the open source protocol buffers
> libraries today is a lack of built-in support for handling streams of
> messages.  True, it's not too hard for users to support it manually, by
> prefixing each message with its size as described here:
>
>
> http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
>
> However, this is awkward, and typically requires users to reach into the
> low-level CodedInputStream/CodedOutputStream classes and do a lot of work
> manually.  Furthermore, many users want to handle streams
> of heterogeneous message types.  We tell them to wrap their messages in an
> outer type using the "union" pattern:
>
>   http://code.google.com/apis/protocolbuffers/docs/techniques.html#union
>
> But this is kind of ugly and has unnecessary overhead.
>
> These problems never really came up in our internal usage, because inside
> Google we have an RPC system and other utility code which builds on top of
> protocol buffers and provides appropriate abstraction. While we'd like to
> open source this code, a lot of it is large, somewhat messy, and highly
> interdependent with unrelated parts of our environment, and no one has had
> the time to rewrite it all cleanly (as we did with protocol buffers itself).
>
> *Proposed solution:  Generated Visitors*
>
> I've been wanting to fix this for some time now, but didn't really have a
> good idea how.  CodedInputStream is annoyingly low-level, but I couldn't
> think of much better an interface for reading a stream of messages off the
> wire.
>
> A couple weeks ago, though, I realized that I had been failing to consider
> how new kinds of code generation could help this problem.  I was trying to
> think of solutions that would go into the protobuf base library, not
> solutions that were generated by the protocol compiler.
>
> So then it became pretty clear:  A protobuf message definition can also be
> interpreted as a definition for a streaming protocol.  Each field in the
> message is a kind of item in the stream.
>
>   // A stream of Foo and Bar messages, and also strings.
>   message MyStream {
> option generate_visitors = true;  // enables generation of streaming
> classes
> repeated Foo foo = 1;
> repeated Bar bar = 2;
> repeated string baz = 3;
>   }
>
> All we need to do is generate code appropriate for treating MyStream as a
> stream, rather than one big message.
>
> My approach is to generate two interfaces, each with two provided
> implementations.  The interfaces are "Visitor" and "Guide".
>  MyStream::Visitor looks like this:
>
>   class MyStream::Visitor {
>public:
> virtual ~Visitor();
>
> virtual void VisitFoo(const Foo& foo);
> virtual void VisitBar(const Bar& bar);
> virtual void VisitBaz(const std::string& baz);
>   };
>
> The Visitor class has two standard implementations:  "Writer" and "Filler".
>  MyStream::Writer writes the visited fields to a CodedOutputStream, using
> the same wire format as would be used to encode MyStream as one big message.
>  MyStream::Filler fills in a MyStream message object with the visited
> values.
>
> Meanwhile, Guides are objects that drive Visitors.
>
>   class MyStream::Guide {
>public:
> virtual ~Guide();
>
> // Call the methods of the visitor on the Guide's data.
> virtual void Accept(MyStream::Visitor* visitor) = 0;
>
> // Just fill in a message object directly rather than use a visitor.
> virtual void Fill(MyStream* message) = 0;
>   };
>
> The two standard implementations of Guide are "Reader" and "Walker".
>  MyStream::Reader reads items from a CodedInputStream and passes them to the
> visitor.  MyStream::Walker walks over a MyStream message object and passes
> all the fields to the visitor.
>
> To handle a stream of messages, simply atta

Re: [protobuf] [ANN] Beefcake - A sane Ruby ProtoBuf library

[Jason Hsueh]

Added to the wiki, thanks for contributing!

On Sat, Jan 22, 2011 at 12:46 PM,  wrote:

> I'm please to give the Ruby developers here Beefcake.
>
> https://github.com/bmizerany/beefcake/tree/master/lib/beefcake
>
> I would appreciate it if someone would list this on the ThirdPartyAddon
> page
> of the protobuf wiki.
>
> Feeback is also appreciated.
>
> -blake
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: cannot get extensions from serialized proto -- please help

[Jason Hsueh]

Are you looking for something more explicit than
http://code.google.com/apis/protocolbuffers/docs/reference/java/com/google/protobuf/MessageLite.Builder.html#mergeFrom(com.google.protobuf.CodedInputStream,
com.google.protobuf.ExtensionRegistryLite) ? Perhaps this isn't obvious
unless you know what you're looking for?

On Mon, Jan 24, 2011 at 9:10 AM, hp  wrote:

> Thanks a lot.  ExtensionRegistry resolved the issue.  I didnt find
> this in any of documentation. Sorry for late response.. I tried this
> after reading in another post and worked!!! Thanks Again!.
>
> On Jan 21, 1:40 pm, Jason Hsueh  wrote:
> > You didn't provide a code snippet, so it's hard to say, but I would guess
> > that you need to provide an ExtensionRegistry and pass that to the
> parsing
> > method. (It looks like you are working in Java)
> >
> > e.g.
> > ExtensionRegistry extensionRegistry = ExtensionRegistry.newInstance();
> > YourFileContainerProto.registerAllExtensions(extensionRegistry);
> > Results r = Results.parseFrom(data, extensionRegistry);
> >
> >
> >
> > On Fri, Jan 21, 2011 at 8:39 AM, hp  wrote:
> > > test below is failing..please please help
> >
> > > message Results {
> > >  extensions 100 to max
> >
> > > }
> >
> > > message Item1 {
> > >  extend Resultes {
> > >repeated Item1 items = 100;
> > >}
> > > }
> >
> > > message Item2 {
> > >  extend Resultes {
> > >repeated Item2 items = 100;
> > >}
> > > }
> >
> > > Results.getExtension(items) does not return anything when converted to
> > > and from byte array.. Works fine otherwise
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to protobuf@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.- Hide quoted text -
> >
> > - Show quoted text -
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] ProtocolBuffers and existing classes

[Jason Hsueh]

No, you cannot refer to arbitrary classes. Doing so would be difficult to
support across languages, and significantly complicates features like
reflection.

On Tue, Jan 25, 2011 at 6:39 AM, Marco@worldcorp wrote:

> Hello
>  i was wondering if protobuffer supports having an already existing
> class embedded in the message.
> What i mean: from example i have seen, use writes a  .proto buffer and
> protobuffer generates all related java classes.
>
> However, i have already existing classe which i'd like to embed in
> a  .proto file.
>
> Is that supported?
>
> I guess if protobuffer generates interfaces, i can get away with it
> and 'rewrite' my code .. but if it has the functionality to use
> existing classes, that would be appreciated
>
> any constraints on existing classes? i guess they all have to
> implement Serializable
>
> thanks and regarsd
>  marco
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] message forward declaration

[Jason Hsueh]

There are many things that need to be read from imported .proto files to
determine if the .proto is valid, or to produce correct code. e.g.:
- need to differentiate between enum and message imports
- when referencing a qualified type like foo.bar.Baz.Qux, you need to know
what components are package specifiers, and which are objects. This changes
the forward declaration scheme: if that was package foo.bar; message Bar {
message Qux { } }, the declaration is namespace foo { namespace bar { class
Bar_Qux; } }, whereas maybe Bar is just a namespace, so you should instead
produce namespace foo { namespace bar { namespace Bar { class Qux; } } }
- options like java_multiple_files affects the generated code
- when defining extensions, you need to know whether the extendee accepts
extensions, and what range of extension numbers the extendee allows

To use the .proto, you need to build the imported proto files anyway, so
even if you could capture all of this in a simple forward-declaration-type
scheme, I don't think you would save very much.

On Wed, Jan 19, 2011 at 9:02 AM, George  wrote:

> Hi,
>
> It looks to me that the generated result of a single proto file
> doesn’t change significantly based on the content of the imported
> proto files, but mostly on the fact that it is imported.
>
> Did you have considered replacing the need of actual import with some
> kind of message forward declaration?
>
> In my opinion having a single proto file self-sufficient even in case
> it actually refers messages from other proto files could significantly
> simplify the build process.
>
> What the protobuf community thinks about this?
>
> Thanks,
> George
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Problem with accents in python while unpacking a message

[Jason Hsueh]

Can you provide a small, self-contained reproduction of the program?

On Thu, Jan 20, 2011 at 10:39 AM, Louhike  wrote:

> Hi,
> I'm using Google Protobuf with python on a project.
> My problem is I get an error while my program tries to build an
> instance with the function google.protobuf.text_format.merge() if the
> message contains accents (“utf-8 can't decode the byte \xe9” with the
> character 'é' for example).
> I need to keep the accents but I don’t find a solution  to do it. It
> may seem simple but I’m in an early learning phase of programming and
> I'm often stuck on little things like that. Any help would be useful.
>
> Thanks,
> Louhike
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] java_multiple_files option

[Jason Hsueh]

This contains file-level code. In addition to any file-scope extensions you
may have defined, it provides the file's descriptor and initialization code
for the .proto file.

On Thu, Jan 20, 2011 at 2:20 AM, Antoine DESSAIGNE <
antoine.dessai...@gmail.com> wrote:

> Hi everyone,
>
> First I want to say thank you for creating Protobuf, it truely makes my
> life easier. And I'm always thankful for that kind of thing :)
>
> I'm currently going deeper and deeper in Protobuf in order to learn how to
> use it more efficiently.
>
> I found the "java_multiple_files" option that is really great but I was
> wondering why it generates an extra class named after the .proto file. What
> is the use of this class?
>
> Thanks a lot,
>
> Antoine.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Dealing with Corrupted Protocol Buffers

[Jason Hsueh]

It will be rather difficult to correct for the error. The point at which the
parse fails may not be the point of corruption: e.g., the corruption may be
in a byte that is part of a varint, and the continuation bit may be set when
it shouldn't. Similarly you could have a corruption in the length delimiter
for a string or nested message field. Both could cause you to read more
bytes than you should have for that particular field. The encoding is dense
enough that the parser may merrily consume more bytes before encountering an
error to complain about.

You can try to mess with the bytes; you might be able to deal with errors
using some assumptions about the serialized data based on your protocol. But
in general, and going forward, you should write small messages in a
container format that allows for error recovery. Various threads from this
search
discuss
this issue.

On Thu, Jan 20, 2011 at 7:11 PM, Julius Schorzman  wrote:

> Thanks for the tip on CodedInputStream Evan!   I will explore it and
> if I get anything out of it will report back my findings for anyone
> else dealing with this issue.
>
> On Thu, Jan 20, 2011 at 6:27 PM, Evan Jones  wrote:
> > On Jan 20, 2011, at 2:48 , julius-schorzman wrote:
> >>
> >> My question is -- can anything be done to retrieve part of the file?
> >> It would be nice to know at which point in the file the problematic
> >> message occurred, and then I could crop to that point or do some
> >> manual exception -- but unfortunately this exception is very general.
> >> I find it hard to believe that a single mis-saved bit makes the whole
> >> file worthless.
> >
> > You are correct: your entire data is not worthless, but at the point of
> the
> > error, you will need some manual intervention to figure out what is going
> > on.
> >
> > It is probably possible to figure out the byte offset where this error
> > occurs. The CodedInputStream tracks some sort of bytesRead counter, I
> seem
> > to recall. However, this will require you to modify the source.
> >
> >
> >> I also find it curious that the source provides no way (that I can
> >> tell) to get at any lower level data in the p.b. since whenever I try
> >> to do anything with it it throws an exception.  Best I can tell I will
> >> have to write from scratch my own code to decode the p.b. file.
> >
> > The lowest level tools that are provided is CodedInputStream. But yes,
> you
> > will effectively have to "parse" the message yourself. Look at the code
> that
> > is generated for the mergeFrom method of your message to get an idea for
> how
> > it works, and you can read the encoding documentation:
> >
> > http://code.google.com/apis/protocolbuffers/docs/encoding.html
> >
> > You can definitely figure out what is going on, but it will be a bit of a
> > pain. Good luck,
> >
> > Evan Jones
> >
> > --
> > http://evanjones.ca/
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] cannot get extensions from serialized proto -- please help

[Jason Hsueh]

You didn't provide a code snippet, so it's hard to say, but I would guess
that you need to provide an ExtensionRegistry and pass that to the parsing
method. (It looks like you are working in Java)

e.g.
ExtensionRegistry extensionRegistry = ExtensionRegistry.newInstance();
YourFileContainerProto.registerAllExtensions(extensionRegistry);
Results r = Results.parseFrom(data, extensionRegistry);

On Fri, Jan 21, 2011 at 8:39 AM, hp  wrote:

> test below is failing..please please help
>
> message Results {
>  extensions 100 to max
>
> }
>
>
> message Item1 {
>  extend Resultes {
>repeated Item1 items = 100;
>}
> }
>
>
> message Item2 {
>  extend Resultes {
>repeated Item2 items = 100;
>}
> }
>
>
> Results.getExtension(items) does not return anything when converted to
> and from byte array.. Works fine otherwise
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] I have a program use protobuf, and I wan't use this program in a machine which hasn't install protobuf, how can I do this?

[Jason Hsueh]

Add -static to your ld command to link the program statically.

On Wed, Jan 19, 2011 at 12:38 AM, triStone  wrote:

> When I use this program in another computer, it reminder me that "
> error while loading shared libraries: libprotobuf.so.6: cannot open
> shared object file: No such file or directory".
> How can I use this program in a computer without libprotobuf.so.6?
> Should I change the compile script?
> In my makefile, I use this
>
> INCS :=  /usr/local/include
> LIBS := -pthread -L/usr/local/lib -lprotobuf -lprotoc -lz
> 
> ${OUTPUT}: ${OBJS}
>${LD} -o $@ ${LDFLAGS} ${OBJS} ${LIBS}
>
> %.o : %.cpp
>${CXX} -c ${CXXFLAGS} ${INCS} $< -o $@
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Nested messages using C++

[Jason Hsueh]

By the way this is documented here:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp-generated.html#fields

On Thu, Jan 13, 2011 at 5:42 PM, Jason Hsueh  wrote:

>
>
> On Thu, Jan 13, 2011 at 5:38 PM, Linus  wrote:
>
>> I am new to PB and I just ran into this. Is it possible that the PB
>> compiler does not generate set_<...> methods for some nested messages?
>>
>> Here is an example: I don't see set_<...> methods for ANY of the
>> parameters in the db message.
>>
>
> For message type fields, there is no set_<...> method. The codegen produces
> mutable_<...> methods instead. Otherwise it is easy to write inefficient
> code - set would require a potentially expensive copy of the message.
>
>
>>
>> What am I missing???
>>
>> package pd;
>>
>> message nv
>> {
>>repeated int32 Length = 1;
>> }
>>
>> message dp
>> {
>>required int32 DesignID = 1;
>>repeated double Design = 2;
>> }
>>
>> message Shape
>> {
>>required int32 size_m = 1;
>>required int32 size_n = 2;
>> }
>>
>> message ds
>> {
>>required int32 DesignID = 1;
>>repeated double data = 2;
>> }
>>
>> message db
>> {
>>required nv numVars = 1;
>>repeated dp despar = 2;
>>required Shape db_size = 3;
>>repeated ds dtst = 4;
>> }
>>
>>
>> On Jan 13, 1:48 pm, Linus  wrote:
>> > Hello,
>> >
>> > I have nested messages like the following (it is a little more
>> > complicated, but i am trying to simplify with this example).
>> >
>> > package DB;
>> >
>> > message Header
>> > {
>> >  required int32 ID=1;
>> >  message param
>> >  {
>> >  required int32 size_m = 1;
>> >  required int32 size_n = 2;
>> >  }
>> >  required param p = 2;
>> >
>> > }
>> >
>> > message data
>> > {
>> >  repeated double = 1;
>> >
>> > }
>> >
>> > message DB
>> > {
>> >   required Header = 1;
>> >   required data = 2;
>> >
>> > }
>> >
>> > The problem I am having is that the accessor methods generated for the
>> > "DB " does not have a
>> > set_param( DB::Header::param ).
>> >
>> > Is there something wrong with how I am structuring my messages? Any
>> > help is appreciated.
>> > Please let me know if this post is unclear and I will try and explain
>> > myself better.
>> >
>> > Thanks!
>> > Linus
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to protobuf@googlegroups.com.
>> To unsubscribe from this group, send email to
>> protobuf+unsubscr...@googlegroups.com
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Nested messages using C++

[Jason Hsueh]

On Thu, Jan 13, 2011 at 5:38 PM, Linus  wrote:

> I am new to PB and I just ran into this. Is it possible that the PB
> compiler does not generate set_<...> methods for some nested messages?
>
> Here is an example: I don't see set_<...> methods for ANY of the
> parameters in the db message.
>

For message type fields, there is no set_<...> method. The codegen produces
mutable_<...> methods instead. Otherwise it is easy to write inefficient
code - set would require a potentially expensive copy of the message.


>
> What am I missing???
>
> package pd;
>
> message nv
> {
>repeated int32 Length = 1;
> }
>
> message dp
> {
>required int32 DesignID = 1;
>repeated double Design = 2;
> }
>
> message Shape
> {
>required int32 size_m = 1;
>required int32 size_n = 2;
> }
>
> message ds
> {
>required int32 DesignID = 1;
>repeated double data = 2;
> }
>
> message db
> {
>required nv numVars = 1;
>repeated dp despar = 2;
>required Shape db_size = 3;
>repeated ds dtst = 4;
> }
>
>
> On Jan 13, 1:48 pm, Linus  wrote:
> > Hello,
> >
> > I have nested messages like the following (it is a little more
> > complicated, but i am trying to simplify with this example).
> >
> > package DB;
> >
> > message Header
> > {
> >  required int32 ID=1;
> >  message param
> >  {
> >  required int32 size_m = 1;
> >  required int32 size_n = 2;
> >  }
> >  required param p = 2;
> >
> > }
> >
> > message data
> > {
> >  repeated double = 1;
> >
> > }
> >
> > message DB
> > {
> >   required Header = 1;
> >   required data = 2;
> >
> > }
> >
> > The problem I am having is that the accessor methods generated for the
> > "DB " does not have a
> > set_param( DB::Header::param ).
> >
> > Is there something wrong with how I am structuring my messages? Any
> > help is appreciated.
> > Please let me know if this post is unclear and I will try and explain
> > myself better.
> >
> > Thanks!
> > Linus
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] compile protobuf with gcc2.96

[Jason Hsueh]

This compiler is 10 years old, and apparently is not even an official
release: http://gcc.gnu.org/gcc-2.96.html. You should upgrade to a newer
version.

On Thu, Jan 13, 2011 at 9:20 AM, Pinakin Mevawala wrote:

> Not able to do so. Any help?
>
> [root@localhost protobuf-2.3.0]# ./configure && make
> make  all-recursive
> make[1]: Entering directory `/build/installer/ext_src/protobuf-2.3.0'
> Making all in .
> make[2]: Entering directory `/build/installer/ext_src/protobuf-2.3.0'
> make[2]: Leaving directory `/build/installer/ext_src/protobuf-2.3.0'
> Making all in src
> make[2]: Entering directory `/build/installer/ext_src/protobuf-2.3.0/
> src'
> /bin/sh ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -
> I..-pthread -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-
> compare -O2 -g -DNDEBUG -MT common.lo -MD -MP -MF .deps/common.Tpo -c -
> o common.lo `test -f 'google/protobuf/stubs/common.cc' || echo
> './'`google/protobuf/stubs/common.cc
> libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -pthread -Wall -Wwrite-
> strings -Woverloaded-virtual -Wno-sign-compare -O2 -g -DNDEBUG -MT
> common.lo -MD -MP -MF .deps/common.Tpo -c google/protobuf/stubs/
> common.cc  -fPIC -DPIC -o .libs/common.o
> google/protobuf/stubs/common.cc: In function `void
> google::protobuf::ShutdownProtobufLibrary ()':
> google/protobuf/stubs/common.cc:356: no matching function for call to
> `vector >::at (int &)'
> make[2]: *** [common.lo] Error 1
> make[2]: Leaving directory `/build/installer/ext_src/protobuf-2.3.0/
> src'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory `/build/installer/ext_src/protobuf-2.3.0'
> make: *** [all] Error 2
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: how to deal with the byte 0xFF in the business logic?

[Jason Hsueh]

As Adam said, -1 has a representation that is not just a single 0xFF. To
decode a varint, you have to read the bytes until the most significant bit
is 0. (See
http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints)
Note that int32 values are sign-extended for wire compatibility with int64,
so you will get a 10-byte varint.

On Thu, Jan 13, 2011 at 3:49 AM, Adam Skutt  wrote:

>
> On Jan 13, 2:23 am, 飞 杨  wrote:
> > Dear sir,
> >
> > I found, in the protocolbuf int encode, the byte 0xFF may appear, then
> > how can i distinguish the EOS and the business -1, the two both are -1
> > when use the inputstream.read()..
>
> Pay closer attention to the definition of InputStream.read().  The
> return type is an int, which is 32-bits.  '-1' and 0x00FF are
> distinct values.  Make the check before casting the return value to an
> byte.  That being said, I wouldn't read data one byte at a time either
> without a good reason.
>
> Adam
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] strange problem while parsing an extension

[Jason Hsueh]

See other thread

On Thu, Jan 13, 2011 at 3:37 AM, Simeon Mitev wrote:

> Hi all,
>
> I have a strange behavior with java release 2.3.0 and I need your help:
>
> Protos.proto:
>
> ...
> message Request {
>required Api api_call = 1;
>extensions 100 to 200;
> }
>
> message RefundRequest {
>extend Request {
>optional RefundRequest refund_request = 100;
>}
>required int32 orderId = 1;
> }
> ...
>
> Client (java):
>
> Protos.RefundRequest refundRequest =
>Protos.RefundRequest.newBuilder()
>.setOrderId(2)
>.build();
>
> Protos.Request request =
>Protos.Request.newBuilder()
>.setApiCall(Protos.Api.REFUND)
>.setExtension(Protos.RefundRequest.refundRequest, refundRequest)
>.build();
>
>
> Server (java):
>
> ...
> byte[] data;  // data is properly initialized with all bytes transmitted
> over the wire.
>
> Protos.Request request = Protos.Request.parseFrom(data);
>
> // request.getApiCall() is correct. now I need to get transmitted order Id.
> I do expect to have 2.
>
> ExtensionRegistry registry = ExtensionRegistry.newInstance();
> registry.add(Protos.RefundRequest.refundRequest);
>
> RefundRequest refundRequest = Protos.RefundRequest.parseFrom(
>data, registry
>);
> ...
>
> here is the problem:
>
> refundRequest.getOrderId() returns always "1" ?!? no exceptions during
> encoding, transmitting and receiving the data.
>
> What I do wrong?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] strange behavior while parsing an extension

[Jason Hsueh]

You are parsing the serialization as a RefundRequest, not a Request. The "1"
that is being returned is the value of the api_call field - on the wire this
looks like the order_id field - both have tag 1 and are varint encoded.

On Thu, Jan 13, 2011 at 8:58 AM, Simeon Mitev wrote:

> to make it simple, here is one proto file and two jUnits tests. The first
> one passes through and the second one fails. See the comments for details.
>
> Protos.proto:
> 
>
> enum Api {
>REFUND = 1;
> }
>
> message Request {
>required Api api_call = 1;
>extensions 100 to 200;
> }
>
> message RefundRequest {
>required int32 orderId = 1;
> }
>
> extend Request {
>optional RefundRequest refund_request = 100;
> }
>
>
> TestUnit1: (succeeds)
> ==
>
> Protos.RefundRequest rr =
>   Protos.RefundRequest.newBuilder()
>  .setOrderId(3)
>  .build();
>
> byte[] data = rr.toByteArray();
>
>
> Protos.RefundRequest rr2 = Protos.RefundRequest.parseFrom(data);
>
> // suceeds
> Assert.isTrue( rr2.getOrderId() == 3 );
>
>
> TestUnit2: (fails at the end)
> ==
>
>
> Protos.RefundRequest rr =
>   Protos.RefundRequest.newBuilder()
>  .setOrderId(3)
>  .build();
>
> Protos.Request request =
>   Protos.Request.newBuilder()
>  .setApiCall(Protos.Api.REFUND)
>  .setExtension(Protos.refundRequest, rr)
>  .build();
>
> // returns true
> Assert.isTrue( request.hasExtension(Protos.refundRequest) );
>
> // returns true
> Assert.isTrue( request.getExtension(Protos.refundRequest).equals(rr) );
>
> byte[] data = request.toByteArray();
>
> ExtensionRegistry registry = ExtensionRegistry.newInstance();
> registry.add(Protos.refundRequest);
>
> Protos.RefundRequest refundRequest = Protos.RefundRequest.parseFrom(
>  data, registry
>  );
>
> // it fails and returns always 1
>
> Assert.isTrue( refundRequest.getOrderId() == 3 );
>
>
>
> What is wrong that the orderId is always returned as 1 and not 3 as it is
> expected?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to protobuf@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Reducing code size of protocol buffers

[Jason Hsueh]

Have you looked at using the optimize_for CODE_SIZE option rather than the
lite runtime? The lite runtime minimizes the size of the core protobuf
runtime library, but that means you have to generate a more code for each
message. Using CODE_SIZE will make your messages use reflection-based
algorithms in the "heavy" protobuf library, rather than generating
type-specific code. If you have a large number of messages, this may reduce
your binary size more than using the lite runtime.

On Tue, Dec 21, 2010 at 6:57 AM, Tim Turner  wrote:

> I'm trying to minimize the code size of the protocol buffers.  I've
> enabled Lite Runtime, but I'd like to reduce it further.  There are
> several protocol buffers that I am not using which I would like to
> conditionally compile out.  Is there a way to do this?
>
> I have been able to remove the protobufs from the code, but it re-
> numbers them and this appears to cause some sort of runtime memory
> corruption.
>

I'm not sure what you mean by things getting renumbered...are you referring
to tag numbers?


>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Deserialize messages from CodedInputStream without a message size prefix

[Jason Hsueh]

You should be using toByteArray(), not getBytes(), to serialize to the
protobuf wire format. You also need to delimit the messages. Otherwise, the
first ParseFromCodedStream call will consume as many bytes as are available
in the byte array.

On Sun, Dec 12, 2010 at 10:13 PM, Fishtank  wrote:

> Hello,
>
> I have a CodedInputStream created from a byte array (in C++)
>
> CodedInputStream cds((uint8_t*)a,tbytes);
> cds.SetTotalBytesLimit(N,M);
>
> I have written bytes to this array (a) from Java like so
>
> b.writeRawBytes(k.getBytes(),0,k.getLength());
> b.writeRawBytes(v.getBytes(),0,v.getLength());
>
> Given the number N of k,v pairs written, I'd like to deserialize them.
> Notice I haven't prepended byte sizes. I thought I could do something like
> this
>
> for(i = 1 to N){
>   rexp_container->ParseFromCodedStream(&cds) //k
>   // do something with rexp_container
>   rexp_container.Clear();
>   rexp_container->ParseFromCodedStream(&cds) //v
>   // do something with rexp_container
>   rexp_container.Clear();
> }
>
> Is this the correct way to do it? I get a missing field error (not supposed
> to be the case)
> I tried ParsePartialFromCodedStream but I get incorrect results.
>
> Is it okay to provide a CodedInputStream and pick of the messages one by
> one?
>
> Cheers
> Joy
>
> [1] PB ERROR[LOGLEVEL_ERROR](google/protobuf/message_lite.cc:123) Can't
> parse message of type "REXP" because it is missing required fields: rclass
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Float Scalar Type

[Jason Hsueh]

Yup.

On Wed, Dec 8, 2010 at 11:31 AM, quickRiposte wrote:

> Is the GPBUF Float the equivalent of the IEEE754 Float (i.e. 4 bytes)?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Submitting a Patch for Google Protocol Buffers

[Jason Hsueh]

Hate to tell you this, but this looks pretty much identical to the
TextFormat (text_format.{cc,h}) that's already provided by the library (and
the wrapper methods Message::DebugString() and friends). Or are you using
lite mode, which doesn't support this because it's reflection based?

(Generating code for this is probably a non-starter due to the amount of
additional code that would be produced. When you have lots of messages, it's
typically cheaper to just use non-lite mode than to generate special code
for each type.)

On Fri, Nov 19, 2010 at 4:17 PM, Casey Johnson  wrote:

> Hi! My name is Casey Johnson and I am currently working as an intern
> at a company named MusicMasterMind. Recently, my boss asked me to edit
> the google protocol generator for the c++ language to make an extra
> function for the purpose of debugging. The function is created for
> every message and enum and allows the user to receive a string that
> represents that message/enum.
>
> For example, let's consider the addressbook example from the
> tutorials. If a person were to call the void toString(string &str)
> const function, the parameter string would be concatenated to print
> out something similar when the addressbook c++ tutorial demonstrates
> how to write the addressbook to the console.
>
> The following is an example of how the string is constructed based on
> the address book example from the c++ tutorial:
> "number: 805 789 8878
>  Phone_Type: HOME
>  id: 89098
>  name: Casey
>  email: b...@bla.com"
>
> I would appreciate it if someone could point me somewhere where I can
> submit this patch so that you can take a look at the edited
> cpp_message.cc, cpp_message.h, and cpp_file.c files and try out the
> usefulness of the function to make a decision if google will include
> this function in their next release.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Renaming Fields

[Jason Hsueh]

Yes, renaming fields is a wire compatible change. You might break things
that use human readable formats like TextFormat, and of course any code that
references the old field name.

On Thu, Nov 18, 2010 at 1:56 PM, Cameron wrote:

> I'm pretty sure I already know the answer to this, but I haven't found
> it spelled out anywhere so I wanted to ask to be sure:
>
> Can you change or rename a field name without affecting backwards
> compatibility?
>
> I assume the answer is yes because you qualify a field with a tag
> number. Field names are just for the compiled java/cpp/py files that
> are generated, correct?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] memory usage problem of using a group record

[Jason Hsueh]

Well, that depends on how you set up your map. You could instead make the
key a pointer to the protobuf field, and define your comparator so that it
dereferences the pointer. Of course you need to make sure that the pointer
is actually valid: if you mutate the field you may invalidate the pointer.
If you're concerned about that, another approach would be to make your key
type a pointer to the entire protobuf value, and have the comparator look at
the field. When doing lookups in the map, you will need to construct a dummy
protobuf to use as the lookup key.

On Tue, Nov 16, 2010 at 9:49 PM, Ma Jin  wrote:

>
> I try to use the pb to store a group of record with the same schema.
> I wonder to know whether the key of the properties would be copied for
> every record like map.
> if like this, it seems to a waste of memory.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Parse from a iterator range possible?

[Jason Hsueh]

It should be pretty straightforward to implement an io::ZeroCopyInputStream
that provides the byte stream using these iterators.

On Mon, Nov 15, 2010 at 1:30 PM, idleman  wrote:

> Hello everyone,
>
> I am just wondering if it should be possible to parse from a iterator
> range? Say something like this:
>
> template 
> bool ParseFromRange(ForwardIterator, ForwordIterator);
>
> Is it possible? Could it easily be implemented?
>
>
> Thanks in advance!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Dynamic Message

[Jason Hsueh]

On Fri, Nov 5, 2010 at 3:26 PM, AdamM  wrote:

> Is there a way to return the entire extension part of the message into
> a separate message like:
>
> message Foo {
>  extensions 100 to 199;
> }
>
>
> message Baz {
>  extend Foo {
>optional int32 bar = 126;
>  }
> }
>
> In C++ Code:
>
> Foo foo;
> foo.SetExtension(Baz::bar, 15);
>
> Elsewhere in code, obviously the code below does not work but is there
> a way to do something like that?


> Baz baz;
> baz = foo.GetExtension(Baz);
>

No, you can't do that. In your .proto definition, Baz is just a container
for the extension identifier. There is no relationship between the
extensions and the Baz message type. You could have written your proto
definition as:

message Foo {
 extensions 100 to 199;
}

extend Foo {
  optional int32 bar = 126;
}

And done:
Foo foo;
foo.SetExtension(bar, 15);
int value = foo.GetExtension(bar);   // equals 15

If you want to use a message type, do:

message Baz {
  optional int32 bar = 1;
  ...
}
extend Foo {
  optional Baz baz_extension = 126;
}

To access the message in the extensions:
Foo foo;
Baz* baz = foo.MutableExtension(baz_extension);
const Baz& baz = foo.GetExtension(baz_extension);


>
>
>
>
> On Nov 5, 5:28 pm, Daniel Wright  wrote:
> > On Fri, Nov 5, 2010 at 2:12 PM, AdamM  wrote:
> > > Thank you Daniel is was not aware of this feature in PB I will give it
> > > a try.
> >
> > There is a slight gain for smaller numbers, but it's small enough that
> it's
> > not worth the maintenance headache of trying to choose unique small
> numbers.
> >  Basically, tags with numbers less than 16 use 1 byte.  16 to 2048 use
> two
> > bytes, 2048 to 262144 use 3 bytes, 262144 to 33554432 use 4 bytes etc
> (you
> > get 7 bits per byte, except for the first byte which only gets 4 bits.
> http://code.google.com/apis/protocolbuffers/docs/encoding.htmlhas the
> > details).
> >
> > Note that the actual "extensions 100 to 10;" line in the .proto
> file
> > costs nothing -- the gain I'm talking about above is when you encode a
> > message with a given field number.
> >
> > One question though is there any optimization gain you get from using
> >
> >
> >
> >
> >
> >
> >
> > > "extensions 100 to 200;" over "extensions 100 to 10;"?
> >
> > > On Nov 5, 3:21 pm, Daniel Wright  wrote:
> > > > This is exactly what extensions are for -- seehttp://
> > > code.google.com/apis/protocolbuffers/docs/proto.html#extensions
> >
> > > > It would look something like:
> >
> > > > message BaseMessage {
> > > >   required MsgType type = 1;
> > > >   extensions 100 to 10;
> >
> > > > }
> >
> > > > Then each module would have a message like:
> >
> > > > message Msg1 {
> > > >   extend BaseMessage {
> > > > optional Msg1 msg1 = 
> > > >   }
> > > >   required int32 field = 1;
> >
> > > > }
> >
> > > > Then the main program can pass the entire BaseMessage to the right
> module
> > > > based on type, and the module can retrieve the parsed extension.
> >
> > > > On Fri, Nov 5, 2010 at 10:31 AM, AdamM 
> wrote:
> > > > > Hello PB Group,
> >
> > > > > I am programming a group of programs in C++ that use PB to
> > > > > communicate. In my first version of the program I was using a
> .proto
> > > > > file that looked similar to the example 1 below. I would then run a
> > > > > switch statement on a MsgType and create the proper message.
> >
> > > > > I am now starting to write my second version of the program to
> include
> > > > > a plugin module architecture. My program calls for a core program
> and
> > > > > then multiple modules that are written against a base module. Well
> in
> > > > > my core program I get packets over the network and then parse them
> > > > > info a PB Message.
> >
> > > > > I would like a way to have some sort of Base Message that the my
> core
> > > > > program would use to parse with the  base message would contain a
> > > > > packet packet operation code and the data for the actual message
> > > > > structure. I want to program it so the Core program has no idea
> what
> > > > > the actual message structure is. It would pass it to the respective
> > > > > module based on the operation code who would then  parse the actual
> > > > > message structure because it knows what that structure should be.
> >
> > > > > Does anyone have any suggestions how to do this? Any help would be
> > > > > much appreciated.
> >
> > > > > Example 1.
> >
> > > > > enum MsgType {
> > > > >   MSG1,
> > > > >   MSG2,
> > > > >   MSG3
> > > > > }
> >
> > > > > Message Msg1 {
> > > > >   required int32 field = 1 ;
> > > > > }
> >
> > > > > Message Msg2 {
> > > > >   required int32 field = 1 ;
> > > > > }
> >
> > > > > Message Msg3 {
> > > > >   required int32 field = 1 ;
> > > > > }
> >
> > > > > Message BaseMessage {
> > > > >   required MsgType type = 1 ;
> > > > >   optional Msg1 = 2 ;
> > > > >   optional Msg2 = 3 ;
> > > > >   optional Msg3 = 4 ;
> > > > > }
> >
> > > > > --
> > > > > You received this message because you are subscribe

Re: [protobuf] Odd behavior while deserializing PB

[Jason Hsueh]

You can't just parse any message type into *DescriptorProto. Those are
protocol buffers which represent the type definition of other protocol
buffers (e.g. your Request msg). You want to use DynamicMessage in order to
operate on generic messages.

On Wed, Oct 27, 2010 at 9:55 AM, maninder batth wrote:

> I have a PB like
> message Request {
>required int64 supplierId  = 1;
>optional string message_type = 2;
> }
>
> Then i constructed the message as follows :-
>Request request = Request.newBuilder()
>.setSupplierId(100L)
>.setMessageType("blah").build();
>
> Then i check the values by parsing it from the "known" Request type
> as
>Request request1 = Request.parseFrom(request.toByteArray());
>System.out.println(request1.getSupplierId());
>
> It correctly responds with 100.
>
> Then i try to parse it via generic fashion
>
>FileDescriptorProto fdp =
> FileDescriptorProto.parseFrom(request.toByteArray());
>FieldDescriptor supp =
> fdp.getDescriptor().findFieldByNumber(1);
>System.out.println(fdp.getField(supp)); // Nothing prints
>
>FieldDescriptor supp1 =
> fdp.getDescriptor().findFieldByNumber(2);
>System.out.println(fdp.getField(supp1)); // Prints correctly
> the
> value "blah"
>
> 1. My question is what happened to my supplierId field in the above
> case?
> FieldDescriptorProto gives me a correct FIeldDescriptor for field
> number 2, but it acts like it has no awarness of field name 1.
>
> 2. Given a FieldDescriptor, how can i find the name of the field? I
> would like to see field names such as supplierId or message_type.
> FieldDescriptor.getName() returns the value "package"... i dont
> understand why it returns the word "package"
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] New Ruby Protocol Buffers library

[Jason Hsueh]

Looks like Kenton didn't get around to this, so I added it to the wiki.

On Wed, Oct 27, 2010 at 11:48 AM, Brian Palmer wrote:

> I think just the github page for the project would be great. Thanks!
>
> http://github.com/mozy/ruby-protocol-buffers
>
> -- Brian
>
> On Oct 26, 2010, at 10:56 PM, Kenton Varda wrote:
>
> I'd like to add this to the third-party wiki, but I'm not sure which link
> to use.  Can you suggest (or create) a general-purpose landing page?
>
> On Thu, Oct 21, 2010 at 10:02 AM, Brian Palmer wrote:
>
>> Mozy has just open sourced their implementation of Protocol Buffers
>> for Ruby. The implementation has been in use internally at Mozy for
>> over a year. This implementation has put a lot of focus on
>> serialization/deserialization performance, and completeness.
>>
>> The ruby protobuf compiler calls out to protoc to do the heavy
>> lifting, so it's using the same parser as the official compiler. It
>> looks like now there's a plugin system for protoc itself, so if we
>> were writing this project today we probably would've just made the
>> compile-to-ruby functionality a plugin.
>>
>> Enjoy!
>>
>> The repo: http://github.com/mozy/ruby-protocol-buffers
>> The gem: https://rubygems.org/gems/ruby-protocol-buffers
>> The docs: http://rubydoc.info/gems/ruby-protocol-buffers/0.8.4/frames
>>
>> -- Brian Palmer
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to proto...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> protobuf+unsubscr...@googlegroups.com
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Serialize message in C++, parse using Java

[Jason Hsueh]

On Wed, Oct 27, 2010 at 11:50 AM, Jun8  wrote:

> I've Google for a day now and could not find full information on the
> following problem.
>
> I want to serialize protobuf messages in C++, send them to a JMS
> (using activemq-cpp API) and parse in my Java server. Based on what I
> found in my searches, here's my C++ function that serializes the
> message:
>
> /*
>  *  Serialize given protobuf message and send to Active MQ JMS using
> the producer.
>  */
> void MessageProducer::send( const diva::messaging::Message&
> proto_mesg )
> {
>using namespace google::protobuf::io;
>
>long bufLength = proto_mesg.ByteSize() +
> CodedOutputStream::VarintSize32( proto_mesg.ByteSize() );
>unsigned char buf[bufLength];
>
>ZeroCopyOutputStream* raw_output = new ArrayOutputStream( buf,
> bufLength );
>CodedOutputStream* coded_output = new CodedOutputStream(raw_output);
>
>// Prepend the message size to wire message.
>coded_output->WriteVarint32( proto_mesg.ByteSize() );
>

You're writing the message size as a prefix to the message data here...


>
>proto_mesg.SerializeToCodedStream(coded_output);
>
>// Create an ACtive JMS message and insert task & module information
> in header.
>cms::BytesMessage* message = m_session->createBytesMessage();
>
>// Write serialized protobuf message to the JMS message and send.
>message->writeBytes( buf, 0, bufLength );
>m_producer->send( message );
>
>delete message;
>delete coded_output;
>delete raw_output;
> }
>
> And here's the part in Java that parses teh received messages from
> JMS:
>
>// Create a byte array for received message.
>BytesMessage receivedMessage =
> (BytesMessage)received_message;
>logger.info("received message in NAC");
>byte[] mesg_bytes = new
> byte[(int)receivedMessage.getBodyLength()];
>int num_read =
> receivedMessage.readBytes(mesg_bytes);
>

Presumably these bytes contain all the data that you wrote out, including
the size prefix...


>if ( num_read != receivedMessage.getBodyLength() ) {
>throw new Exception("Error reading message
> into byte array");
>}
>
>// Create registry for all possible DIVA messages.
>ExtensionRegistry er =
> ExtensionRegistry.newInstance();
>DivaBase.registerAllExtensions(er);
>
>// Parse the received message.
>diva.messaging.DivaBase.Message m =
> diva.messaging.DivaBase.Message.parseFrom( mesg_bytes, er );
>

...but parseFrom expects only the message data, not a message length prefix.

I'd just remove the WriteVarint32(proto_mesg.ByteSize()) call, since you are
using a transport mechanism that has its own means for delimiting messages.


> Currently, I get
>
> Problem parsing message received in NAC:
> com.google.protobuf.InvalidProtocolBufferException: Protocol message
> end-group tag did not match expected tag.
> Problem parsing message received in NAC: java.lang.ClassCastException:
> org.apache.activemq.command.ActiveMQTextMessage cannot be cast to
> javax.jms.BytesMessage
>
> errors for each message and cannot see what I'm doing wrong.
>
> Thanks for any comments on the code and/or pointers you might provide.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] DescriptorPool::BuildFile() seems not parsing nested messages in order

[Jason Hsueh]

You would need to implement your own sorting routines, moving them around
with SwapElements().

Maps would make it easy for you to search based on some key field; until
then you could just build your own map of the elements on top of the
protobuf.

On Fri, Oct 8, 2010 at 7:35 AM, Himanshu Srivastava wrote:

> Hi Jason,
>
> I was out of this discussion for quite some time, but during this time I
> was able to make my large application's design work with dynamic messages
> using the protobuf::Importer. However, I am now stuck with the issue of
> indexing/sorting and searching messages.
>
> I understand that the code generated messages could be sorted with
> stl::sort(itor:begin, itoe::end, functor), but, that's because there is a
> RepeatedPtrField iterator and RepeatedPtrField list available through the
> message class.
>
> I have to sort/search on a repeated nested message declared in my main
> message depending upon the fields of the nested message, like so:
>
> message nested_msg
> {
> required uint64 indx =1; //sort on this field
> optional uint64 classId =2; //sort on this field
> optional string network_name = 3; //search on this field
>
> }
>
> message coverMessage
> {
> required uint64 nested_msg_count =1;
> repeated nested_msg  msg_instance =2;
> }
>
> Could this be achieved with dynamic messages through polymorphic message
> pointers? My current work on protobuf serialization will remain an
> experimentation otherwise.
>
> P.S.: I hear you people are thinking on the design of protobuf maps. Will
> it be possible that protobufs parent level messages be sorted/searched at
> nested message levels through maps of nested message fields? Will '[key]'
> and '[value]' operators be available for the maps?
>
>
> Regards,
> Himanshu
>
> On Fri, Sep 24, 2010 at 6:02 PM, Himanshu Srivastava 
> wrote:
>
>>  >>Gotcha. If you run protoc on the file produced by your generator
>> class, does it succeed?
>>
>> protoc.exe succeeds parsing my proto file. I have emailed the file to you
>> and you can verify that.
>>
>> >>>>descriptor_pool is of course NULL.
>>
>> >>I meant that you should check the return value of BuildFile(). If it
>> returns NULL, then it could not build the file. The ErrorCollector  >>would
>> tell you why
>> I meant the same, Build_File() returns Null.
>>
>> However, I will get back to this issue. It will take some time to verify
>> issues in my implementation.
>>
>> Thanks & Regards,
>> Himanshu
>>
>>   On Thu, Sep 23, 2010 at 10:37 PM, Jason Hsueh wrote:
>>
>>>
>>>
>>>  On Thu, Sep 23, 2010 at 9:52 AM, Himanshu Srivastava <
>>> him.s...@gmail.com> wrote:
>>>
>>>> Hi Jason,
>>>>
>>>> In code:
>>>> ////
>>>>  DescriptorPool descriptor_pool;
>>>>
>>>> for (int i = 0; i < f.file_size(); ++i) {
>>>>
>>>> descriptor_pool.BuildFile(f.file(i));
>>>> ////
>>>> descriptor_pool is of course NULL.
>>>>
>>>
>>> I meant that you should check the return value of BuildFile(). If it
>>> returns NULL, then it could not build the file. The ErrorCollector would
>>> tell you why.
>>>
>>>
>>>> And for the Importer approach my code does SEG-FAULT.
>>>>
>>>> Further I do need conversion from proto file directly and also a static
>>>> FileDescriptor so that it can be passed around for construction of my new
>>>> messages in a recursive tree algorithm in  another distinct class object.
>>>>
>>>
>>> That's up to you: you'll need to ensure that the DescriptorPool has the
>>> lifetime you want.
>>>
>>> Previously, you mentioned that the importer approach returned NULL. That
>>> would be indicative of a problem in the .proto file: the file could not be
>>> built. Even if you were to successfully build the file, your code would
>>> segfault, because the FileDescriptor is no longer valid, as the Importer,
>>> and thus the DescriptorPool containing the FileDescriptor, have been
>>> deleted.
>>>
>>>
>>>>
>>>> I do not want the descriptor set approach, since that is only available
>>>> when protoc exe is present.
>>>> -
>>>>  >>I doubt that your code will work, as Importer should

Re: [protobuf] Feature proposal: mapped fields

[Jason Hsueh]

On Thu, Oct 7, 2010 at 7:02 PM, Igor Gatis  wrote:

>
>
> On Thu, Oct 7, 2010 at 6:19 PM, Jason Hsueh  wrote:
>
>> Indeed, maps have been brought up repeatedly. I forget the current state
>> of the discussion, but I think it's generally agreed that it would be a good
>> thing to add; it's just a matter of how to implement it (and finding the
>> time to do it).
>>
>> A couple of the major issues:
>> - backward compatibility with existing repeated fields
>>
>
> I believe this has been addressed by the proposal I have made. Maps a
> repeated fields. So they can be exposed as list of pairs (string, message).
>

For backwards compatibility, the idea was to allow current repeated message
fields to be converted to maps. You might expose these as pairs (string,
message), or just as regular message accessors, except with a key parameter
instead of an index. On the wire though, the map field would be serialized
as the messages only. Keys are not serialized separately; they're just a
special field within the message, as in the example in the .proto file. That
leads to ...


>
>
>> - ensuring that the client can't change the map key in the mapped value
>> without also updating the map structure.
>>
>
> Could you provide an example?
>

In the example from the .proto, suppose we generate key-based accessors for
the items map field, and internally the field is represented as a
hash_map.

Item* mutable_items(const string& name);
const Item& items(const string& name);

A client could do something like:
const string key = "foo";
foo.mutable_items(key)->set_name("bar");

i.e. its logical key (and its key when serialized and reparsed) is "bar" but
the hash_map points to it using the old key "foo".


>
>
>>
>>
>> On Wed, Oct 6, 2010 at 11:17 AM, Igor Gatis  wrote:
>>
>>>   // EXPERIMENTAL.  DO NOT USE.
>>>   // For "map" fields, the name of the field in the enclosed type that
>>>   // is the key for this map.  For example, suppose we have:
>>>   //   message Item {
>>>   // required string name = 1;
>>>   // required string value = 2;
>>>   //   }
>>>   //   message Config {
>>>   // repeated Item items = 1 [experimental_map_key="name"];
>>>   //   }
>>>   // In this situation, the map key for Item will be set to "name".
>>>   // TODO: Fully-implement this, then remove the "experimental_" prefix.
>>>   optional string experimental_map_key = 9;
>>>
>>> Interesting. It basically pins a field within repeated object to be the
>>> key. I like that. That's very aligned with my ideas. This allows other key
>>> types fairly easily. I guess it would make sense to support string, int and
>>> enum by default. I like the syntax I described better though. :)
>>>
>>> On Wed, Oct 6, 2010 at 12:07 PM, David Yu wrote:
>>>
>>>> In descriptor.proto, you'll see an experimental map field.  It's not
>>>> usable atm.
>>>> In the meantime, you could always simulate a map serialization using a
>>>> repeated message with
>>>> odd field numbers as $key and even as $value (sequential).
>>>>
>>>> On Wed, Oct 6, 2010 at 2:23 PM, Igor Gatis  wrote:
>>>>
>>>>> Not sure whether this has been discussed before. In any case...
>>>>>
>>>>> It would be nice to have mapped fields, e.g. key-value pairs. It would
>>>>> work similar to repeated fields, which are implicit maps, e.g 0..N keyed
>>>>> messages. Mapped fields would break from 0..N keys to int or string keys.
>>>>> Integers are very compact and that is very attractive in terms of wire
>>>>> format but settle with integer keys are not really greater than 0..N keys.
>>>>> Thus, string seems more suitable keys of mapped fields. Thus, it seems 
>>>>> each
>>>>> item of a mapped field could be defined by the following template-like 
>>>>> proto
>>>>> message:
>>>>>
>>>>> message KeyValuePair_of_SomeMessageType {
>>>>>   required string key = 1;
>>>>>   optional SomeMessageType value = 2;
>>>>> }
>>>>>
>>>>>
>>>>> Let's pick a example. Consider the following messages:
>>>>>
>>>>> message Foo {
>>>>>   optional int int_field = 1;
>>>>>   ...
>>>>> }
>>>>>
>>>&g

Re: [protobuf] Re: Best Practices

[Jason Hsueh]

On Thu, Oct 7, 2010 at 5:29 AM, users   wrote:

> Thank you Kenton and Jason for the advice.  I have a couple more
> questions as I move forward with PB implementations.
>
> In the documentation it states that PBs don't make good first class
> citizens in an OO domain model and the suggestion is of course to wrap
> the PB.  Through some other posts it is quite evident that you do not
> want people subclassing or using 'extends' in Java on PBs.  However,
> in process of looking at wrapping classes it is quite clear the
> Message and Builder interfaces will be common across every PB I
> create, but for each PB there will be a uniqe set of setters, getters,
> and has functions for my protocol member fields.  Am I going to get
> into trouble creating the following type of abstract class using
> generics:
>
>
> public abstract class MyAbstractMessage extends GeneratedMessage.Builder> {
>
>
>protected T message;
>protected B builder;
>
>
>// This is implemented here only to work around an apparent bug in
> the
>// Java compiler and/or build system.  See bug #1898463.  The mere
> presence
>// of this dummy clone() implementation makes it go away.
>@Override
>public B clone() {
>  throw new UnsupportedOperationException(
>  "This is supposed to be overridden by subclasses.");
>}
>/**
> * see {...@link GeneratedMessage#newBuilderForType()}...@link
> MessageLite#newBuilderForType()}
> * @param obj
> * @return
> */
>@SuppressWarnings("unchecked")
>public final B newBuilder() {
>return (B) message.newBuilderForType();
>}
>
>/**
> * see {...@link GeneratedMessage.Builder#build()} {...@link
> MessageLite.Builder#build()}
> * @return
> */
>@SuppressWarnings("unchecked")
>public final T build() {
>message = (T) builder.build();
>builder = null;
>return message;
>}
>
>// ... wrap all the Message and Builder interface methods  ...
>
> }
>
> Then for each PB I create I would wrap them in an implementation of
> this abstract class and add my additional setters and getters, hide
> what I want about builder creation or message object creation and
> implement interfaces as necessary.
>
> Is this approach heading down a path that will create chaos in my life
> going forward as migrate numerous protocols to the PB implementation?


If you have incredibly complex protos this could be cumbersome to maintain.
But if the API you want to expose users is more than just the setters and
getters that are provided by the generated code, it's the way to go.


> One final question, I have realized enumeration changes may be
> particular difficult to adapt to with compatibility issues.  Can you
> point to any strategies here?
>

The advice to avoid required is especially true for enums: if you send a
value that's unknown to the recipient, and the field is required, the client
will fail to parse the entire message (unless it is using the Partial
variants). Otherwise, it's ok that it gets "dropped" (it's just stored in
the unknown fields) since presumably the client has no idea how to handle
that enumeration value anyway.


>
> Thank you for all the input and advice!
>
> On Sep 21, 3:38 pm, Kenton Varda  wrote:
> > On Wed, Sep 15, 2010 at 6:03 AM, users   wrote:
> > > I am implementing PB to provide an external interface and API to a
> > > developing system.  I know the interface will change and the extension/
> > > backwards compatibility is a major benefit.  When designing the .proto
> > > files should I, for pure flexibility, make everything optional or
> > > repeated fields? If I do this I can then write separate Validate
> > > functions for each implementation revision.  I can see an opportunity
> > > to automate the validate function creation with the code generating
> > > capabilities.  This will remove the burden of "required is forever"
> > > but will it impact performance or have other negative effects that I
> > > can not see?  Is this a good practice for flexible implementations or
> > > are there better patterns?
> >
> > This is exactly what I usually recommend to people, and do in my own
> code.
> >  There are no significant performance effects (if anything, performance
> will
> > be better because there are fewer checks to perform when parsing).  But
> as
> > Jason says, opinions vary.
> >
> >
> >
> >
> >
> > > Thanks
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.- Hide quoted text -
> >
> > - Show quoted text -
>
> --
> You received thi

Re: [protobuf] Feature proposal: mapped fields

[Jason Hsueh]

Indeed, maps have been brought up repeatedly. I forget the current state of
the discussion, but I think it's generally agreed that it would be a good
thing to add; it's just a matter of how to implement it (and finding the
time to do it).

A couple of the major issues:
- backward compatibility with existing repeated fields
- ensuring that the client can't change the map key in the mapped value
without also updating the map structure.

On Wed, Oct 6, 2010 at 11:17 AM, Igor Gatis  wrote:

>   // EXPERIMENTAL.  DO NOT USE.
>   // For "map" fields, the name of the field in the enclosed type that
>   // is the key for this map.  For example, suppose we have:
>   //   message Item {
>   // required string name = 1;
>   // required string value = 2;
>   //   }
>   //   message Config {
>   // repeated Item items = 1 [experimental_map_key="name"];
>   //   }
>   // In this situation, the map key for Item will be set to "name".
>   // TODO: Fully-implement this, then remove the "experimental_" prefix.
>   optional string experimental_map_key = 9;
>
> Interesting. It basically pins a field within repeated object to be the
> key. I like that. That's very aligned with my ideas. This allows other key
> types fairly easily. I guess it would make sense to support string, int and
> enum by default. I like the syntax I described better though. :)
>
> On Wed, Oct 6, 2010 at 12:07 PM, David Yu  wrote:
>
>> In descriptor.proto, you'll see an experimental map field.  It's not
>> usable atm.
>> In the meantime, you could always simulate a map serialization using a
>> repeated message with
>> odd field numbers as $key and even as $value (sequential).
>>
>> On Wed, Oct 6, 2010 at 2:23 PM, Igor Gatis  wrote:
>>
>>> Not sure whether this has been discussed before. In any case...
>>>
>>> It would be nice to have mapped fields, e.g. key-value pairs. It would
>>> work similar to repeated fields, which are implicit maps, e.g 0..N keyed
>>> messages. Mapped fields would break from 0..N keys to int or string keys.
>>> Integers are very compact and that is very attractive in terms of wire
>>> format but settle with integer keys are not really greater than 0..N keys.
>>> Thus, string seems more suitable keys of mapped fields. Thus, it seems each
>>> item of a mapped field could be defined by the following template-like proto
>>> message:
>>>
>>> message KeyValuePair_of_SomeMessageType {
>>>   required string key = 1;
>>>   optional SomeMessageType value = 2;
>>> }
>>>
>>>
>>> Let's pick a example. Consider the following messages:
>>>
>>> message Foo {
>>>   optional int int_field = 1;
>>>   ...
>>> }
>>>
>>> message Bar {
>>>   mapped Foo foo = 1;
>>> }
>>>
>>> Internally, protobuf would read the above code as something like:
>>>
>>> message Foo {
>>>   optional int int_field = 1;
>>>   ...
>>> }
>>>
>>> // Known in code generation time only.
>>> message KeyValuePair_of_Foo {
>>>   required string key = 1;
>>>   optional Foo value = 2;
>>> }
>>>
>>> message Bar {
>>>   repeated KeyValuePair_of_Foo foo = 1;
>>> }
>>>
>>>
>>> And generated C++ code for Bar would look like:
>>>
>>> int32 foo_size() const;
>>> bool has_foo(const string& key) const;
>>> const Foo& foo(const string& key) const;
>>> Foo* mutable_foo(const string& key);
>>> void put_foo(const string& key, const Foo& foo);
>>> void remove_foo(const string& key);
>>> const RepeatedPtrField& foo_keys() const;
>>> const RepeatedPtrField& foo_values() const;
>>>
>>>
>>> Thoughts?
>>>
>>> -Gatis
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Protocol Buffers" group.
>>> To post to this group, send email to proto...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> protobuf+unsubscr...@googlegroups.com
>>> .
>>> For more options, visit this group at
>>> http://groups.google.com/group/protobuf?hl=en.
>>>
>>
>>
>>
>> --
>> When the cat is away, the mouse is alone.
>> - David Yu
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Python installation does not build plugin_pb2

[Jason Hsueh]

As Kenton said, including plugin.proto would bloat the core library. Only
people implementing proto compilers such as yourself need to use it.

On the C++ side, you would typically just build a statically linked binary
that has all of the plugin generated code linked in. It's not included in
the C++ core library either. There are various mechanisms to package python
programs, e.g. http://www.pyinstaller.org/, that would allow you to give
users a functioning plugin program without requiring that they install the
generated modules for plugin.proto.


On Wed, Oct 6, 2010 at 12:42 AM, Louis-Marie  wrote:

> Thanks for your answer.
>
> To give you a little bit more information, here's what I'm trying to
> do. I want to deliver a tool using a custom protoc plugin implemented
> in python, so that end user can generate code from its own proto
> files.
>
> There would be nothing special to do before using this plugin, but
> since it depends on plugin_pb2, I need to find the plugin.proto file
> (using pkg-config), compile it, and put it in some appropriate
> location.
>
> Also, I can't put it in its "natural" parent python package
> (google.protobuf) which is already provided by protobuf installation.
> On the other side, the c++ code is generated (and I guess, compiled
> into the shared library), which makes it a little bit more
> straightforward to write c++ plugins.
>
> What would be the best way to achieve what I'm trying to do? If the
> python library size increase is not acceptable there, couldn't the
> plugin_pb2 file be generated in an independent location, so that one
> could still rely on it being present on any protobuf install?
>
> Thanks for your advices,
>
> Louis-Marie
>
>
> 2010/10/6 Kenton Varda :
> > It's not generated because none of the python implementation actually
> uses
> > it.  So, generating it and including it in the egg would just increase
> the
> > library size for everyone, when most people don't need it.
> > What makes you feel uncomfortable about generating it yourself?
> >
> > On Fri, Oct 1, 2010 at 6:01 AM, Louis-Marie  wrote:
> >>
> >> Hi all,
> >>
> >> It looks like python installation of protocol buffers does not
> >> generate the google.protobuf.compiler.plugin_pb2 python file, while
> >> google.protobuf.descriptor_pb2 is explicitly generated by
> >> protobuf/python/setup.py
> >>
> >>  generate_proto("../src/google/protobuf/descriptor.proto")
> >>
> >> Shouldn't the plugin.proto file be compiled and installed the same
> >> way? Maybe I am missing something there, be I feel very uncomfortable
> >> recompiling it when I need to write a plugin.
> >>
> >> Thanks,
> >>
> >> Louis-Marie
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Protocol Buffers" group.
> >> To post to this group, send email to proto...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> protobuf+unsubscr...@googlegroups.com
> .
> >> For more options, visit this group at
> >> http://groups.google.com/group/protobuf?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com
> .
> > For more options, visit this group at
> > http://groups.google.com/group/protobuf?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Regarding various concepts in google protocol buffers.

[Jason Hsueh]

On Fri, Sep 24, 2010 at 2:21 PM, Raghav  wrote:

> Hi I am new to learning google protocol buffers. I have following
> questions on my mind when I went throught all te content availiable on
> website.
>
> 1. What should I know for learning google protocol buffers. I mean is
> there any pre requisites to learn for understanding how protocol
> buffers work (I am working as a Java developer since I year)?

2. What exactly do you mean by term “serializing the Structured data”?


Maybe the wiki page on protocol buffers, and the page on serialization
linked in the first sentence, would help with these first two?
http://en.wikipedia.org/wiki/Protocol_Buffers


>

3. What does following paragraph mean? Can anyone explain with an
> example?
>  The " = 1", " = 2" markers on each element identify the unique "tag"
> that field uses in the binary encoding. Tag numbers 1-15 require one
> less byte to encode than higher numbers, so as an optimization you can
> decide to use those tags for the commonly used or repeated elements,
> leaving tags 16 and higher for less-commonly used optional elements.
> Each element in a repeated field requires re-encoding the tag number,
> so repeated fields are particularly good candidates for this
> optimization.
>

Each field is marked in the wire format with a tag number. If you care about
the encoded size of the data, you should give frequently occurring fields
tag numbers between 1-15 to take advantage of this optimization.


> 4. What does this paragraph mean? Can you please elaborate on this or
> provide an example so that people using protocol buffers can benefit?
> Protocol Buffers and O-O Design Protocol buffer classes are basically
> dumb data holders (like structs in C++); they don't make good first
> class citizens in an object model. If you want to add richer behaviour
> to a generated class, the best way to do this is to wrap the generated
> protocol buffer class in an application-specific class. Wrapping
> protocol buffers is also a good idea if you don't have control over
> the design of the .proto file (if, say, you're reusing one from
> another project). In that case, you can use the wrapper class to craft
> an interface better suited to the unique environment of your
> application: hiding some data and methods, exposing convenience
> functions, etc. You should never add behaviour to the generated
> classes by inheriting from them. This will break internal mechanisms
> and is not good object-oriented practice anyway.
>

Don't inherit from generated protocol buffer classes. Use composition
instead - "has a" relationships instead of "is a" relationships.


> 5. Look at the following paragraph
> "Specifying Field Rules
> You specify that message fields are one of the following:
> •   required: a well-formed message must have exactly one of this
> field.
> •   optional: a well-formed message can have zero or one of this field
> (but not more than one).
> •   repeated: this field can be repeated any number of times (including
> zero) in a well-formed message. The order of the repeated values will
> be preserved."
> It says, in a well formed message , optional  fields can have zero or
> one of this field.
> But in your own examples, I can see that your messages have multiple
> optional?
> Can you please elaborate on this?
>

For a given optional field, a message instance may or may not have that
field. This is independent from a message type having multiple optional
fields in its definition.


> 6. I didn’t get the concept of extensions. Can you please elaborate
> and give an example on it?


What specific questions do you have about it?
http://code.google.com/apis/protocolbuffers/docs/proto.html#extensionsprovides
an example.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Java Protobuf Extension

[Jason Hsueh]

In Java, you need to make sure that you provide an ExtensionRegistry when
you parse the message. You will need to create an ExtensionRegistry
instance, add the extensions that you want to access, and then pass it to
mergeFrom/parseFrom. Otherwise, the extensions will just get treated as an
unknown field.

On Thu, Sep 30, 2010 at 9:21 AM, Squid  wrote:

> So I have a message in one file, and in another file, I have a message
> with extends.  For example:
>
> file1.proto:
>
> message baseMessage {
>  required int32 test = 1;
>  extensions 1000 to max;
> }
>
> file2.proto
> extend baseMessage {
>  required int32 test2 = 1000;
> }
>
> My client then does something along the lines of:
> builder.setExtenssion (File2.test2, 12345);
>
> When my server gets the call that uses this message, I see the data
> being displayed correctly for both test1 and test2 in my log file
> including the extension number of 1000.  However, I cannot get the
> value of test2 and:
> request.hasExtension(File2.test2) == FALSE
>
> For some reason, protobuf doesn't seem to think that the extensions
> are there though I can see them coming through.
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: import statement

[Jason Hsueh]

You typically want to specify imports relative to your "project root", and
run protoc from that directory. Then the includes would be generated
as #include
"pathCommon/CommonMsgs.pb.h" (I'm assuming you have your project's root in
the include path for your compiler)

On Fri, Sep 24, 2010 at 9:58 AM, LakeshoreGuy wrote:

> I can add pathCommon to my compiler's include path to solve the
> problem.  Previously it was only part of the protoc include path
>
> There may be more issues here than I realize, but it's inconvenient
> that the generated function uses the filename AND the path, seems like
> filename should be sufficient.  If this were the case I could add the
> path to the import with ".." and that would also work with the include
> statement.
>
> Jason
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Why ParsePartialFromZeroCopyStream consumes all the data?

[Jason Hsueh]

Yes, this behavior is normal. The wire format is not self delimiting, so
there's no way to tell when one message ends and another starts.
http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
As you mentioned, you can use ParseFromBoundedZeroCopyStream to restrict the
number of bytes read.

On Fri, Sep 24, 2010 at 5:42 AM, Greg Burri  wrote:

> Re,
>
> Ok, I tried with the default implementation
> 'google::protobuf::io::FileInputStream' and the result is the same, so
> I think the described behavior in my first email is correct... but I
> don't understand why!?
>
> /Greg
>
> On Sep 24, 2:32 pm, Greg Burri  wrote:
> > Hi everyone,
> >
> > I wrote my own 'ZeroCopyInputStream' class and did some tests :
> >
> > I serialize two protocol buffer messages in a file and then I read
> > them (usually it's not a file, it's a special device but for the tests
> > it is).
> >
> > Here is my proto file :
> >message Hash {
> >   optional bytes hash = 1; // Always 20 bytes.
> >}
> >
> > The first hash is "2C 58 3D 41 4E 4A 9E B9 56 22 82 09 B3 67 E4 8F 59
> > 07 8A 4B"
> > My second one is "5C 9C 37 41 BD ED 23 1F 84 B8 A8 20 0E AF 3E 30 A9
> > C0 A9 51"
> >
> > The serialization process is correct and my file look-like :
> > 0a 14 2c 58 3d 41 4e 4a 9e b9 56 22 82 09 b3 67 e4 8f 59 07 8a 4b 0a
> > 14 5c 9c 37 41 bd ed 23 1f 84 b8 a8 20 0e af 3e 30 a9 c0 a9
> >
> > Then, I try to read back my first hash with 'ParseFromZeroCopyStream'.
> > 'Next(const void** data, int* size)' from my own class will be called
> > and a buffer to the data will be returned with a size of 44 (the
> > entire file), everything fine until now.
> > The issue is the hash returned is not the first but the last :  "5C 9C
> > 37 41 BD ED 23 1F 84 B8 A8 20 0E AF 3E 30 A9 C0 A9 51" and there is no
> > call at all to 'BackUp(int count)'!
> >
> > Is this behavior is normal ? Do I have to use
> > 'ParseFromBoundedZeroCopyStream' instead ?
> >
> > The documentation[1] says "If successful, the entire input will be
> > consumed." but with no more details, why ?
> >
> > [1] :
> http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google...
> >
> > /Greg
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] DescriptorPool::BuildFile() seems not parsing nested messages in order

[Jason Hsueh]

On Thu, Sep 23, 2010 at 9:52 AM, Himanshu Srivastava wrote:

> Hi Jason,
>
> In code:
> ////
> DescriptorPool descriptor_pool;
>
> for (int i = 0; i < f.file_size(); ++i) {
>
> descriptor_pool.BuildFile(f.file(i));
> ////
> descriptor_pool is of course NULL.
>

I meant that you should check the return value of BuildFile(). If it returns
NULL, then it could not build the file. The ErrorCollector would tell you
why.


> And for the Importer approach my code does SEG-FAULT.
>
> Further I do need conversion from proto file directly and also a static
> FileDescriptor so that it can be passed around for construction of my new
> messages in a recursive tree algorithm in  another distinct class object.
>

That's up to you: you'll need to ensure that the DescriptorPool has the
lifetime you want.

Previously, you mentioned that the importer approach returned NULL. That
would be indicative of a problem in the .proto file: the file could not be
built. Even if you were to successfully build the file, your code would
segfault, because the FileDescriptor is no longer valid, as the Importer,
and thus the DescriptorPool containing the FileDescriptor, have been
deleted.


>
> I do not want the descriptor set approach, since that is only available
> when protoc exe is present.
> -
> >>I doubt that your code will work, as Importer should be looking for a
> file named "dynamicProtoMessages.proto", which probably >>does not exist.
> 
> I create the proto file "dynamicProtoMessages.proto" parmanently on disk
> and read it from disk to Parser after it is generated from my
> CXMLToProtoFileGenerator class.
>

Gotcha. If you run protoc on the file produced by your generator class, does
it succeed?


>
> Thanks & Regards,
> Himanshu
>
>
> On Thu, Sep 23, 2010 at 9:39 PM, Jason Hsueh  wrote:
>
>>
>>
>>  On Thu, Sep 23, 2010 at 2:40 AM, Himanshu Srivastava > > wrote:
>>
>>> Kenton:
>>>
>>>
>>> I really sent you a only a part of the complete RecordHeader message,
>>> since I was focusing on "nested message not getting parsed" issue for this
>>> particular message. Indeed my complete proto file is a large one. I have
>>> attached the truncated proto file containing this complete message only
>>> for reference with this mail (plese see attachment
>>> 'dynamicProtoMessages.proto '.
>>>
>>> I had used Parser and Importer earlier and generated the FileDescriptor
>>> in a class “CMessageParser” member function as below:
>>>
>>>
>>>
>>> //***//
>>>
>>> class PacketTracerErrorFileErrorCollector: 
>>> publicgoogle::protobuf::compiler::MultiFileErrorCollector
>>>
>>> {
>>>
>>> ...
>>>
>>> };
>>>
>>>
>>>
>>>
>>>
>>> const google::protobuf::FileDescriptor* CMessageParser
>>> ::generateMessageFileDescriptor()
>>>
>>> {
>>>
>>> std::pair filePair;
>>>
>>> google::protobuf::compiler::DiskSourceTree source_tree;
>>>
>>> source_tree.MapPath(/*filePair.first*/"", "."/*filePair.second*/);
>>>
>>> PacketTracerErrorFileErrorCollector error_collector;//Defined above
>>>
>>> google::protobuf::compiler::Importer importer(&source_tree,
>>> &error_collector);
>>>
>>> std::string outFileNameString = “dynamicProtoMessages.proto”;
>>> const google::protobuf::FileDescriptor* file_desc =
>>> importer.Import(outFileNameString);
>>> return file_desc;
>>>
>>> }
>>>
>>> //***//
>>>
>>>
>>> However, the problem with that approach was, as soon as I returned the
>>> pointer from the FileDescriptor to a client class object of
>>> CMessageParser, the FileDescriptor* becomes NULL  or invalid  once out
>>> of the class and it becomes useless.
>>>
>>  //***//
>>>
>>> CMessageParser myParser;
>>>
>>>  const google::protobuf::FileDescriptor *file_desc = 
>>> myParser.generateMessageFileDescriptor();
>>> //file_desc is NULL
>>>
>>> //

Re: [protobuf] DescriptorPool::BuildFile() seems not parsing nested messages in order

[Jason Hsueh]

scriptor_pool;
>
> for (int i = 0; i < f.file_size(); ++i) {
>
> descriptor_pool.BuildFile(f.file(i));
>

Did you check that these return non-null and don't have any errors?


> }
>
>
> const Descriptor* descriptor =
> descriptor_pool.FindMessageTypeByName("RecordHeader");
> // descriptor =>NULL here
>

You need to use the fully-qualified name to find the message type:
descriptor_pool.FindMessageTypeByName("traceFileMessagesProto.RecordHeader");


> //const FileDescriptor* new_file_desc =
> descriptor_pool.FindFileByName("dynMessage.dsc");
> /*
> DynamicMessageFactory factory;
>
> Message* my_message = factory.GetPrototype(descriptor)->New();
> delete my_message;
>*/
>
>
>
> }
>
>
> Finally,  the point raised by Kenton and you that the proto file did not
> parse correctly to yield the valid FileDescriptor might be correct, since, I
> am generating the .proto file containing the messages on the fly by
> converting corresponding tags from my XML file.
> When earlier I used a handmade proto file for the messages, which parsed
> correctly to provide the descriptors, I was able to stream data into
> protocol buffers and use the buffers correctly. I will be inspecting this
> file at my end for any typing errors.
>
> Thanks for the answer and for opening this protobuf library on public
> domain, though.
>
>
> Thanks and Regards,
>
> Himanshu
>
>
>
>
>
>
> On Thu, Sep 23, 2010 at 7:53 AM, Jason Hsueh  wrote:
>
>> It occurred to me that protoc shouldn't accept your file either. Did
>> it really compile successfully? What do you actually get in your
>> FileDescriptorSet?
>>
>> On Wednesday, September 22, 2010, Jason Hsueh  wrote:
>> > 1) I would guess the problem is that the "action" message type refers to
>> undefined types eaction_type and ereason. There are probably error messages
>> farther up indicating that action could not be built. Because action isn't a
>> valid type, RecordHeader can't refer to it, and thinks that action is an
>> undefined_symbol as well. The DescriptorBuilder will read in all inputs, and
>> cross-link the types once everything has been processed. It doesn't matter
>> what order they are defined in.
>> > 2) You can manually build the FileDescriptorProto representation
>> yourself, or use the classes used by the compiler: see
>> google::protobuf::compiler::Parser or google::protobuf::compiler::Importer.
>> >
>> > On Wed, Sep 22, 2010 at 4:42 AM, Himanshu Srivastava <
>> him.s...@gmail.com> wrote:
>> >
>> > Hi Kenton,
>> > I have the following message in my .proto file:
>> > ///
>> > package traceFileMessagesProto;
>> > message RecordHeader {
>> > message  myqueue{
>> >  optional uint32 id = 1;
>> >  optional uint32 Priority = 2;
>> > }
>> >
>> > message  action{
>> >  optional eactiontypeeactiontype_instance = 1;
>> >  optional ereason ereason_instance = 2;
>> >  optional myqueue myqueue_instance = 3;
>> > }
>> > optional action action_instance =1;
>> > } //end RecordHeader
>> > /***/
>> > Thus, "action" is nested message in "RecordHeader" and has "queue"
>> submessage in it.
>> > I used the protoc compiler -o  option to generate the
>> descriptor set file and loaded it through istream in c++ application,
>> followed by streaming FileDescriptorSet object with this descriptor set
>> file:
>> >
>> >
>> > //***//
>> > ifstream desc_file("mydescriptor.dsc",ios::in);
>> > FileDescriptorSet f;
>> > f.ParseFromIstream(&desc_file);
>> > //***//
>> >
>> >
>> > However, while parsing through DescriptorPool::BuildFile(f.file(0));
>> > at the line corresponding to the message file line:
>> > => optional action action_instance =1;
>> > the BuildFile() function exits by giving error:
>> > undefined_symbol "action"
>> > The code returns through the following function in protoc library:
>> > //***//
>> > void DescriptorBuilder::AddNotDefinedError(
>> > const string& element

Re: [protobuf] DescriptorPool::BuildFile() seems not parsing nested messages in order

[Jason Hsueh]

It occurred to me that protoc shouldn't accept your file either. Did
it really compile successfully? What do you actually get in your
FileDescriptorSet?

On Wednesday, September 22, 2010, Jason Hsueh  wrote:
> 1) I would guess the problem is that the "action" message type refers to 
> undefined types eaction_type and ereason. There are probably error messages 
> farther up indicating that action could not be built. Because action isn't a 
> valid type, RecordHeader can't refer to it, and thinks that action is an 
> undefined_symbol as well. The DescriptorBuilder will read in all inputs, and 
> cross-link the types once everything has been processed. It doesn't matter 
> what order they are defined in.
> 2) You can manually build the FileDescriptorProto representation yourself, or 
> use the classes used by the compiler: see google::protobuf::compiler::Parser 
> or google::protobuf::compiler::Importer.
>
> On Wed, Sep 22, 2010 at 4:42 AM, Himanshu Srivastava  
> wrote:
>
> Hi Kenton,
> I have the following message in my .proto file:
> ///
> package traceFileMessagesProto;
> message RecordHeader {
> message  myqueue{
>  optional uint32 id = 1;
>  optional uint32 Priority = 2;
> }
>
> message  action{
>  optional eactiontype    eactiontype_instance = 1;
>  optional ereason ereason_instance = 2;
>  optional myqueue myqueue_instance = 3;
> }
> optional action action_instance =1;
> } //end RecordHeader
> /***/
> Thus, "action" is nested message in "RecordHeader" and has "queue" submessage 
> in it.
> I used the protoc compiler -o  option to generate the descriptor 
> set file and loaded it through istream in c++ application, followed by 
> streaming FileDescriptorSet object with this descriptor set file:
>
>
> //***//
> ifstream desc_file("mydescriptor.dsc",ios::in);
> FileDescriptorSet f;
> f.ParseFromIstream(&desc_file);
> //***//
>
>
> However, while parsing through DescriptorPool::BuildFile(f.file(0));
> at the line corresponding to the message file line:
> => optional action action_instance =1;
> the BuildFile() function exits by giving error:
> undefined_symbol "action"
> The code returns through the following function in protoc library:
> //***//
> void DescriptorBuilder::AddNotDefinedError(
>     const string& element_name,
>     const Message& descriptor,
>     DescriptorPool::ErrorCollector::ErrorLocation location,
>     const string& undefined_symbol) {
>   if (possible_undeclared_dependency_ == NULL) {
>     AddError(element_name, descriptor, location,
>  "\"" + undefined_symbol + "\" is not defined.");
> //***//
> The only possible explanation could be that the message "action" is not 
> parsed before its instance declaration from my .proto/descriptor_set file.
> Probably all nested messages should be parsed on their instance declarations 
> in a recursive manner in the DescriptorPool::Buildfile() function.
>
>
> Qustion1:
> How do I solve the above issue?
> Question 2:
> Can not I simply load my messages in DescriptorPool through the .proto file 
> instead of calling the service of protoc compiler to generate a descriptor 
> file? I have to generate .proto file at runtime and I don't want to call 
> another exe (protoc.exe) from my application for dependency reasons.
>
>
> Thanks in advance for any solutions /directions.
> Regards,
> Himanshu
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to 
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/protobuf?hl=en.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] DescriptorPool::BuildFile() seems not parsing nested messages in order

[Jason Hsueh]

1) I would guess the problem is that the "action" message type refers to
undefined types eaction_type and ereason. There are probably error messages
farther up indicating that action could not be built. Because action isn't a
valid type, RecordHeader can't refer to it, and thinks that action is an
undefined_symbol as well. The DescriptorBuilder will read in all inputs, and
cross-link the types once everything has been processed. It doesn't matter
what order they are defined in.
2) You can manually build the FileDescriptorProto representation yourself,
or use the classes used by the compiler: see
google::protobuf::compiler::Parser or google::protobuf::compiler::Importer.

On Wed, Sep 22, 2010 at 4:42 AM, Himanshu Srivastava wrote:

> Hi Kenton,
>
> I have the following message in my .proto file:
>
> ///
> package traceFileMessagesProto;
>
> message RecordHeader {
>
> message  myqueue{
>  optional uint32 id = 1;
>  optional uint32 Priority = 2;
> }
>
> message  action{
>  optional eactiontypeeactiontype_instance = 1;
>  optional ereason ereason_instance = 2;
>  optional myqueue myqueue_instance = 3;
> }
> optional action action_instance =1;
>
> } //end RecordHeader
> /***/
> Thus, "action" is nested message in "RecordHeader" and has "queue"
> submessage in it.
>
> I used the protoc compiler -o  option to generate the descriptor
> set file and loaded it through istream in c++ application, followed by
> streaming FileDescriptorSet object with this descriptor set file:
>
> //***//
> ifstream desc_file("mydescriptor.dsc",ios::in);
> FileDescriptorSet f;
> f.ParseFromIstream(&desc_file);
> //***//
>
> However, while parsing through DescriptorPool::BuildFile(f.file(0));
> at the line corresponding to the message file line:
> => optional action action_instance =1;
>
> the BuildFile() function exits by giving error:
>
> undefined_symbol "action"
>
> The code returns through the following function in protoc library:
> //***//
> void DescriptorBuilder::AddNotDefinedError(
> const string& element_name,
> const Message& descriptor,
> DescriptorPool::ErrorCollector::ErrorLocation location,
> const string& undefined_symbol) {
>   if (possible_undeclared_dependency_ == NULL) {
> AddError(element_name, descriptor, location,
>  "\"" + undefined_symbol + "\" is not defined.");
> //***//
>
> The only possible explanation could be that the message "action" is not
> parsed before its instance declaration from my .proto/descriptor_set file.
> Probably all nested messages should be parsed on their instance
> declarations in a recursive manner in the DescriptorPool::Buildfile()
> function.
> Qustion1:
> How do I solve the above issue?
>
> Question 2:
> Can not I simply load my messages in DescriptorPool through the .proto file
> instead of calling the service of protoc compiler to generate a descriptor
> file? I have to generate .proto file at runtime and I don't want to call
> another exe (protoc.exe) from my application for dependency reasons.
>
> Thanks in advance for any solutions /directions.
>
> Regards,
> Himanshu
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Does protobuf work in Visual Studio 2008 Express?

[Jason Hsueh]

MSVC has different versions of STL: it appears it may change between
compiler versions, and also between debug and release, even for the same
compiler version. You need to make sure that you build your project with the
same C runtime as the probouf lib. Here are some past threads on the issue:
http://groups.google.com/group/protobuf/search?group=protobuf&q=msvc+stl+linker+error

On Mon, Sep 20, 2010 at 10:00 AM, Navigateur  wrote:

> I get 23 linker errors such as the following after linking to
> libprotobuf.lib:
>
> 1>msvcprt.lib(MSVCP90.dll) : error LNK2005: "public: __thiscall
> std::basic_string,class
> std::allocator >::basic_string std::char_traits,class std::allocator >(class
> std::basic_string,class
> std::allocator > const &)" (??0?$basic_str...@du?
> $char_tra...@d@std@@v?$alloca...@d@2@@std@@q...@abv01@@Z) already
> defined in main2.obj
>
> Any clear guesses what I'm doing wrong, or do you need more info?
> Or is it limited to Visual Studio 2005? (that's what the project file
> was made for i.e. VS8.0, I'm using VS9.0).
>
> Ideas?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Trouble compiling 2.3.0 ("Segmentation fault")

[Jason Hsueh]

Hmm, a lovely ICE in gcc. You might try filing a bug with them or checking
to see if there are known issues with Mingw32's gcc. I don't know of an easy
way to omit python support. A quick hack would be to just modify
python_generator.cc to be empty, which would hopefully get rid of the error.

On Wed, Sep 15, 2010 at 2:22 PM, Thomas Wrobel  wrote:

> alternatively, how to skip python compiling would also do I think...I
> just need the c++
>
> On 15 September 2010 22:23, ThomasWrobel  wrote:
> > I'm struggling to compile 2.3.0
> > I'm using Mingw32.
> > The first issue I came across was this one;
> >
> > http://code.google.com/p/protobuf/issues/detail?id=155
> >
> > I followed the advice and got further, but now I'm getting;
> >
> > "$ make check
> > Making check in .
> > make[1]: Entering directory `/c/qt/protobuf-2.3.0'
> > make  check-local
> > make[2]: Entering directory `/c/qt/protobuf-2.3.0'
> > Making lib/libgtest.a lib/libgtest_main.a in gtest
> > make[3]: Entering directory `/c/qt/protobuf-2.3.0/gtest'
> > make[3]: `lib/libgtest.la' is up to date.
> > make[3]: `lib/libgtest_main.la' is up to date.
> > make[3]: Leaving directory `/c/qt/protobuf-2.3.0/gtest'
> > make[2]: Leaving directory `/c/qt/protobuf-2.3.0'
> > make[1]: Leaving directory `/c/qt/protobuf-2.3.0'
> > Making check in src
> > make[1]: Entering directory `/c/qt/protobuf-2.3.0/src'
> > /bin/sh ../libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -
> > I.. -
> > Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -O2 -g -
> > DNDEBUG -MT
> > python_generator.lo -MD -MP -MF .deps/python_generator.Tpo -c -o
> > python_generato
> > r.lo `test -f 'google/protobuf/compiler/python/python_generator.cc' ||
> > echo './'
> > `google/protobuf/compiler/python/python_generator.cc
> > libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -Wall -Wwrite-strings -
> > Woverload
> > ed-virtual -Wno-sign-compare -O2 -g -DNDEBUG -MT python_generator.lo -
> > MD -MP -MF
> >  .deps/python_generator.Tpo -c google/protobuf/compiler/python/
> > python_generator.
> > cc  -DDLL_EXPORT -DPIC -o .libs/python_generator.o
> > google/protobuf/compiler/python/python_generator.cc: In member
> > function 'void go
> > ogle::protobuf::compiler::python::Generator::PrintEnum(const
> > google::protobuf::E
> > numDescriptor&) const':
> > google/protobuf/compiler/python/python_generator.cc:353:6: internal
> > compiler err
> > or: Segmentation fault
> > Please submit a full bug report,
> > with preprocessed source if appropriate.
> > See  for instructions.
> > make[1]: *** [python_generator.lo] Error 1
> > make[1]: Leaving directory `/c/qt/protobuf-2.3.0/src'
> > make: *** [check-recursive] Error 1"
> >
> > 
> > Any ideas/help would be appreciated.
> > Trying to get a demo ready for a show on the 18th ;)
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> > For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
> >
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Best Practices

[Jason Hsueh]

Different people have different opinions on this. There are cases where
required may work fine, and with care, you can even change a required field
to an optional one. However, required field checking has led to some
significant problems in the past, so many advocate avoiding required
altogether. There's no hard and fast rule here. I personally think that this
largely depends on your codebase. If it is pretty well-contained, and you
can tightly control who is using your protocols, it might be reasonable to
use required. I think the problems with required fields tend to show up in
larger codebases where a proto may have many clients and many of its own
dependencies. In these cases it's harder to ensure that the
required->optional change is a safe one, or that every client wants to has
the same definition of a valid message.

The only thing you lose by not using required is the convenience of the
automated validation - optional and required fields are otherwise treated
exactly the same. If you wish, you can write your own validation functions
with equivalent (or perhaps even slightly better) performance.

On Wed, Sep 15, 2010 at 6:03 AM, users   wrote:

> I am implementing PB to provide an external interface and API to a
> developing system.  I know the interface will change and the extension/
> backwards compatibility is a major benefit.  When designing the .proto
> files should I, for pure flexibility, make everything optional or
> repeated fields? If I do this I can then write separate Validate
> functions for each implementation revision.  I can see an opportunity
> to automate the validate function creation with the code generating
> capabilities.  This will remove the burden of "required is forever"
> but will it impact performance or have other negative effects that I
> can not see?  Is this a good practice for flexible implementations or
> are there better patterns?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Java build having trouble with imports

[Jason Hsueh]

You technically don't need to qualify StringMap with foobar at all, since
both .proto files are in the same package.
http://code.google.com/apis/protocolbuffers/docs/proto.html#packages

On Mon, Sep 13, 2010 at 8:00 AM, jebrick  wrote:

> The problem turned out to be how I was referencing the 1st file from
> the 2nd rather than the compile command.
>
> file 1 is foo.proto  and it package is foobar
>
> file 2 is bar.proto which is in the same package and imports foo.proto
>
> when referencing a member of foo.proto for use in bar.proto I had
> foo.StringMap.
>
> What I needed was forbar.StringMap.
>
> The protos were made and compiled first for .Net which seem to work
> with the file reference.member
> The Java compiler wanted package.member.
>
> thanks for the help.
>
> On Sep 10, 12:30 pm, Jason Hsueh  wrote:
> > Yes, imports are paths, so use import "com/foo/bar/some.proto"; The paths
> > need to be specified relative to the --proto_path, not the location of
> the
> > file with the import statement.
> >
> > On Fri, Sep 10, 2010 at 6:40 AM, users   wrote:
> > > Have you tried putting the import path in quotes?
> >
> > > import "foo.proto"
> >
> > > Second ... The import function needs a path, not the dot notation.
> >
> > > import "com/foo/bar/some.proto" might work?
> >
> > > On Sep 9, 9:26 am, jebrick  wrote:
> > > > I want to import 2 files to my proto for compiling.
> >
> > > > The package for all of the files are the same.  They are all in the
> > > > same place (src/com/foo/bar)
> >
> > > > When  I write the import statement, is it import
> > > > com.foo.bar.some.proto?
> >
> > > > My compile command looks like this:
> > > > protoc proto_path=src --java_out=java src/com/foo/bar/some.proto
> >
> > > > The errors are simple, it can not find any of the imports and thus
> can
> > > > not find any thing that references the other files.
> >
> > > > What is the correct set up for doing the imports?
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> >
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Could Java initializers be elided for primitive fields in many cases?

[Jason Hsueh]

Indeed, this has actually been fixed internally but the changes have not
been released yet.

(Although, the code you referenced is the code to generate the clearFoo()
method, which still must set things to the default)

On Wed, Sep 8, 2010 at 1:05 PM, David Dabbs  wrote:

>
> IIRC Java initializes these primitive types as
>
>   private boolean foo_boolean;// false
>   private int foo_int;// 0
>   private longfoo_long;   // 0L
>   private float   foo_float;  // 0.0f
>   private double  foo_double; // 0.0d
>
> But java_primitive_field.cc unconditionally emits initialization code even
> when the proto -derived default value from is the same as the organic
> default.
>
> Was there a performance reason for including the (seemingly) unnecessary
> init code?
> If not, would there be any harm is NOT explicitly initializing these
> primitive field(s)?
> Seems like not doing so would reduce code size.
>
>
>
> //
> // from java_primitive_field.cc
> //
>
>  JavaType type = GetJavaType(descriptor_);
>  if (type == JAVATYPE_STRING || type == JAVATYPE_BYTES) {
>// The default value is not a simple literal so we want to avoid
> executing
>// it multiple times.  Instead, get the default out of the default
> instance.
>printer->Print(variables_,
>  "  result.$name$_ = getDefaultInstance().get$capitalized_name$();\n");
>  } else {
>printer->Print(variables_,
>  "  result.$name$_ = $default$;\n");
>  }
>
>
> I didn't go as far as creating a patch in case there were specific reasons
> for the init code.
>
>
> Thanks,
>
> David
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] .proto file version constant

[Jason Hsueh]

Can you provide a small reproduction of the problem? A couple common errors:
- custom options need to be specified as "option () = ;"
(parentheses around the identifier)
- if you're using the option in a different package than the one in which
its defined, it needs to be qualified with the package name

Also, note that protocol buffers were originally designed to get away from
having to deal with version numbers (see "A bit of history":
http://code.google.com/apis/protocolbuffers/docs/overview.html) Typically,
you would define your protocol in such a way that different versions are
still compatible. When that's not practical/possible, people may just switch
to entirely new messages.

On Wed, Sep 8, 2010 at 11:42 PM, Jesper  wrote:

> I would like to be able to have a version constant in the .proto file,
> which can be encoded into messages, so that the communicating parties
> can verify that they are using a sufficiently new .proto file. I tried
> using a custom FileOption as described here>
> http://code.google.com/apis/protocolbuffers/docs/proto.html#options,
> but I was unable to compile the resulting code (I don't have the exact
> compiler errors at hand at the moment, but they were about missing
> definitions related to the Descriptor C++ type).
>
> I could put the protocol version number in a header file, but since
> I'm using both C++ and Java I would like to be able to encode the
> version such that it can be easily picked up by both C++ and Java.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Java build having trouble with imports

[Jason Hsueh]

Yes, imports are paths, so use import "com/foo/bar/some.proto"; The paths
need to be specified relative to the --proto_path, not the location of the
file with the import statement.

On Fri, Sep 10, 2010 at 6:40 AM, users   wrote:

> Have you tried putting the import path in quotes?
>
> import "foo.proto"
>
> Second ... The import function needs a path, not the dot notation.
>
> import "com/foo/bar/some.proto" might work?
>
>
>
> On Sep 9, 9:26 am, jebrick  wrote:
> > I want to import 2 files to my proto for compiling.
> >
> > The package for all of the files are the same.  They are all in the
> > same place (src/com/foo/bar)
> >
> > When  I write the import statement, is it import
> > com.foo.bar.some.proto?
> >
> > My compile command looks like this:
> > protoc proto_path=src --java_out=java src/com/foo/bar/some.proto
> >
> > The errors are simple, it can not find any of the imports and thus can
> > not find any thing that references the other files.
> >
> > What is the correct set up for doing the imports?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Scala protocol buffers protoc plugin

[Jason Hsueh]

Default values for required fields do mean that: use the default if not
explicitly set. This only affects the in-memory object, though -
IsInitialized() will still fail if a required field is not set. However,
such a proto would be usable by a client that does not do required field
checking by using the Partial versions of the serialization/parsing
routines. But most clients use the versions with IsInitialized() checks, so
default values are not really useful for required fields in practice, aside
from documentation purposes in the .proto file.

On Thu, Sep 2, 2010 at 6:34 PM, Jeff Plaisance wrote:

> Does default on required fields currently have a meaning?  It compiles but
> I'm not sure if there are cases where the default is ever usable.  Maybe
> required with default specified could mean use the default if not explicitly
> set.  This change would also allow you to get rid of unused required fields
> over 2 releases.  This might be too serious of an api change at this point,
> but it seems like required has fallen out of favor anyway, so maybe this
> wouldn't be too bad.
>
>
> On Tue, Aug 24, 2010 at 4:47 PM, Kenton Varda  wrote:
>
>> On Wed, Aug 18, 2010 at 7:07 AM, Jeff Plaisance 
>> wrote:
>>
>>> It seems like the issue here is that optional has been overloaded to mean
>>> two different things:
>>>
>>> 1) Not really optional but in order to do rolling upgrades we're making
>>> it optional.  The default should be used if it is not set.  In my opinion,
>>> in this case there should be no "has" method because either its result is
>>> irrelevant or it is being used to overload some other meaning on top of
>>> optional.
>>> 2) Optional in the sense of Option, Maybe, Nullable, empty, can be null,
>>> whatever you want to call it.  In my opinion this should be encapsulated in
>>> the type so that the programmer is forced to handle all possible cases.  The
>>> has method should not be used for this because it is too easy to ignore.
>>>
>>
>> Yes, I think you're right, and I see how it makes sense to distinguish
>> these two by the presence or absence of an explicit default value.
>>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Announcing Lua Support

[Jason Hsueh]

Thanks! I've added your implementation to the wiki (along with the other Lua
implementation, which didn't seem to be on the list)

On Sun, Sep 5, 2010 at 9:47 PM, Gregory Szorc wrote:

>  Oops! I didn't mean to claim my project marked the arrival of protocol
> buffers to Lua. There is a prior Lua implementation at
> http://code.google.com/p/protoc-gen-lua/. The major difference is most of
> that one is implemented in Lua (only the low-level packing/unpacking is in
> C) and mine is completely done in C/C++.
>
> Sorry for the follow-up spam, but I wanted to correct the false assertion
> in my previous post.
>
> Greg
>
>  *From:* Gregory Szorc 
> *Sent:* Sunday, September 05, 2010 9:30 PM
> *To:* protobuf@googlegroups.com
> *Subject:* Announcing Lua Support
>
> I'm pleased to announce the arrival of protocol buffers to Lua!
>
> The project lives at http://github.com/indygreg/lua-protobuf .
> Documentation can be found on the wiki there.
>
> The plugin generates C++ code that interfaces with both the C++ code
> generated by protoc and the "official" Lua C API. So, support is limited to
> Lua implementations that conform to that API (official distribution on
> lua.org, LuaJIT, possibly others).
>
> There are a number of features still absent and some known bugs and issues
> (see the wiki). But, it is generally usable and has many of the features
> you'd expect, so releasing I am. In the days ahead, I hope to spend time on
> a test suite as well as exposing more of the C++ API to Lua.
>
> Comments, feedback, and suggestions (especially on how to deal with the
> lua_Number conversion problem) are very welcome.
>
> Gregory Szorc
> gregory.sz...@gmail.com
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Java reflection package resolution

[Jason Hsueh]

I'm confused. You mention using pure Java reflection, yet you also said you
are trying to do some stuff with protobuf reflection. You should not be
using Java reflection at all.

To get the field type, as defined in the .proto file, use
FieldDescriptor.getType()

On Tue, Sep 7, 2010 at 6:29 AM, narfm...@yahoo.com wrote:

> Hello,
>
> I am using the google protocol buffer reflection api to try to
> dynamically create a gui for users to fill in Messages manually. I had
> done this completely using Java reflection alone, but this eventually
> led to problems synchronizing the data, and was all around more work
> than google's reflection api should be.
>
> I have implemented as follows: I pass a Descriptor into the main
> JPanel building function. From here, it iterates over the fields, and
> if it is integral, it makes a JTextField, and if it is a Message, it
> calls the same function again. What I can not find out is the type
> that the fielddescriptor as I am iterating over the fields from the
> descriptor. It appears that one "should" be able to figure this out in
> several ways, but none work for me.
>
> First I thought it must be soemthing in JavaType, but that appears to
> only be what google protocol buffers converted the .proto into. Next I
> thought I could get the full package name and me a newInstance of it,
> but when calling fullName, it only returns the class name and no
> package name. Next, I tried getting the file and querying it's package
> name, but it returns "".
>
> Like I said above, I implemented all of this in pure java reflection,
> and the .proto generates the .class files correctly with fully
> qualified names, but the google protocol buffer reflection api (using
> descriptors and field descriptors) does not give me enough
> information. Anyone have any ideas as to why it's not giving me what I
> want?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] C++ "bytes" fields access problem

[Jason Hsueh]

Can you provide a small reproduction of the problem? The generated code
should not be doing any UTF8 validation. Is it possible you have a field
defined elsewhere that has type 'string'?

On Tue, Sep 7, 2010 at 4:40 AM, Mk  wrote:

> Hi.
> I get funny problem. In proto file I have one "bytes" field, it used
> for exchanging small binary array of data(~100 bytes). In java side it
> is ok, because of using ByteString as container. But on C++ consumer I
> get an error, when access to field: libprotobuf ERROR c:\projects
> \protobuf-2.3.0\src\google\protobuf\wire_format.cc:1059] Encountered
> string containing invalid UTF-8 data while parsing protocol buffer.
> Strings must contain only UTF-8; use the 'bytes' type for raw bytes.
>
> In generated binding class I have only getter with return type
> std::string. In this mail list i find two answers:
> 1) strings is ok for binary data
> 2) problem with zero bytes while presenting byte array as string
>
> The first answer is not Ok, because it add constraint for binary data.
> Second one looks like true, but I cant do anything with it because of
> container constrains.
>
> Any ideas how to convert byte array to string without UTF-8 check, or
> how to get raw byte array?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Get enum as string in Python

[Jason Hsueh]

Oh, indeed enum_types_by_name is what you want - my bad!

On Thu, Aug 26, 2010 at 5:51 PM, Mike  wrote:

> Got it, Thanks!
>
> In your first statement, I believe the method is actually
> "enum_types_by_name" - at least that's what worked for me.
>
> On Aug 26, 12:29 am, Jason Hsueh  wrote:
> > You can pull this from the descriptors:
> > my_enum_descriptor = SomeMessage.DESCRIPTOR.enums_by_name['MyType']
> > my_enum_descriptor.values_by_number[SomeMessage.FIRST].name
> >
> > (See the docs on the descriptor module fromhttp://
> code.google.com/apis/protocolbuffers/docs/reference/python/ind...
> > )
> >
> >
> >
> > On Wed, Aug 25, 2010 at 11:07 PM, Mike  wrote:
> > > I'm pretty sure this is possible, but I can't seem to figure it out.
> > > There was another post about how to do this in C++, but I couldn't
> > > figure out how to make it work in Python.
> >
> > > My message is similar to:
> >
> > > message SomeMessage {
> > >  enum MyType {
> > >FIRST = 0;
> > >SECOND = 1;
> > >THIRD = 2;
> > >  }
> >
> > >  required MyType type = 0;
> > > }
> >
> > > How do I programmatically get the string "FIRST" back if someone
> > > passes me this message with the 'type' field set to SomeMessage.FIRST?
> >
> > > Something along the lines of...
> > > >>> print SomeMessage.EnumName(SomeMessage.FIRST)
> > > FIRST
> >
> > > Thanks!
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Get enum as string in Python

[Jason Hsueh]

You can pull this from the descriptors:
my_enum_descriptor = SomeMessage.DESCRIPTOR.enums_by_name['MyType']
my_enum_descriptor.values_by_number[SomeMessage.FIRST].name

(See the docs on the descriptor module from
http://code.google.com/apis/protocolbuffers/docs/reference/python/index.html
)

On Wed, Aug 25, 2010 at 11:07 PM, Mike  wrote:

> I'm pretty sure this is possible, but I can't seem to figure it out.
> There was another post about how to do this in C++, but I couldn't
> figure out how to make it work in Python.
>
> My message is similar to:
>
> message SomeMessage {
>  enum MyType {
>FIRST = 0;
>SECOND = 1;
>THIRD = 2;
>  }
>
>  required MyType type = 0;
> }
>
>
> How do I programmatically get the string "FIRST" back if someone
> passes me this message with the 'type' field set to SomeMessage.FIRST?
>
> Something along the lines of...
> >>> print SomeMessage.EnumName(SomeMessage.FIRST)
> FIRST
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [scala] [protobuf] [ANN] scala protobuf builder 0.1

[Jason Hsueh]

I added this to the ThirdPartyAddOns wiki as well. Thanks for contributing
to the protobuf community!

On Thu, Aug 19, 2010 at 3:09 AM, Eugene Vigdorchik <
eugene.vigdorc...@gmail.com> wrote:

> Hi all,
> I'm pleased to announce the birth of scala-protobuf project. You can
> find the sources at http://code.google.com/p/protobuf-scala .
> The goal of the project is allowing easy construction of Google
> protobuf (http://code.google.com/p/protobuf/) messages from Scala.
> Unlike conventional protobuf code, protobuf-scala allows for a more
> code-as-data way to write your protocol messages. Here is a short
> example.
>
> Given a protocol definition
> message Artist {
>  required string name = 1;
>  message Album {
>required string title = 1;
>repeated string tracks = 2;
>optional uint32 year_produced = 3;
>  }
>
>  repeated Album albums = 2;
> }
>
> one creates instances like this:
>
> val BradMehldau = Artist (
>  name("Brad Mehldau"),
>  albums (
>Album (
>  title("Day Is Done"),
>  tracks(
>"50 ways to leave your lover",
>"Grenada"
>  ),
>  year_produced(2005)
>)
>  )
> )
>
> As most of scalac plugins, protobuf-scala consists of a generator
> plugin and a thin runtime layer in scala. The output of scala code
> generator requries java classes produced with --java_output, so be
> sure to produce it as well.
> Hope those of you who use both scala and protobufs find it useful.
>
> Cheers,
> Eugene.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Combing multiple messages of only builtin C++ types

[Jason Hsueh]

If you want to serialize them as one message, make your definition do
exactly that:
message CombinedTypes {
  optional int32 int32_val = 1;
  optional double double_val = 2;
  optional float float_val = 3;
}

Note that you can make the code generator produce effectively the same code
you describe by just defining a wrapper message

message CombinedTypes {
  optional Proto_int int_msg = 1;
  optional Proto_double double_msg = 2;
  optional Proto_float float_msg = 3;
}

When serializing, it does:
1) write tag and type (1, LENGTH_DELIMITED) for Proto_int (e.g., the enum
value you were writing)
2) write length (out->WriteVarint32(int_msg.ByteSize() - you should _not_
use sizeof)
3) write the data

Repeat for double/float. Likewise the generated parsing code will handle
everything for you. The first approach is more efficient though.

On Wed, Aug 25, 2010 at 1:00 AM, bongo  wrote:

> Hi,
>
> I am trying to string up multiple messages of different C++ built-in
> types into one message. Conceptually, it's something like:
>
> codedstream << int_x << float_y << double_z;
>
> I am thinking of implementing:
>
> =
> .proto File
> =
>
> enum Type
> {
>  // Enums that correspond to
>  // C++ built-in types
>  Tint = 0;
>  Tdouble = 0;
>  Tfloat = 0;
> }
>
> Proto_int{ required int val = 1; }
> Proto_double { required double val = 1; }
> Proto_float  { required float val = 1; }
>
> =
> C++ code to
> write built-in
> types to protobuf
> =
>
> sos = new StringOutputStream(&buf);
> out = new CodedOutputStream(&sos);
>
> ...
>
> // We want to write all 3 of these in ONE protobuf message
> double d = 123.0;
> int i = 456;
> float f = 789.0f;
>
> // Create the messages
> Proto_int   msg_int;
> Proto_doublemsg_double;
> Proto_float msg_float;
>
> msg_int.set_val(i);
> msg_double.set_val(d);
> msg_float.set_val(f);
>
> // Write out the messages:
>
> // 1. First, write out the size of the message ...
> out->WriteVarint32(sizeof(msg_int));
>
> // 2. then the type (so that we know how to parse it when we read
> back) ...
> // QUESTION: WriteVarint32 is used to write ints, what should I use
> to
> // write double/float?
> out->WriteVarint32(Tint);
>
> // 3. then the message itself
> msg_int.SerializeToCodedStream(&out);
>
> // Then repeat 1 - 3 for double and float, using msg_double/Tdouble
> and msg_float/Tfloat respectively.
>
> =
> code to READ from
> protobuf and reconstruct
> built-in types
> =
>
> bool read() {
>
>ArrayInputStream ais(static_cast(&buf[0]),
> buf.size());
>CodedInputStream in(&ais);
>
>google::protobuf::uint32 type = 0;
>google::protobuf::uint32 size = 0;
>
>if (!in.ReadVarint32(&size)) return false;
>
>while(size) {
>if (!in.ReadVarint32(&type)) return false;
>
>CodedInputStream::Limit limit = in.PushLimit(size);
>switch(type) {
>case Tdouble:
>{
>Proto_double p;
>p.ParseFromCodedStream(&in);
>}
>break;
>... similar code for Tint and Tfloat
> }
>
> =
> Question
> =
> The above scheme feels very inefficient, and I suspect I am probably
> doing it wrong.
> For example, do I really need to make dummy wrappers like this:
>"Proto_int{ required int val = 1; }"
> What is the correct way to achieve my goal stated above?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Reusing proto object in a loop instead of creating new one each time

[Jason Hsueh]

This has come up before on the list, for instance
http://groups.google.com/group/protobuf/browse_thread/thread/6f4812e30a63b88aand
http://groups.google.com/group/protobuf/browse_thread/thread/368f5b0518775a9d

On Fri, Aug 20, 2010 at 7:19 AM, Prakash Rao wrote:

> Hi,
> Is it possible to re-use proto object while sending list of proto
> objects instead of building a new proto object each time? Currently
> i'm building a new proto object and it is put in a proto list & return
> it to the client. Is it possible to re-use the proto object as stated
> below?
>
> // Create proto object (the same proto object is going to be used in
> the while loop)
> while loop {
>  // populate proto object
>  // wirte proto object to coded output stream
>  // clear proto object
> }
>
> flush coded outputstream
>
> I'm trying to do this and it says you can't clear proto object once
> it's already built. If this is possible, is there any performance
> overhead with this approach (memory wise we might save something as we
> are reusing proto object but clear API may add performance overhead).
>
> Regards,
> Prakash
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Encoding/Decoding of data - Question on CodedInputStream & CodedOutputStream

[Jason Hsueh]

Yes, it gets encoded in the PB wire format. See Daniel's earlier message:
writeTo(OutputStream os) wraps the stream in a CodedOutputStream for you.

On Thu, Aug 19, 2010 at 2:43 AM, Prakash Rao wrote:

> I'm not using plain java serialization. I'm using protomessage object
> and I'm directly reading/writing from/to inputstream/outputstream that
> i got from httpurlconnection. I want to know whether my data will be
> encoded as i'm using inputstream & outputstream without wrapping it in
> codedinputstream & codedoutputstream.
>
> MyProtoObject.writeTo(outputstream);
> MyProtoObject.parseFrom(inputstream);
>
> Here outputstream & inputstream are got from httpurlconnection and are
> not wrapped within codedinputstream & codedoutstream. I would like to
> know whether PB does data encoding here.
>
> Regards,
> Prakash
>
> On Aug 18, 11:59 pm, Jason Hsueh  wrote:
> > On Wed, Aug 18, 2010 at 6:57 AM, Prakash Rao  >wrote:
> >
> > > Thanks for your response. I wanted to whether it's CodedInputStream &
> > > CodedOutputStream which does encoding of data in PB?.
> >
> > Yes, these classes handle the PB wire format.
> >
> > > Will data get
> > > encoded if I just use OutputStream & InputStream (without wrapping it
> > > in CodedOutputStream & CodedInputStream) while sending & receiving the
> > > same?
> >
> > Are you asking about using Java serialization, and using
> > writeObject/readObject on InputStream and OutputStream? Messages
> implement
> > Serializable, and override it so that the protobuf wire format is used.
> > CodedInput/OutputStream will be used to read and write that format.
> >
> >
> >
> >
> >
> > > Regards,
> > > Prakash
> >
> > > On Aug 17, 12:32 am, Daniel Wright  wrote:
> > > > I'm not completely sure I understand your question, but if you're
> asking
> > > > about the difference between writeTo(OutputStream) and
> > > > writeTo(CodedOutputStream), they're the same -- writeTo(OutputStream)
> > > just
> > > > wraps the OutputStream in a CodedOutputStream and writes to that.
>  Here's
> > > > the code:
> >
> > > >   public void writeTo(final OutputStream output) throws IOException {
> > > > final int bufferSize =
> >
> > > CodedOutputStream.computePreferredBufferSize(getSerializedSize());
> > > > final CodedOutputStream codedOutput =
> > > > CodedOutputStream.newInstance(output, bufferSize);
> > > > writeTo(codedOutput);
> > > > codedOutput.flush();
> > > >   }
> >
> > > > Most users can just use OutputStream and let the above wrapper take
> care
> > > of
> > > > things for you, unless you're writing lots of tiny messages and the
> cost
> > > of
> > > > creating the CodedOutputStream becomes significant, or you're using
> the
> > > > CodedOutputStream to write your own metadata besides the message
> itself
> > > > (e.g. delimiters).
> >
> > > > On Mon, Aug 16, 2010 at 8:02 AM, Prakash Rao <
> prakashrao1...@gmail.com
> > > >wrote:
> >
> > > > > Hi,
> > > > > I'm new to PB and would like to know whether it is CodedInputStream
> &
> > > > > CodedOutputStream which takes care of encoding data while writing
> into
> > > > > streams. In few APIs I'm directly using InputStream & OutputStream
> > > > > taken from HTTP URL connection class and I would like to know if
> the
> > > > > data will be encoded in these cases (not using CodedInputStream &
> > > > > CodedOutputStream for these APIs). What other advantages
> > > > > CodedInputStream & CodedOutputStream will have as compared to
> direct
> > > > > InputStream & OutputStream taken from HTTP URL connection.
> >
> > > > > Regards,
> > > > > Prakash
> >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > Groups
> > > > > "Protocol Buffers" group.
> > > > > To post to this group, send email to proto...@googlegroups.com.
> > > > > To unsubscribe from this group, send email to
> > > > > protobuf+unsubscr...@googlegroups.com
> 
> > > 
> > > > > .
> > > > > For more options, visit this group at
&g

Re: [protobuf] Re: Encoding/Decoding of data - Question on CodedInputStream & CodedOutputStream

[Jason Hsueh]

On Wed, Aug 18, 2010 at 6:57 AM, Prakash Rao wrote:

> Thanks for your response. I wanted to whether it's CodedInputStream &
> CodedOutputStream which does encoding of data in PB?.


Yes, these classes handle the PB wire format.


> Will data get
> encoded if I just use OutputStream & InputStream (without wrapping it
> in CodedOutputStream & CodedInputStream) while sending & receiving the
> same?
>

Are you asking about using Java serialization, and using
writeObject/readObject on InputStream and OutputStream? Messages implement
Serializable, and override it so that the protobuf wire format is used.
CodedInput/OutputStream will be used to read and write that format.


>
> Regards,
> Prakash
>
> On Aug 17, 12:32 am, Daniel Wright  wrote:
> > I'm not completely sure I understand your question, but if you're asking
> > about the difference between writeTo(OutputStream) and
> > writeTo(CodedOutputStream), they're the same -- writeTo(OutputStream)
> just
> > wraps the OutputStream in a CodedOutputStream and writes to that.  Here's
> > the code:
> >
> >   public void writeTo(final OutputStream output) throws IOException {
> > final int bufferSize =
> >
> CodedOutputStream.computePreferredBufferSize(getSerializedSize());
> > final CodedOutputStream codedOutput =
> > CodedOutputStream.newInstance(output, bufferSize);
> > writeTo(codedOutput);
> > codedOutput.flush();
> >   }
> >
> > Most users can just use OutputStream and let the above wrapper take care
> of
> > things for you, unless you're writing lots of tiny messages and the cost
> of
> > creating the CodedOutputStream becomes significant, or you're using the
> > CodedOutputStream to write your own metadata besides the message itself
> > (e.g. delimiters).
> >
> > On Mon, Aug 16, 2010 at 8:02 AM, Prakash Rao  >wrote:
> >
> >
> >
> > > Hi,
> > > I'm new to PB and would like to know whether it is CodedInputStream &
> > > CodedOutputStream which takes care of encoding data while writing into
> > > streams. In few APIs I'm directly using InputStream & OutputStream
> > > taken from HTTP URL connection class and I would like to know if the
> > > data will be encoded in these cases (not using CodedInputStream &
> > > CodedOutputStream for these APIs). What other advantages
> > > CodedInputStream & CodedOutputStream will have as compared to direct
> > > InputStream & OutputStream taken from HTTP URL connection.
> >
> > > Regards,
> > > Prakash
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.- Hide quoted text -
> >
> > - Show quoted text -
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Performance analysis of RepeatedField versus generated class

[Jason Hsueh]

I think Kenton meant to suggest adding the option to the repeated double
data field.

On Wed, Aug 18, 2010 at 9:42 AM, nirajshr  wrote:

> If I add [packed=true] as you mentioned, I get the following error:
>
> ...[packed = true] can only be specified for repeated primitive
> fields.
>
> I guess packed=true only works with primitive fields.
>
> On Aug 17, 8:53 pm, Kenton Varda  wrote:
> > BTW
> >
> > On Tue, Aug 17, 2010 at 8:51 AM, nirajshr  wrote:
> > >   repeated DoubleData bucketDouble = 2;
> >
> > You should declare this field "packed":
> >
> >   repeated DoubleData bucketDouble = 2 [packed=true];
> >
> > This will improve efficiency on the wire.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Warning messages from compiler

[Jason Hsueh]

This actually got fixed in
r302,
which I think was after 2.3.0.

On Wed, Aug 18, 2010 at 9:30 AM, nirajshr  wrote:

> I am using version 2.3.0
>
> On Aug 17, 8:24 pm, Kenton Varda  wrote:
> > No, the warning is not specific to your system.  I believe we fixed these
> > warnings in version 2.3.0 -- maybe you should upgrade?
> >
> > On Tue, Aug 17, 2010 at 7:17 AM, nirajshr  wrote:
> > > I am getting the following warning message from the compiler while
> > > compiling my program with google protocol buffers using g++ (gcc
> > > version 4.1.2  (Red Hat 4.1.2-46)) with a -Wall option.
> >
> > > /usr/include/google/protobuf/io/coded_stream.h: In member function
> > > 'bool
> >
> > >
> google::protobuf::io::CodedInputStream::ReadLittleEndian32(google::protobuf::uint32*)':
> > > /home/nshresth/usr/include/google/protobuf/io/coded_stream.h:776:
> > > warning: comparison between signed and unsigned integer expressions
> >
> > > /usr/include/google/protobuf/io/coded_stream.h: In member function
> > > 'bool
> >
> > >
> google::protobuf::io::CodedInputStream::ReadLittleEndian64(google::protobuf::uint64*)':
> > > /usr/include/google/protobuf/io/coded_stream.h:791: warning:
> > > comparison between signed and unsigned integer expressions
> >
> > > I want to know if this warning is specific to my system.
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> >
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Why protobuf generates IMMUTABLEjava message?

[Jason Hsueh]

Here are some previous threads on the subject:
http://groups.google.com/group/protobuf/browse_thread/thread/1699791071e92c83
http://groups.google.com/group/protobuf/browse_thread/thread/30d76b08545a0097

On Thu, Aug 12, 2010 at 9:52 AM, Peng  wrote:

> Why does protobuf generate IMMUTABLE java message? but it seems it is
> not immutable for c++ codes.
>
> Thanks.
>
> Peng
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] New RPC implementation for C++/C#

[Jason Hsueh]

Added to the wiki.

On Wed, Aug 11, 2010 at 4:17 PM, Niall wrote:

> Could I get a link to it added on the ThirdPartyAddOns wiki page? The
> project is at:
>
> http://code.google.com/p/protobuf-remote/
>
> Feedback is also welcome.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: How to retrieve parameters using tag numbers using CodedInputStream

[Jason Hsueh]

A tag indicates to the decoder what is coming next in the input stream, and
allows the decoder to handle it appropriately. This is a key feature of the
protobuf wire format - this enables forward and backward compatibility, as
well omitting fields that aren't present from the serialization. The APIs
that we have are built around this. A readString(int tag) method would never
be used by the generated code.

If you really want to take this approach, you can still call readTag(),
check that the tag number is what you want, and then read the string.
However, I would suggest just using a message. You may find that over time
those two or three parameters will grow to the point that maintaining this
is unwieldy.

On Wed, Aug 11, 2010 at 10:55 PM, Prakash Rao wrote:

> I don't have .proto message created for these. Most of my APIs
> takes .proto messages as parameters (as they represent my business
> data objects) and in 1-2 APIs I need to pass only 2-3 primitive
> parameters as arguments. In these cases I’m avoiding creation
> of .proto messages (as these don’t represent my business data object).
> As an alternative, I thought of using CodedOutputstream to pass
> parameters along with tag so that retrieval is easy on the receiving
> end using CodedInputStream but I don't see a way to retrieve parameter
> using tags with CodedInputStream. Currently I've the below code and
> thought the tag approach would make it simple but CodedInputStream
> doesn't support parameter retrieval based on tags.
>
> // Client Side code - which sends s1 & s2 strings to server
> int s1Size = CodedOutputStream.computeStringSizeNoTag(s1);
> codedOutputStream.writeRawVarint32(s1Size);
> codedOutputStream.writeStringNoTag(s1);
>
> int s2Size = CodedOutputStream.computeStringSizeNoTag(s2);
>

Note: computeStringSizeNoTag() gives the size of the string as well as the
encoded length. If you write the result as the encoded length, you will end
up reading more bytes than you should. You simply want to write the size of
the string.


> codedOutputStream.writeRawVarint32(s2Size);
> codedOutputStream.writeStringNoTag(s2);
>
> // Server side code - which reads s1 & s2 string
> CodedInputStream codedInputStream =
> CodedInputStream.newInstance(inputStream);
> int s1Size = codedInputStream.readRawVarint32();
> int s1SizeEntry = codedInputStream.pushLimit(s1Size);
> String s1 = codedInputStream.readString();
> codedInputStream.popLimit(s1SizeEntry);
>
> int s2Size = codedInputStream.readRawVarint32();
> int s2SizeEntry = codedInputStream.pushLimit(s2Size);
> String s2 = codedInputStream.readString();
> codedInputStream.popLimit(modifiedBySizeEntry);
>
> Regards,
> Prakash
>
> On Aug 11, 9:17 pm, Jason Hsueh  wrote:
> > Suppose those strings are all defined as optional fields in the message.
> > They may not be present in the serialization, so the decoder needs to
> read a
> > tag to determine the field number and type before handling the value.
> >
> > On Wed, Aug 11, 2010 at 8:37 AM, Prakash Rao  >wrote:
> >
> >
> >
> > > Hi,
> > > I was just looking at CodedInputStream class and there are APIs to
> > > write primitives with or without tag (for example, writeString &
> > > writeStringNoTag). I thought if we set unique tags for multiple
> > > parameters then we would be able to retrieve these parameters using
> > > tag numbers specified using CodedInputStream class. I don't see any
> > > APIs to read values based on tags in CodedInputStream. Am I missing
> > > something? For the time being I’m using pushLimit & popLimit APIs to
> > > get different parameter values (based on serializedSize).
> >
> > > For example,
> > > int test(String s1, String s2, String s3) {
> > > }
> >
> > > CodedOutputStream out = CodedOutputStream.newIInstance("Some Output
> > > Stream");
> > > out.writeString(1, s1);
> > > out.writeString(2, s2);
> > > out.writeString(3, s3);
> > > out.flush();
> >
> > > Why these are not possible?
> > > CodedInputStream in = CodedInputStream.newIInstance("Some Input
> > > Stream");
> > > String s1 = in.readString(1);
> > > String s2 = in.readString(2);
> > > String s3 = in.readString(3);
> >
> > > Regards,
> > > Prakash
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+u

Re: [protobuf] Re: beginner question about extensions

[Jason Hsueh]

Is this in a loop? It appears there are two occurrences of that log line,
and one of them has a completely empty message.

On Wed, Aug 11, 2010 at 4:44 PM, Jun8  wrote:
>
>
> libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse
> message of type "myevents.Event" because it is missing required
> fields: task_id, module_name
> Received event:<--- This appears to
> be an empty message
>
> Received event:<--- This looks like
> a second occurrence of the same print statement, with a valid message.
> task_id: 1
> module_name: "face_detector"
> [diva.events.status_event] {
>  type: MODULE_START
>  data {
>key: "keyframe_width"
>value: "360"
>  }
> }
>
> I couldn't figure out the error message, the fields are clearly there.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: flushing the data

[Jason Hsueh]

ParseFromFileDescriptor will read until there is no more input. This is
consistent with behavior when parsing from other types of input streams, as
the message wire format is not self delimiting. For file descriptors, that
means you have to reach EOF. As such, ParseFromFileDescriptor is only useful
when the whole file contains a single message.

One approach to writing multiple messages to the same stream is to use a
length-delimited format: write the size of the message, then serialize the
message itself. On the receiver side, you would set up a FileInputStream,
and wrap a CodedInputStream around that. You can read the size of the
messages from the stream and then use PushLimit and PopLimit to control how
much data is read.

On Wed, Aug 11, 2010 at 2:19 PM, nirajshr  wrote:

> Hi Kenton,
>
> I really appreciate your quick response. As you mentioned in your
> response, the protobuf library does write all the data to the file
> descriptor before returning from SerializeToFileDescriptor(). But, I
> have to explicitly close the file descriptor (using close(fd) ) before
> the data gets flushed. As a result, ParseFromFileDescriptor keeps
> waiting for the input until the file descriptor has been closed.
>
> I tried breaking this process into two parts to see if the problem
> persists:
> In the first program,
>  1) Firstly, I used the SerializeToString() function to write to a
> string,
>  2) Then, I wrote the string to a file descriptor using the
> write(char*, buffer length) function.
>
> Then, I tested the following two ways to receive the data at the other
> end of the socket:
> In second program,
>  1) I used ParseFromFileDescriptor() function to receive the data. The
> problem persisted in this case. The first program had to call
> close(fd) in order for the second program to receive the data.
>
>  2) Alternatively, I used read(char* str, buff_len) &
> ParseFromString() functions to receive the data. In this case, the
> program immediately received the data, without having  to close the
> file descriptor in the first program.
>
> I know that this issue might be outside the scope of google protocol
> buffers. But, this contradictory results show that the
> ParseFromFileDescriptor() function expects an explicit close of the
> filedescriptor before it can retrieve all the data. I guess this makes
> sense as ParseFromFileDescriptor() function has no way of knowing the
> length of the data. Having to close the file descriptor as a way to
> mark the end of the transmission of data is confusing. The
> SerializeToFileDescriptor() function has to take care of this so that
> ParseFromFileDescriptor() function has an idea about the size of data
> being received.
>
> I am sorry to bother you with this long post, but, your insight will
> be very helpful.
>
> On Aug 10, 5:40 pm, Kenton Varda  wrote:
> > The protobuf library writes all data to the file descriptor before
> returning
> > from SerializeToFileDescriptor().  So, the problem is not in protobufs.
> >  Sorry.
> >
> > On Tue, Aug 10, 2010 at 10:58 AM, nirajshr  wrote:
> >
> > > Particularly, I am using the SerializeToFileDescriptor() function to
> > > send the serialized data over a socket interface. The problem is that
> > > the data does not get sent before I explicitly close the socket using
> > > close(fileDescriptor). I even tried letting the google protocol buffer
> > > object go out of scope (hoping the destructor would be called, and
> > > hence flushing of the data). I am sending large amounts of data
> > > (around 1MB). I did not face this problem while sending smaller amount
> > > of data.
> >
> > > Isn't there any way I can force flushing of data without destroying
> > > the google protocol buffer object?
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> >
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] How to retrieve parameters using tag numbers using CodedInputStream

[Jason Hsueh]

Suppose those strings are all defined as optional fields in the message.
They may not be present in the serialization, so the decoder needs to read a
tag to determine the field number and type before handling the value.

On Wed, Aug 11, 2010 at 8:37 AM, Prakash Rao wrote:

> Hi,
> I was just looking at CodedInputStream class and there are APIs to
> write primitives with or without tag (for example, writeString &
> writeStringNoTag). I thought if we set unique tags for multiple
> parameters then we would be able to retrieve these parameters using
> tag numbers specified using CodedInputStream class. I don't see any
> APIs to read values based on tags in CodedInputStream. Am I missing
> something? For the time being I’m using pushLimit & popLimit APIs to
> get different parameter values (based on serializedSize).
>
> For example,
> int test(String s1, String s2, String s3) {
> }
>
> CodedOutputStream out = CodedOutputStream.newIInstance("Some Output
> Stream");
> out.writeString(1, s1);
> out.writeString(2, s2);
> out.writeString(3, s3);
> out.flush();
>
> Why these are not possible?
> CodedInputStream in = CodedInputStream.newIInstance("Some Input
> Stream");
> String s1 = in.readString(1);
> String s2 = in.readString(2);
> String s3 = in.readString(3);
>
> Regards,
> Prakash
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Scala protocol buffers protoc plugin

[Jason Hsueh]

Added to the wiki, thanks!

On Wed, Aug 11, 2010 at 12:57 AM, Jeff Plaisance wrote:

> Hello,
>
> I have written a compiler plugin that generates type safe scala
> wrappers for the java protoc output.  It creates wrappers for each
> message type that use the scala Option type for all optional values.
> It also forces all required parameters to be included in the
> constructor of the builder so that calling build will never throw an
> exception.
>
> here is a link to the repository:
> http://github.com/jeffplaisance/scala-protobuf
>
> here is a short description of the output:
>
> http://wiki.github.com/jeffplaisance/scala-protobuf/what-you-get-from-scala-protobuf
>
> Jeff
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] is option optimized_for=SPEED default in java PB 2.3.0

[Jason Hsueh]

Yes, that is now the default. The tradeoff is that the generated code is
larger.

On Wed, Aug 11, 2010 at 1:14 AM, Prakash Rao wrote:

> I was just reading few discussions on option optimized_for=SPEED. Is
> this default in java protocol buffer 2.3.0 or should I set this option
> explicitly in my .proto files to take advantage of it?
>
> And is there any caveats in using optimized_for=SPEED?
>
> Regards,
> Prakash
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.