[protobuf] intern strings in java

[Kalle Kärkkäinen]

Hi,

I'm modeling contracts that need to reside in a group. Groups are
strings, basically.

I've got rather many contracts, and I was thinking if it was possible
to have strings interned so they would take less memory. Since this
would be a load time change (load string up, intern it, and use what
ever that produced) I see little chance of getting it done good with
option extensions. (which I think are really just tags??)

It would be difficult (although not impossible) to model groups as
numbered-ids and have different message for them.

In my domain, I'd be left with the bigger problem of handling
uniqueness of the id (name would be superficially easier to guarantee
unique).

What say you? Would interned strings be reasonable java enhancement?
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Adding large binary attachments to protobuf messages

[duselbaer]

Hi there,

we're using protobuf messages here to do some kind of RPC over HTTP.
Now we have the need for a service which returns either a metadata
structure or a message containing some large binary data.

The caller does not know the return type upon request instanciation
(eg. a call getContent(pathname) were pathname is either a directory
or a file - for directory a list of files is returned and for files
the content is returned - packed into the same structure). No problem
so far. It would be nice to add an attachment without reading its
content into memory.

But: these binary contents will become very large (hundreds of MBytes)
and as I understood the documentation protobuf is designed to transfer
large binary data (and as the Java developers told me, reading a byte
array with protobuf creates 3 copies of it in memory).

So what would be a good solution to this?

We thought of the following solutions:

1. Make this a transport issue :)

Replace the buffer in the message which contains the attachment by a
unique identifier which references an attachment. Attachments are
added to the RpcChannel / RpcService implementation as a reference to
some stream instance where they can be read from. At transport level
the attachments will be delivered as parts of a multipart HTTP
request / response identified by a Content-Id header.

Sure - when reading attachments in the wrong order some attachments
have to be read into memory - this should be no problem.

This approach does not add full-fledged support of attachments to
protocol buffers messages because attachments will be stored
externally and methods like 'SerializeToOStream()' will not work
anymore because these don't know anything about attachments.

2. Add attachment support to protocol buffers :)

What about adding a new datatype called 'attachment'? Lets create a
new datatype called Attachment which holds either a reference to a
stream or a pointer to a buffer containing the binary data.

When serializing a message all instances of the Attachment class will
be put into a list and will be appended after the message. The fields
of type attachmend will become some kind of reference to an attachment
(eg. an integer type which marks the attachment index).

Deserializing a message could be done by another method which creates
an instance of Attachment for each referenced attachment (eg.
parseFromStreamWithoutAttachments). An application should query the
existence of an attachment before accessing it. If it does not exist
yet, the application should call 'loadAttachment
(attachment_identifier, Attachment*)' which appends the content of an
attachment to a previously created instance of Attachment (remember:
pointing to a buffer or to a stream). Methods like parseFromStream or
parseFromFile will create Attachment instances automatically and load
the content into memory. Of course there should be another method
which reads from the stream until the last attachment has been read.

Thus we should be able to use all of the common serialization and
parsing methods and if we're interested in handling attachments
efficiently, we should use the parseFromStreamWithoutAttachments and
loadAttachment methods.

I am not sure how attachments of embedded messages could be handled -
should these be serialized at the end of the message itself or at the
end of the containing message(s)? Of course, this requires an instance
of the stream where the attachments could be read from in the
application.

What do you think?

 - Would this be a solution to the attachment issue and would you like
to see this in protocol buffers?
 - How would you deal with the issue?

Regards,

Ronny
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Adding large binary attachments to protobuf messages

[Kenton Varda]

On Tue, Jan 19, 2010 at 4:46 AM, duselbaer wrote:

> (and as the Java developers told me, reading a byte
> array with protobuf creates 3 copies of it in memory).
>

This is not true in the common case.  However, it looks like there is a code
path which creates 3 copies but could fairly easily be improved.  Remember
that if you run into a performance problem with protobufs, it's usually
because no one else has exercised that path before.  Please do not be afraid
to send me patches to improve performance!


> So what would be a good solution to this?
>

Ideally:
1) Extend ByteString so that it can hold its data in multiple pieces instead
of one, big, flat array.  Allocating gigantic flat arrays is dangerous so
should be avoided.
2) Improve the parsing code so that it can parse large ByteStrings without
making multiple copies of the data.
3) Do not count large ByteStrings against the 64MB message size limit.

Then you can use the "bytes" type to store blobs of hundreds of megs with no
problems.  No new features are needed -- this is purely an optimization
problem.

For the first step above, I think ByteString should have an internal class
like:

  private static interface Node {
int size();
byte get(int index);
// other stuff
  }

There should then be two implementations of Node:  One that is a flat array
and implemented as ByteString is currently, and one which is a concatenation
of some set of sub-nodes.  The ByteString itself then just has a single
field which points to the root Node.  Hopefully, this approach would allow
the JIT to inline calls to get() in the common case where ByteStrings are
composed of just one flat array, so that this case does not become
significantly slower than it is now.  This approach also makes it easy to
implement concatenation in O(1) time and substring in O(log n).
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] intern strings in java

[Kenton Varda]

I'm not sure I understand.  I think you are using the word "group" to mean
something different than what it means in the .proto language.  At least, I
hope so, because the "group" in the .proto language is deprecated.

Are you suggesting that the parser should automatically intern every String
that it parses?  That sounds like it could be expensive for those who don't
need it.  Maybe you should re-design your data structure so that it doesn't
end up containing lots of copies of the same strings?

On Tue, Jan 19, 2010 at 1:23 AM, Kalle Kärkkäinen
wrote:

> Hi,
>
> I'm modeling contracts that need to reside in a group. Groups are
> strings, basically.
>
> I've got rather many contracts, and I was thinking if it was possible
> to have strings interned so they would take less memory. Since this
> would be a load time change (load string up, intern it, and use what
> ever that produced) I see little chance of getting it done good with
> option extensions. (which I think are really just tags??)
>
> It would be difficult (although not impossible) to model groups as
> numbered-ids and have different message for them.
>
> In my domain, I'd be left with the bigger problem of handling
> uniqueness of the id (name would be superficially easier to guarantee
> unique).
>
> What say you? Would interned strings be reasonable java enhancement?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: intern strings in java

[Kalle Kärkkäinen]

On 19 tammi, 20:31, Kenton Varda  wrote:
> I'm not sure I understand.  I think you are using the word "group" to mean
> something different than what it means in the .proto language.  At least, I
> hope so, because the "group" in the .proto language is deprecated.

Yes, I've got a domain model containing many to one relation between
'Contract' and 'Group'.

These Groups can be called ContractGroups if you will.

> Are you suggesting that the parser should automatically intern every String
> that it parses?  That sounds like it could be expensive for those who don't
> need it.  Maybe you should re-design your data structure so that it doesn't
> end up containing lots of copies of the same strings?

Usually I've got some 20-40 groups, each containing around 200 - 2000
contracts. Since these Groups are just strings (name of ..) I was
thinking that modeling them via intern would be an option.

Would it be possible to have such an option for specific string field,
something like [intern = true]? This would, of course, only help
runtime overhead, and be of no help on-the-wire.

I'm now moving towards having each contract carry an int, specifying
'id' of such ContractGroup.

>
> On Tue, Jan 19, 2010 at 1:23 AM, Kalle Kärkkäinen
> wrote:
>
>
>
> > Hi,
>
> > I'm modeling contracts that need to reside in a group. Groups are
> > strings, basically.
>
> > I've got rather many contracts, and I was thinking if it was possible
> > to have strings interned so they would take less memory. Since this
> > would be a load time change (load string up, intern it, and use what
> > ever that produced) I see little chance of getting it done good with
> > option extensions. (which I think are really just tags??)
>
> > It would be difficult (although not impossible) to model groups as
> > numbered-ids and have different message for them.
>
> > In my domain, I'd be left with the bigger problem of handling
> > uniqueness of the id (name would be superficially easier to guarantee
> > unique).
>
> > What say you? Would interned strings be reasonable java enhancement?
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com > om>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/protobuf?hl=en.
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: intern strings in java

[Daniel Wright]

I agree that making a Contract have an int specifying the ContractGroup id
looks preferable.  If it's very difficult to generate a unique id for each
ContractGroup, you could consider using a fingerprint of the string -- of
course there's the risk of collisions, but with 64-bit fingerprints that
risk is very low, and with 128-bit fingerprints it's a non-issue (assuming a
decent fingerprint function).

2010/1/19 Kalle Kärkkäinen 

> On 19 tammi, 20:31, Kenton Varda  wrote:
> > I'm not sure I understand.  I think you are using the word "group" to
> mean
> > something different than what it means in the .proto language.  At least,
> I
> > hope so, because the "group" in the .proto language is deprecated.
>
> Yes, I've got a domain model containing many to one relation between
> 'Contract' and 'Group'.
>
> These Groups can be called ContractGroups if you will.
>
> > Are you suggesting that the parser should automatically intern every
> String
> > that it parses?  That sounds like it could be expensive for those who
> don't
> > need it.  Maybe you should re-design your data structure so that it
> doesn't
> > end up containing lots of copies of the same strings?
>
> Usually I've got some 20-40 groups, each containing around 200 - 2000
> contracts. Since these Groups are just strings (name of ..) I was
> thinking that modeling them via intern would be an option.
>
> Would it be possible to have such an option for specific string field,
> something like [intern = true]? This would, of course, only help
> runtime overhead, and be of no help on-the-wire.
>
> I'm now moving towards having each contract carry an int, specifying
> 'id' of such ContractGroup.
>
> >
> > On Tue, Jan 19, 2010 at 1:23 AM, Kalle Kärkkäinen
> > wrote:
> >
> >
> >
> > > Hi,
> >
> > > I'm modeling contracts that need to reside in a group. Groups are
> > > strings, basically.
> >
> > > I've got rather many contracts, and I was thinking if it was possible
> > > to have strings interned so they would take less memory. Since this
> > > would be a load time change (load string up, intern it, and use what
> > > ever that produced) I see little chance of getting it done good with
> > > option extensions. (which I think are really just tags??)
> >
> > > It would be difficult (although not impossible) to model groups as
> > > numbered-ids and have different message for them.
> >
> > > In my domain, I'd be left with the bigger problem of handling
> > > uniqueness of the id (name would be superficially easier to guarantee
> > > unique).
> >
> > > What say you? Would interned strings be reasonable java enhancement?
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Visual Studio Visualizer for Repeated Messages

[Kenton Varda]

We never did anything special to support visualization in MSVC.  I guess it
was somehow smart enough to do it automatically, but after a major
refactoring of repeated_field.h it could no longer figure it out.  In either
2.1.0 or 2.2.0 (I forget), RepeatedPtrField was changed to subclass
internal::RepeatedPtrFieldBase, so that most of the implementation could
avoid template overhead.  I guess MSVC didn't like that?

Since I'm not as MSVC user I don't really have any idea what to do about
this.

On Mon, Jan 18, 2010 at 6:23 AM, naderp  wrote:

> Hi
>
> Back in 2.0.3 I was able to visualise repeated messages under the VS C+
> + 2008 debugger. In 2.2.0 they are no longer inspectable.
>
> Can anyone shed any light on this or maybe submit a visualiser?
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: A new ActionScript 3 protobuf implementation

[elenzil]

Hi Atry -

i am curious to try out your new protobuf plugin.
however i encountered a problem running the plugin,
and also a problem reporting a new issue on the protoc-gen-as3
homepage,
so i'm putting the report here.

best,
Orion

issue report:

What steps will reproduce the problem?
1. download and build the 2.3.1 repo of protoc
2. download and extract protoc-gen-as3-0.8rc2-bin.tar.gz
3. run protoc with the following command line:
/home/orion/sandbox/orion/pb2.3.0/linux/bin/protoc --plugin=protoc-gen-
as3=/home/orion/sandbox/orion/pb2.3.0/protoc-gen-as3/dist/protoc-gen-
as3 --as3_out foo ./xs1.proto
(full paths included in case i'm doing something dumb)

What is the expected output? What do you see instead?
the result is the following error:
-
/home/orion/sandbox/orion/pb2.3.0/protoc-gen-as3/dist/protoc-gen-as3:
program not found or is not executable
--as3_out: protoc-gen-as3: Plugin failed with status code 1.
-

if i remove "-e" from the first line of protoc-gen-as3,
then the error changes to:
-
Exception in thread "main" java.lang.NoClassDefFoundError: com/netease/
protocGenAs3/Main
Caused by: java.lang.ClassNotFoundException:
com.netease.protocGenAs3.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
Could not find the main class: com/netease/protocGenAs3/Main. Program
will exit.
--as3_out: protoc-gen-as3: Plugin failed with status code 1.
-


What version of the product are you using? On what operating system?

these results are consistent on OS X (snow leopard) and on a centos
linux box:
Linux version 2.6.18-164.6.1.el5 (mockbu...@builder10.centos.org)



Please provide any additional information below.


-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] intern strings in java

[Oliver Jowett]

2010/1/20 Kenton Varda :
> Are you suggesting that the parser should automatically intern every String
> that it parses?  That sounds like it could be expensive for those who don't
> need it.

More than "expensive" - it's a permanent memory leak for any app that
passes around many unique strings. It'd be a showstopper for our app.

-O
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Issue 155 in protobuf: Compiler complains with 'error: extra qualification'

[protobuf]

Updates:
Status: Fixed

Comment #1 on issue 155 by ken...@google.com: Compiler complains  
with 'error: extra qualification'

http://code.google.com/p/protobuf/issues/detail?id=155

How strange.  That never should have compiled, but it passed testing on  
MSVC, MinGW,

and Cygwin (GCC 3.x for the latter two).

I've fixed this in revision 308.

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Issue 157 in protobuf: win32's subprocess.cc doesn't compile if UNICODE is defined.

[protobuf]

Updates:
Status: Fixed
Labels: FixedIn-2.3.1

Comment #1 on issue 157 by ken...@google.com: win32's subprocess.cc doesn't  
compile if UNICODE is defined.

http://code.google.com/p/protobuf/issues/detail?id=157

Your revision 305 fixed this.  Thanks!

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Issue 159 in protobuf: 2.3.0 build fails under QNX fails

[protobuf]

Updates:
Status: Fixed
Labels: FixedIn-2.3.1

Comment #1 on issue 159 by ken...@google.com: 2.3.0 build fails under QNX  
fails

http://code.google.com/p/protobuf/issues/detail?id=159

Thanks, committed as revision 309.

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] shared library opened with dlopen, crash in google::protobuf::MessageFactory::InternalRegisterGeneratedFile

[Kenton Varda]

Can you create a small, self-contained program that demonstrates this bug?
 Ideally it would contain a simple .proto file, a simple .cc file for the
dynamic library, a simple .cc file with a main() function that dlopens the
library, and a simple Makefile to build it all.  Each of these files should
be no more than 5-10 lines.

On Fri, Jan 15, 2010 at 2:55 AM, Cosmin Cremarenco <
cosmin.cremare...@gmail.com> wrote:

> Hi all
> I'm on Solaris 10 x86, Sun Studio 12.1, protobuf 2.1.0.
> I think this is linked to Issue 128:
> http://code.google.com/p/protobuf/issues/detail?id=128
>
> I'm doing dlopen/dlclose/dlopen on a shared library that contains pb
> generated classes.
> Given the words of advice in Issue 128 I configured with --disable-
> shared, compiled protobuf and linked statically with libprotobuf.a
> At the second dlopen I get a SEGV with the callstack below.
>
> I cannot use the lite runtime because we need extensions.
>
> Thanks for your help,
>
> Cosmin
>
> t...@1 (l...@1) terminated by signal SEGV (no mapping at the fault address)
> 0xfdbe4ad8: strcmp+0x0108:  movl (%esi),%eax
> Current function is
>
> pb::murex::system::exception::protobuf_AddDesc_pb_2fmurex_2fsystem_2fexception_2fMXExceptionLevel_2eproto
>
> 84 ::google::protobuf::MessageFactory::InternalRegisterGeneratedFile
> (
> (dbx) where
> current thread: t...@1
>  [1] strcmp(0xfca84790, 0xfc481ec4), at 0xfdbe4ad8
>  [2] std::hashtable,const
> char*,google::protobuf::hash char*>,std::_Select1st
> >,google::protobuf::streq,std::allocator char*const,void(*)()> > >::insert_unique_noresize(0x8045140,
> 0xfe6e0ebc, 0x8045138), at 0xfe58026e
>  [3]
> google::protobuf::InsertIfNotPresent char*,void(*)(),google::protobuf::hash char*>,google::protobuf::streq>,const char*,void(*)()>(0xfe6e0ebc,
> 0x80451ac, 0x80451b0), at 0xfe57f39a
>  [4]
>
> google::protobuf::__unnamed_sWhGbtftTLGgu::GeneratedMessageFactory::RegisterFile
> (0xfe6e0eb8, 0xfc481ec4, 0xfc2c3450), at 0xfe57d83e
>  [5] google::protobuf::MessageFactory::InternalRegisterGeneratedFile
> (0xfc481ec4, 0xfc2c3450), at 0xfe57df86
> =>[6]
>
> pb::murex::system::exception::protobuf_AddDesc_pb_2fmurex_2fsystem_2fexception_2fMXExceptionLevel_2eproto
> (), line 84 in "MXExceptionLevel.pb.cc"
>  [7]
>
> pb::murex::system::exception::protobuf_AddDesc_pb_2fmurex_2fsystem_2fexception_2fMXExceptionAdapter_2eproto
> (), line 81 in "MXExceptionAdapter.pb.cc"
>  [8]
>
> pb::murex::system::exception::StaticDescriptorInitializer_pb_2fmurex_2fsystem_2fexception_2fMXExceptionAdapter_2eproto::StaticDescriptorInitializer_pb_2fmurex_2fsystem_2fexception_2fMXExceptionAdapter_2eproto
> (this = 0xfc481560), line 107 in "MXExceptionAdapter.pb.cc"
>  [9] __SLIP.INIT_A(), line 109 in "MXExceptionAdapter.pb.cc"
>  [10] __STATIC_CONSTRUCTOR(), line 109 in "MXExceptionAdapter.pb.cc"
>  [11] __cplus_fini_at_exit(0xfe7a0150, 0xfcae0ee8, 0xfeffa840,
> 0xfcae0eec, 0xfcae0fac, 0xfc428e50), at 0xfc428f0b
>  [12] call_init(0xfcae0ee8, 0x1), at 0xfefd58fc
>  [13] load_completion(0xfcae0590, 0xfe7a0150), at 0xfefd5db9
>  [14] dlmopen_intn(0xfeffa220, 0x8140100, 0xd01, 0xfe7a0150, 0x0,
> 0x0, 0x8045314), at 0xfefd9b6d
>  [15] dlmopen_check(0xfeffa220, 0x8140100, 0xd01, 0xfe7a0150,
> 0x8045314), at 0xfefd9c3f
>  [16] _dlopen(0x8140100, 0x101), at 0xfefd9d2b
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] shared library opened with dlopen, crash in google::protobuf::MessageFactory::InternalRegisterGeneratedFile

[Kenton Varda]

Oh, and please verify that the problem still exists with 2.3.0 before doing
any of this.

Also, if the problem is Solaris-specific I won't be able to help you, as I
don't have access to this platform.  But if I can repro on Linux I will try
to fix it.

On Tue, Jan 19, 2010 at 6:07 PM, Kenton Varda  wrote:

> Can you create a small, self-contained program that demonstrates this bug?
>  Ideally it would contain a simple .proto file, a simple .cc file for the
> dynamic library, a simple .cc file with a main() function that dlopens the
> library, and a simple Makefile to build it all.  Each of these files should
> be no more than 5-10 lines.
>
> On Fri, Jan 15, 2010 at 2:55 AM, Cosmin Cremarenco <
> cosmin.cremare...@gmail.com> wrote:
>
>> Hi all
>> I'm on Solaris 10 x86, Sun Studio 12.1, protobuf 2.1.0.
>> I think this is linked to Issue 128:
>> http://code.google.com/p/protobuf/issues/detail?id=128
>>
>> I'm doing dlopen/dlclose/dlopen on a shared library that contains pb
>> generated classes.
>> Given the words of advice in Issue 128 I configured with --disable-
>> shared, compiled protobuf and linked statically with libprotobuf.a
>> At the second dlopen I get a SEGV with the callstack below.
>>
>> I cannot use the lite runtime because we need extensions.
>>
>> Thanks for your help,
>>
>> Cosmin
>>
>> t...@1 (l...@1) terminated by signal SEGV (no mapping at the fault address)
>> 0xfdbe4ad8: strcmp+0x0108:  movl (%esi),%eax
>> Current function is
>>
>> pb::murex::system::exception::protobuf_AddDesc_pb_2fmurex_2fsystem_2fexception_2fMXExceptionLevel_2eproto
>>
>> 84 ::google::protobuf::MessageFactory::InternalRegisterGeneratedFile
>> (
>> (dbx) where
>> current thread: t...@1
>>  [1] strcmp(0xfca84790, 0xfc481ec4), at 0xfdbe4ad8
>>  [2] std::hashtable,const
>> char*,google::protobuf::hash> char*>,std::_Select1st
>> >,google::protobuf::streq,std::allocator> char*const,void(*)()> > >::insert_unique_noresize(0x8045140,
>> 0xfe6e0ebc, 0x8045138), at 0xfe58026e
>>  [3]
>> google::protobuf::InsertIfNotPresent> char*,void(*)(),google::protobuf::hash> char*>,google::protobuf::streq>,const char*,void(*)()>(0xfe6e0ebc,
>> 0x80451ac, 0x80451b0), at 0xfe57f39a
>>  [4]
>>
>> google::protobuf::__unnamed_sWhGbtftTLGgu::GeneratedMessageFactory::RegisterFile
>> (0xfe6e0eb8, 0xfc481ec4, 0xfc2c3450), at 0xfe57d83e
>>  [5] google::protobuf::MessageFactory::InternalRegisterGeneratedFile
>> (0xfc481ec4, 0xfc2c3450), at 0xfe57df86
>> =>[6]
>>
>> pb::murex::system::exception::protobuf_AddDesc_pb_2fmurex_2fsystem_2fexception_2fMXExceptionLevel_2eproto
>> (), line 84 in "MXExceptionLevel.pb.cc"
>>  [7]
>>
>> pb::murex::system::exception::protobuf_AddDesc_pb_2fmurex_2fsystem_2fexception_2fMXExceptionAdapter_2eproto
>> (), line 81 in "MXExceptionAdapter.pb.cc"
>>  [8]
>>
>> pb::murex::system::exception::StaticDescriptorInitializer_pb_2fmurex_2fsystem_2fexception_2fMXExceptionAdapter_2eproto::StaticDescriptorInitializer_pb_2fmurex_2fsystem_2fexception_2fMXExceptionAdapter_2eproto
>> (this = 0xfc481560), line 107 in "MXExceptionAdapter.pb.cc"
>>  [9] __SLIP.INIT_A(), line 109 in "MXExceptionAdapter.pb.cc"
>>  [10] __STATIC_CONSTRUCTOR(), line 109 in "MXExceptionAdapter.pb.cc"
>>  [11] __cplus_fini_at_exit(0xfe7a0150, 0xfcae0ee8, 0xfeffa840,
>> 0xfcae0eec, 0xfcae0fac, 0xfc428e50), at 0xfc428f0b
>>  [12] call_init(0xfcae0ee8, 0x1), at 0xfefd58fc
>>  [13] load_completion(0xfcae0590, 0xfe7a0150), at 0xfefd5db9
>>  [14] dlmopen_intn(0xfeffa220, 0x8140100, 0xd01, 0xfe7a0150, 0x0,
>> 0x0, 0x8045314), at 0xfefd9b6d
>>  [15] dlmopen_check(0xfeffa220, 0x8140100, 0xd01, 0xfe7a0150,
>> 0x8045314), at 0xfefd9c3f
>>  [16] _dlopen(0x8140100, 0x101), at 0xfefd9d2b
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to proto...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> protobuf+unsubscr...@googlegroups.com
>> .
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>>
>>
>>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Issue 156 in protobuf: extension_set.h needs GetMessage / GetMessageA / GetMessageW workaround

[protobuf]

Updates:
Status: Fixed
Labels: FixedIn-2.3.1

Comment #1 on issue 156 by ken...@google.com: extension_set.h needs  
GetMessage / GetMessageA / GetMessageW workaround

http://code.google.com/p/protobuf/issues/detail?id=156

Your revision 305 fixed this.  Thanks!

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Issue 155 in protobuf: Compiler complains with 'error: extra qualification'

[protobuf]

Updates:
Labels: FixedIn-2.3.1

Comment #2 on issue 155 by ken...@google.com: Compiler complains  
with 'error: extra qualification'

http://code.google.com/p/protobuf/issues/detail?id=155

(No comment was entered for this change.)

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: Issue 153 in protobuf: 2.3 Python code not importing

[protobuf]

Updates:
Status: CannotReproduce

Comment #5 on issue 153 by ken...@google.com: 2.3 Python code not importing
http://code.google.com/p/protobuf/issues/detail?id=153

(No comment was entered for this change.)

--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Java UTF-8 encoding/decoding: possible performance improvements

[Kenton Varda]

I think the 30-80% speed boost would be well worth the extra code size /
memory overhead.  Please send me the patch!

On Sun, Jan 17, 2010 at 9:33 AM, Evan Jones  wrote:

> I've implemented a rough prototype of an optimization to the Java
> implementation that serializes Java strings once when optimize_for == SPEED,
> rather than twice. Brief overview:
>
> * Add a private volatile byte[] cache for each string field.
> * When computing the serialized size, serialize the string and store it in
> the cache.
> * When serializing, use the cache if available, then set the cache to null.
>
> I used the ProtoBench.java program included in the SVN repository, using
> the messages included in the repository. Brief summary of results:
>
> * Serializing a protocol buffer more than once is possibly slightly slower
> (~2-10%). I'm guessing the reason is that since it already has the message
> length, the extra conditionals and fields for string caching just get in the
> way.
> * Serializing a protocol buffer once, with the length prepended, is
> significantly faster (~33% for SpeedMessage1, with 10 strings, ~80% for
> SpeedMessage2, with lots of strings; the benchmark measures the time to
> create the protocol buffer, so the improvement is probably actually larger).
> * Classes are a bit bigger (SpeedMessage1.class with 8 strings: 24802 ->
> 25577)
> * Size messages are unchanged.
> * Messages without strings are unchanged.
>
>
> Pros:
> * Faster serialization of messages that contain strings.
>
> Cons:
> * More code (extra fields; conditional checking of the cache)
> * More RAM (extra fields)
> * Some changes to CodedOutputStream (more code to handle cached strings).
>
> Does this seem like a worthwhile optimization? If so, I'll clean up my
> patch a bit and submit it for code review. Thanks,
>
>
> Evan
>
> --
> Evan Jones
> http://evanjones.ca/
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

[protobuf] Re: STL_HASH.m4

[vikram]

Thanks Kenton,  I configured correctly using following configure
string

./configure CC="/compiler/xlcpp/usr/vac/bin/xlc_r " CXX="/compiler/
xlcpp/usr/vacpp/bin/xlC_r" CXXFLAGS="-g -qlanglvl=extended -
D__IBMCPP_TR1__ -qidirfirst -I/compiler/xlcpp/usr/vacpp/include "
CFLAGS="-g -qlanglvl=extc99"

Configure detects unordered_map correctly and uses that but when I
tried with simple proto file I got following error
bash-3.00$ ./lt-protoc -I. test.proto --cpp_out=.
test.proto:4:12: "string" is not defined.
test.proto:5:12: "int32" is not defined.
test.proto:6:12: "int32" is not defined.

test.proto

package tutorial;

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional int32 result_per_page = 3;
}



Its seems like descriptor.cc hold kTypeTonName map which identifies
basic google protocol buffer supported datatypes but compiled compiler
could not figure it out.
It seems like function using this array is never called when I
debugged

Function from descriptor.cc
void FieldDescriptor::DebugString(int depth, string *contents) const {

Please provide some idea on this

Thanks & Regards,
Vikram
On Jan 13, 2:17 pm, Kenton Varda  wrote:
> stl_hash.m4 should automatically look it whatever directory your compiler
> uses.  If for some reason your compiler does not automatically look in the
> directory you want, then you should add the proper CXXFLAGS to make it look
> there, e.g.:
>
>   ./configure CXXFLAGS=-I/XYZ/vacpp/include
>
> (-I is GCC's flag for this; your compiler may be different.)
>
> On Wed, Jan 13, 2010 at 12:20 PM, vikram  wrote:
> > Hello Guys,
>
> >     I am seeing that google protocol buffer is now supporting
> > unorderd_map with new modification in hash.h . But I am confused where
> > exactly stl_hash.m4 looks for unordered_map by default . Can we make
> > it to look in different directly as xlc compiler on AIX is installed
> > under XYZ/vacpp/include which is different that default /usr/include
> > directory?
>
> > I tried to run m4 with stl_hash.m4 as input and XYZ/vacpp/include as
> > include directory but it failed. saying " end quote is not provided"
> > Is there anyway I can make stl_hash.m4 to look into
> > different include file than /usr/include
>
> > Thanks & Regards,
> > Vikram
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/protobuf?hl=en.
-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: STL_HASH.m4

[Kenton Varda]

This sounds like another problem with your compiler -- it can't find
std::string.

Note that in common.h we use "using namespace std;" to import all of std
into the google::protobuf namespace.  This is not good practice but we
didn't think it was worth the effort to "fix" it.

On Tue, Jan 19, 2010 at 6:42 PM, vikram  wrote:

> Thanks Kenton,  I configured correctly using following configure
> string
>
> ./configure CC="/compiler/xlcpp/usr/vac/bin/xlc_r " CXX="/compiler/
> xlcpp/usr/vacpp/bin/xlC_r" CXXFLAGS="-g -qlanglvl=extended -
> D__IBMCPP_TR1__ -qidirfirst -I/compiler/xlcpp/usr/vacpp/include "
> CFLAGS="-g -qlanglvl=extc99"
>
> Configure detects unordered_map correctly and uses that but when I
> tried with simple proto file I got following error
> bash-3.00$ ./lt-protoc -I. test.proto --cpp_out=.
> test.proto:4:12: "string" is not defined.
> test.proto:5:12: "int32" is not defined.
> test.proto:6:12: "int32" is not defined.
>
> test.proto
>
> package tutorial;
>
> message SearchRequest {
>  required string query = 1;
>  optional int32 page_number = 2;
>  optional int32 result_per_page = 3;
> }
>
>
>
> Its seems like descriptor.cc hold kTypeTonName map which identifies
> basic google protocol buffer supported datatypes but compiled compiler
> could not figure it out.
> It seems like function using this array is never called when I
> debugged
>
> Function from descriptor.cc
> void FieldDescriptor::DebugString(int depth, string *contents) const {
>
> Please provide some idea on this
>
> Thanks & Regards,
> Vikram
> On Jan 13, 2:17 pm, Kenton Varda  wrote:
> > stl_hash.m4 should automatically look it whatever directory your compiler
> > uses.  If for some reason your compiler does not automatically look in
> the
> > directory you want, then you should add the proper CXXFLAGS to make it
> look
> > there, e.g.:
> >
> >   ./configure CXXFLAGS=-I/XYZ/vacpp/include
> >
> > (-I is GCC's flag for this; your compiler may be different.)
> >
> > On Wed, Jan 13, 2010 at 12:20 PM, vikram  wrote:
> > > Hello Guys,
> >
> > > I am seeing that google protocol buffer is now supporting
> > > unorderd_map with new modification in hash.h . But I am confused where
> > > exactly stl_hash.m4 looks for unordered_map by default . Can we make
> > > it to look in different directly as xlc compiler on AIX is installed
> > > under XYZ/vacpp/include which is different that default /usr/include
> > > directory?
> >
> > > I tried to run m4 with stl_hash.m4 as input and XYZ/vacpp/include as
> > > include directory but it failed. saying " end quote is not provided"
> > > Is there anyway I can make stl_hash.m4 to look into
> > > different include file than /usr/include
> >
> > > Thanks & Regards,
> > > Vikram
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> >
> > > .
> > > For more options, visit this group at
> > >http://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: WriteDelimited/parseDelimited in python

[Kenton Varda]

On Tue, Jan 5, 2010 at 3:57 AM, Graham Cox  wrote:

> I was saying the user *could* do that, and that it's currently what I'm
> doing in my server-side code. The reason being, as you said, if you naively
> read from a stream and the message isn't all present then you need to block
> until it is with the way that the Java code works at present. If you are
> using it for client-side code then likely this is not an issue in the
> slightest, but a server that needs to be able to handle many clients at once
> just can not block on one of them...
>

Well, you should be able to read just the size prefix, then wait until all
the bytes have arrived before attempting to parse.  If the size prefix
itself has not completely arrived, then you would have to retry reading it.


> As to your other alternative, (a), I would suggest that this leaves too
> much of the underlying network protocol bare to the caller. This will make
> it very difficult to change the way that delimiting messages happens in the
> future should such a thing be required. If - for example - it is decided to
> go from having the length prefixed to having a special delimiting sequence
> after the message then it will cause all current calling code to need to be
> changed. It might be that this is considered a low enough level library that
> this is acceptable, but that would be a Google decision...
>

Realistically, we would never be able to change the encoding like this
without requiring some sort of update to the callers, because otherwise we'd
be breaking all the callers' abilities to talk to code compiled before the
change, which is unacceptable.


> One more alternative would be how the asn1c library works for parsing ASN.1
> streams into objects, which is to be resumable. The decoder reads all the
> data it is given, and tries to build the object from this. If it doesn't
> have enough data yet then it does what it can, remembers where it got to and
> returns back to the user who can then supply more data when it becomes
> available. If the entire message does parse from the data provided then
> return back to the user the amount of data consumed so that they can discard
> this (reading from the stream directly makes this slightly cleaner still).
> At present, the Protobuf libraries (any of them) can not support this method
> of decoding an object, and it is not a trivial change to make it possible to
> do, but it does - IMO - give a much cleaner and easier to use method of use.
>

I think this would add far too much complication to the system, probably
harming performance and increasing code size and memory usage.

Maybe we want is a class called DelemitedMessageReader which has a method
Add(bytes).  You read some bytes off the wire, then pass it to Add().  Add()
returns a list of byte strings representing messages that are now complete.
 It may return an empty list if no new messages were completed.  So you have
a loop like:

  while True:
bytes = ReadSomeBytes()
messages = reader.Add(bytes)
for message in messages:
  parsed = MyProtobuf()
  parsed.ParseFromString(message)
  HandleMessage(parsed)

If you want to do event-driven I/O, you simply remove the "while True:" and
instead execute this code each time you detect that bytes are available on
the input.


> --
> Graham Cox
>
> On Tue, Jan 5, 2010 at 1:32 AM, Kenton Varda  wrote:
>
>> Make sure to "reply all" so that the group is CC'd.
>>
>> So you are saying that the user should read whatever data is on the
>> socket, then attempt to parse it, and if it fails, assume that it's because
>> there is more data to read?  Seems rather wasteful.  I think what we ideally
>> want is either:
>> (a) Provide a way for the caller to read the size independently, so that
>> they can then make sure to read that many bytes from the input before
>> parsing.
>> (b) Provide a method that reads from a stream, so that the protobuf
>> library can automatically take care of reading all necessary bytes.
>>
>> Option (b) is obviously cleaner but has a few problems:
>> - We have to choose a particular stream interface to support.  While the
>> Python "file-like" interface is pretty common I'm not sure if it's universal
>> for this kind of task.
>> - If not all bytes of the message are available yet, we'd have to block.
>>  This might be fine most of the time, but would be unacceptable for some
>> uses.
>>
>> Thoughts?
>>
>> On Mon, Jan 4, 2010 at 3:09 PM, Graham Cox  wrote:
>>
>>> I'm using it for reading/writing to sockets in my functional tests -
>>> works well enough there...
>>> In my Java-side server code, I read from the socket into a byte buffer,
>>> then deserialize the byte buffer into Protobuf objects, throwing away the
>>> data that has been deserialized. The python "MergeDelimitedFromString"
>>> function also returns the number of bytes that were processed to build up
>>> the Protobuf object, so the user could easily do the same - read the socket
>>> onto the end of a buffer, and th

Re: [protobuf] Re: STL_HASH.m4

[Kenton Varda]

Wait, I misread your error report.  It looks like the errors are coming from
protoc.  However, the errors are very odd -- it appears that protoc is
failing to recognize built-in types like "string" and "int32".  This could
happen if the hash_map/unordered_map implementation is broken and not
properly matching string keys.

On Tue, Jan 19, 2010 at 6:53 PM, Kenton Varda  wrote:

> This sounds like another problem with your compiler -- it can't find
> std::string.
>
> Note that in common.h we use "using namespace std;" to import all of std
> into the google::protobuf namespace.  This is not good practice but we
> didn't think it was worth the effort to "fix" it.
>
> On Tue, Jan 19, 2010 at 6:42 PM, vikram  wrote:
>
>> Thanks Kenton,  I configured correctly using following configure
>> string
>>
>> ./configure CC="/compiler/xlcpp/usr/vac/bin/xlc_r " CXX="/compiler/
>> xlcpp/usr/vacpp/bin/xlC_r" CXXFLAGS="-g -qlanglvl=extended -
>> D__IBMCPP_TR1__ -qidirfirst -I/compiler/xlcpp/usr/vacpp/include "
>> CFLAGS="-g -qlanglvl=extc99"
>>
>> Configure detects unordered_map correctly and uses that but when I
>> tried with simple proto file I got following error
>> bash-3.00$ ./lt-protoc -I. test.proto --cpp_out=.
>> test.proto:4:12: "string" is not defined.
>> test.proto:5:12: "int32" is not defined.
>> test.proto:6:12: "int32" is not defined.
>>
>> test.proto
>>
>> package tutorial;
>>
>> message SearchRequest {
>>  required string query = 1;
>>  optional int32 page_number = 2;
>>  optional int32 result_per_page = 3;
>> }
>>
>>
>>
>> Its seems like descriptor.cc hold kTypeTonName map which identifies
>> basic google protocol buffer supported datatypes but compiled compiler
>> could not figure it out.
>> It seems like function using this array is never called when I
>> debugged
>>
>> Function from descriptor.cc
>> void FieldDescriptor::DebugString(int depth, string *contents) const {
>>
>> Please provide some idea on this
>>
>> Thanks & Regards,
>> Vikram
>> On Jan 13, 2:17 pm, Kenton Varda  wrote:
>> > stl_hash.m4 should automatically look it whatever directory your
>> compiler
>> > uses.  If for some reason your compiler does not automatically look in
>> the
>> > directory you want, then you should add the proper CXXFLAGS to make it
>> look
>> > there, e.g.:
>> >
>> >   ./configure CXXFLAGS=-I/XYZ/vacpp/include
>> >
>> > (-I is GCC's flag for this; your compiler may be different.)
>> >
>> > On Wed, Jan 13, 2010 at 12:20 PM, vikram  wrote:
>> > > Hello Guys,
>> >
>> > > I am seeing that google protocol buffer is now supporting
>> > > unorderd_map with new modification in hash.h . But I am confused where
>> > > exactly stl_hash.m4 looks for unordered_map by default . Can we make
>> > > it to look in different directly as xlc compiler on AIX is installed
>> > > under XYZ/vacpp/include which is different that default /usr/include
>> > > directory?
>> >
>> > > I tried to run m4 with stl_hash.m4 as input and XYZ/vacpp/include as
>> > > include directory but it failed. saying " end quote is not provided"
>> > > Is there anyway I can make stl_hash.m4 to look into
>> > > different include file than /usr/include
>> >
>> > > Thanks & Regards,
>> > > Vikram
>> >
>> > > --
>> > > You received this message because you are subscribed to the Google
>> Groups
>> > > "Protocol Buffers" group.
>> > > To post to this group, send email to proto...@googlegroups.com.
>> > > To unsubscribe from this group, send email to
>> > > protobuf+unsubscr...@googlegroups.com
>> 
>> >
>> > > .
>> > > For more options, visit this group at
>> > >http://groups.google.com/group/protobuf?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to proto...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> protobuf+unsubscr...@googlegroups.com
>> .
>>
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>>
>>
>>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: STL_HASH.m4

[vikram patil]

Hmm

Could  you please point me to code which is responsible for recognizing
built in types ? I will try to debug more . My independent tests which
evaluates unordered_map are working fine. But when I am trying to use it
with protocol buffer it fails.


Thanks & Regards,
Vikram

On Tue, Jan 19, 2010 at 7:48 PM, Kenton Varda  wrote:

> Wait, I misread your error report.  It looks like the errors are coming
> from protoc.  However, the errors are very odd -- it appears that protoc is
> failing to recognize built-in types like "string" and "int32".  This could
> happen if the hash_map/unordered_map implementation is broken and not
> properly matching string keys.
>
>
> On Tue, Jan 19, 2010 at 6:53 PM, Kenton Varda  wrote:
>
>> This sounds like another problem with your compiler -- it can't find
>> std::string.
>>
>> Note that in common.h we use "using namespace std;" to import all of std
>> into the google::protobuf namespace.  This is not good practice but we
>> didn't think it was worth the effort to "fix" it.
>>
>> On Tue, Jan 19, 2010 at 6:42 PM, vikram  wrote:
>>
>>> Thanks Kenton,  I configured correctly using following configure
>>> string
>>>
>>> ./configure CC="/compiler/xlcpp/usr/vac/bin/xlc_r " CXX="/compiler/
>>> xlcpp/usr/vacpp/bin/xlC_r" CXXFLAGS="-g -qlanglvl=extended -
>>> D__IBMCPP_TR1__ -qidirfirst -I/compiler/xlcpp/usr/vacpp/include "
>>> CFLAGS="-g -qlanglvl=extc99"
>>>
>>> Configure detects unordered_map correctly and uses that but when I
>>> tried with simple proto file I got following error
>>> bash-3.00$ ./lt-protoc -I. test.proto --cpp_out=.
>>> test.proto:4:12: "string" is not defined.
>>> test.proto:5:12: "int32" is not defined.
>>> test.proto:6:12: "int32" is not defined.
>>>
>>> test.proto
>>>
>>> package tutorial;
>>>
>>> message SearchRequest {
>>>  required string query = 1;
>>>  optional int32 page_number = 2;
>>>  optional int32 result_per_page = 3;
>>> }
>>>
>>>
>>>
>>> Its seems like descriptor.cc hold kTypeTonName map which identifies
>>> basic google protocol buffer supported datatypes but compiled compiler
>>> could not figure it out.
>>> It seems like function using this array is never called when I
>>> debugged
>>>
>>> Function from descriptor.cc
>>> void FieldDescriptor::DebugString(int depth, string *contents) const {
>>>
>>> Please provide some idea on this
>>>
>>> Thanks & Regards,
>>> Vikram
>>> On Jan 13, 2:17 pm, Kenton Varda  wrote:
>>> > stl_hash.m4 should automatically look it whatever directory your
>>> compiler
>>> > uses.  If for some reason your compiler does not automatically look in
>>> the
>>> > directory you want, then you should add the proper CXXFLAGS to make it
>>> look
>>> > there, e.g.:
>>> >
>>> >   ./configure CXXFLAGS=-I/XYZ/vacpp/include
>>> >
>>> > (-I is GCC's flag for this; your compiler may be different.)
>>> >
>>> > On Wed, Jan 13, 2010 at 12:20 PM, vikram 
>>> wrote:
>>> > > Hello Guys,
>>> >
>>> > > I am seeing that google protocol buffer is now supporting
>>> > > unorderd_map with new modification in hash.h . But I am confused
>>> where
>>> > > exactly stl_hash.m4 looks for unordered_map by default . Can we make
>>> > > it to look in different directly as xlc compiler on AIX is installed
>>> > > under XYZ/vacpp/include which is different that default /usr/include
>>> > > directory?
>>> >
>>> > > I tried to run m4 with stl_hash.m4 as input and XYZ/vacpp/include as
>>> > > include directory but it failed. saying " end quote is not provided"
>>> > > Is there anyway I can make stl_hash.m4 to look into
>>> > > different include file than /usr/include
>>> >
>>> > > Thanks & Regards,
>>> > > Vikram
>>> >
>>> > > --
>>> > > You received this message because you are subscribed to the Google
>>> Groups
>>> > > "Protocol Buffers" group.
>>> > > To post to this group, send email to proto...@googlegroups.com.
>>> > > To unsubscribe from this group, send email to
>>> > > protobuf+unsubscr...@googlegroups.com
>>> 
>>> >
>>> > > .
>>> > > For more options, visit this group at
>>> > >http://groups.google.com/group/protobuf?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Protocol Buffers" group.
>>> To post to this group, send email to proto...@googlegroups.com.
>>> To unsubscribe from this group, send email to
>>> protobuf+unsubscr...@googlegroups.com
>>> .
>>>
>>> For more options, visit this group at
>>> http://groups.google.com/group/protobuf?hl=en.
>>>
>>>
>>>
>>>
>>
>
-- 

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To post to this group, send email to proto...@googlegroups.com.

To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: STL_HASH.m4

[Kenton Varda]

It just looks up the type name in a hash_map:
http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/parser.cc#1007

kTypeNames
is initialized here:
http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/parser.cc#61

On Tue, Jan 19, 2010 at 7:53 PM, vikram patil  wrote:

> Hmm
>
> Could  you please point me to code which is responsible for recognizing
> built in types ? I will try to debug more . My independent tests which
> evaluates unordered_map are working fine. But when I am trying to use it
> with protocol buffer it fails.
>
>
> Thanks & Regards,
> Vikram
>
>
> On Tue, Jan 19, 2010 at 7:48 PM, Kenton Varda  wrote:
>
>> Wait, I misread your error report.  It looks like the errors are coming
>> from protoc.  However, the errors are very odd -- it appears that protoc is
>> failing to recognize built-in types like "string" and "int32".  This could
>> happen if the hash_map/unordered_map implementation is broken and not
>> properly matching string keys.
>>
>>
>> On Tue, Jan 19, 2010 at 6:53 PM, Kenton Varda  wrote:
>>
>>> This sounds like another problem with your compiler -- it can't find
>>> std::string.
>>>
>>> Note that in common.h we use "using namespace std;" to import all of std
>>> into the google::protobuf namespace.  This is not good practice but we
>>> didn't think it was worth the effort to "fix" it.
>>>
>>> On Tue, Jan 19, 2010 at 6:42 PM, vikram  wrote:
>>>
 Thanks Kenton,  I configured correctly using following configure
 string

 ./configure CC="/compiler/xlcpp/usr/vac/bin/xlc_r " CXX="/compiler/
 xlcpp/usr/vacpp/bin/xlC_r" CXXFLAGS="-g -qlanglvl=extended -
 D__IBMCPP_TR1__ -qidirfirst -I/compiler/xlcpp/usr/vacpp/include "
 CFLAGS="-g -qlanglvl=extc99"

 Configure detects unordered_map correctly and uses that but when I
 tried with simple proto file I got following error
 bash-3.00$ ./lt-protoc -I. test.proto --cpp_out=.
 test.proto:4:12: "string" is not defined.
 test.proto:5:12: "int32" is not defined.
 test.proto:6:12: "int32" is not defined.

 test.proto

 package tutorial;

 message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional int32 result_per_page = 3;
 }



 Its seems like descriptor.cc hold kTypeTonName map which identifies
 basic google protocol buffer supported datatypes but compiled compiler
 could not figure it out.
 It seems like function using this array is never called when I
 debugged

 Function from descriptor.cc
 void FieldDescriptor::DebugString(int depth, string *contents) const {

 Please provide some idea on this

 Thanks & Regards,
 Vikram
 On Jan 13, 2:17 pm, Kenton Varda  wrote:
 > stl_hash.m4 should automatically look it whatever directory your
 compiler
 > uses.  If for some reason your compiler does not automatically look in
 the
 > directory you want, then you should add the proper CXXFLAGS to make it
 look
 > there, e.g.:
 >
 >   ./configure CXXFLAGS=-I/XYZ/vacpp/include
 >
 > (-I is GCC's flag for this; your compiler may be different.)
 >
 > On Wed, Jan 13, 2010 at 12:20 PM, vikram 
 wrote:
 > > Hello Guys,
 >
 > > I am seeing that google protocol buffer is now supporting
 > > unorderd_map with new modification in hash.h . But I am confused
 where
 > > exactly stl_hash.m4 looks for unordered_map by default . Can we make
 > > it to look in different directly as xlc compiler on AIX is installed
 > > under XYZ/vacpp/include which is different that default /usr/include
 > > directory?
 >
 > > I tried to run m4 with stl_hash.m4 as input and XYZ/vacpp/include as
 > > include directory but it failed. saying " end quote is not provided"
 > > Is there anyway I can make stl_hash.m4 to look into
 > > different include file than /usr/include
 >
 > > Thanks & Regards,
 > > Vikram
 >
 > > --
 > > You received this message because you are subscribed to the Google
 Groups
 > > "Protocol Buffers" group.
 > > To post to this group, send email to proto...@googlegroups.com.
 > > To unsubscribe from this group, send email to
 > > protobuf+unsubscr...@googlegroups.com
 
 >
 > > .
 > > For more options, visit this group at
 > >http://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google
 Groups "Protocol Buffers" group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com
 .

 For more options, visit this group at
 http://groups.google.com/group/prot

Re: [protobuf] Re: STL_HASH.m4

[vikram patil]

Thanks Kenton. I will try to debug it and will let you know.  Did anyone
successfully compiled protocol buffer on AIX ? I've seen couple of posts but
never saw some reply with success.

Thanks & Regards,
Vikram

On Tue, Jan 19, 2010 at 7:56 PM, Kenton Varda  wrote:

> It just looks up the type name in a hash_map:
>
> http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/parser.cc#1007
>
>
> kTypeNames
> is initialized here:
>
> http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/parser.cc#61
>
>
> On Tue, Jan 19, 2010 at 7:53 PM, vikram patil wrote:
>
>> Hmm
>>
>> Could  you please point me to code which is responsible for recognizing
>> built in types ? I will try to debug more . My independent tests which
>> evaluates unordered_map are working fine. But when I am trying to use it
>> with protocol buffer it fails.
>>
>>
>> Thanks & Regards,
>> Vikram
>>
>>
>> On Tue, Jan 19, 2010 at 7:48 PM, Kenton Varda  wrote:
>>
>>> Wait, I misread your error report.  It looks like the errors are coming
>>> from protoc.  However, the errors are very odd -- it appears that protoc is
>>> failing to recognize built-in types like "string" and "int32".  This could
>>> happen if the hash_map/unordered_map implementation is broken and not
>>> properly matching string keys.
>>>
>>>
>>> On Tue, Jan 19, 2010 at 6:53 PM, Kenton Varda  wrote:
>>>
 This sounds like another problem with your compiler -- it can't find
 std::string.

 Note that in common.h we use "using namespace std;" to import all of std
 into the google::protobuf namespace.  This is not good practice but we
 didn't think it was worth the effort to "fix" it.

 On Tue, Jan 19, 2010 at 6:42 PM, vikram  wrote:

> Thanks Kenton,  I configured correctly using following configure
> string
>
> ./configure CC="/compiler/xlcpp/usr/vac/bin/xlc_r " CXX="/compiler/
> xlcpp/usr/vacpp/bin/xlC_r" CXXFLAGS="-g -qlanglvl=extended -
> D__IBMCPP_TR1__ -qidirfirst -I/compiler/xlcpp/usr/vacpp/include "
> CFLAGS="-g -qlanglvl=extc99"
>
> Configure detects unordered_map correctly and uses that but when I
> tried with simple proto file I got following error
> bash-3.00$ ./lt-protoc -I. test.proto --cpp_out=.
> test.proto:4:12: "string" is not defined.
> test.proto:5:12: "int32" is not defined.
> test.proto:6:12: "int32" is not defined.
>
> test.proto
>
> package tutorial;
>
> message SearchRequest {
>  required string query = 1;
>  optional int32 page_number = 2;
>  optional int32 result_per_page = 3;
> }
>
>
>
> Its seems like descriptor.cc hold kTypeTonName map which identifies
> basic google protocol buffer supported datatypes but compiled compiler
> could not figure it out.
> It seems like function using this array is never called when I
> debugged
>
> Function from descriptor.cc
> void FieldDescriptor::DebugString(int depth, string *contents) const {
>
> Please provide some idea on this
>
> Thanks & Regards,
> Vikram
> On Jan 13, 2:17 pm, Kenton Varda  wrote:
> > stl_hash.m4 should automatically look it whatever directory your
> compiler
> > uses.  If for some reason your compiler does not automatically look
> in the
> > directory you want, then you should add the proper CXXFLAGS to make
> it look
> > there, e.g.:
> >
> >   ./configure CXXFLAGS=-I/XYZ/vacpp/include
> >
> > (-I is GCC's flag for this; your compiler may be different.)
> >
> > On Wed, Jan 13, 2010 at 12:20 PM, vikram 
> wrote:
> > > Hello Guys,
> >
> > > I am seeing that google protocol buffer is now supporting
> > > unorderd_map with new modification in hash.h . But I am confused
> where
> > > exactly stl_hash.m4 looks for unordered_map by default . Can we
> make
> > > it to look in different directly as xlc compiler on AIX is
> installed
> > > under XYZ/vacpp/include which is different that default
> /usr/include
> > > directory?
> >
> > > I tried to run m4 with stl_hash.m4 as input and XYZ/vacpp/include
> as
> > > include directory but it failed. saying " end quote is not
> provided"
> > > Is there anyway I can make stl_hash.m4 to look into
> > > different include file than /usr/include
> >
> > > Thanks & Regards,
> > > Vikram
> >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Protocol Buffers" group.
> > > To post to this group, send email to proto...@googlegroups.com.
> > > To unsubscribe from this group, send email to
> > > protobuf+unsubscr...@googlegroups.com
> 
> >
> > > .
> > > For more options, visit this group at
>

Re: [protobuf] Re: STL_HASH.m4

[Kenton Varda]

I have no idea.

On Tue, Jan 19, 2010 at 7:58 PM, vikram patil  wrote:

> Thanks Kenton. I will try to debug it and will let you know.  Did anyone
> successfully compiled protocol buffer on AIX ? I've seen couple of posts but
> never saw some reply with success.
>
> Thanks & Regards,
> Vikram
>
>
> On Tue, Jan 19, 2010 at 7:56 PM, Kenton Varda  wrote:
>
>> It just looks up the type name in a hash_map:
>>
>> http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/parser.cc#1007
>>
>>
>> kTypeNames
>> is initialized here:
>>
>> http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/compiler/parser.cc#61
>>
>>
>> On Tue, Jan 19, 2010 at 7:53 PM, vikram patil wrote:
>>
>>> Hmm
>>>
>>> Could  you please point me to code which is responsible for recognizing
>>> built in types ? I will try to debug more . My independent tests which
>>> evaluates unordered_map are working fine. But when I am trying to use it
>>> with protocol buffer it fails.
>>>
>>>
>>> Thanks & Regards,
>>> Vikram
>>>
>>>
>>> On Tue, Jan 19, 2010 at 7:48 PM, Kenton Varda  wrote:
>>>
 Wait, I misread your error report.  It looks like the errors are coming
 from protoc.  However, the errors are very odd -- it appears that protoc is
 failing to recognize built-in types like "string" and "int32".  This could
 happen if the hash_map/unordered_map implementation is broken and not
 properly matching string keys.


 On Tue, Jan 19, 2010 at 6:53 PM, Kenton Varda wrote:

> This sounds like another problem with your compiler -- it can't find
> std::string.
>
> Note that in common.h we use "using namespace std;" to import all of
> std into the google::protobuf namespace.  This is not good practice but we
> didn't think it was worth the effort to "fix" it.
>
> On Tue, Jan 19, 2010 at 6:42 PM, vikram  wrote:
>
>> Thanks Kenton,  I configured correctly using following configure
>> string
>>
>> ./configure CC="/compiler/xlcpp/usr/vac/bin/xlc_r " CXX="/compiler/
>> xlcpp/usr/vacpp/bin/xlC_r" CXXFLAGS="-g -qlanglvl=extended -
>> D__IBMCPP_TR1__ -qidirfirst -I/compiler/xlcpp/usr/vacpp/include "
>> CFLAGS="-g -qlanglvl=extc99"
>>
>> Configure detects unordered_map correctly and uses that but when I
>> tried with simple proto file I got following error
>> bash-3.00$ ./lt-protoc -I. test.proto --cpp_out=.
>> test.proto:4:12: "string" is not defined.
>> test.proto:5:12: "int32" is not defined.
>> test.proto:6:12: "int32" is not defined.
>>
>> test.proto
>>
>> package tutorial;
>>
>> message SearchRequest {
>>  required string query = 1;
>>  optional int32 page_number = 2;
>>  optional int32 result_per_page = 3;
>> }
>>
>>
>>
>> Its seems like descriptor.cc hold kTypeTonName map which identifies
>> basic google protocol buffer supported datatypes but compiled compiler
>> could not figure it out.
>> It seems like function using this array is never called when I
>> debugged
>>
>> Function from descriptor.cc
>> void FieldDescriptor::DebugString(int depth, string *contents) const {
>>
>> Please provide some idea on this
>>
>> Thanks & Regards,
>> Vikram
>> On Jan 13, 2:17 pm, Kenton Varda  wrote:
>> > stl_hash.m4 should automatically look it whatever directory your
>> compiler
>> > uses.  If for some reason your compiler does not automatically look
>> in the
>> > directory you want, then you should add the proper CXXFLAGS to make
>> it look
>> > there, e.g.:
>> >
>> >   ./configure CXXFLAGS=-I/XYZ/vacpp/include
>> >
>> > (-I is GCC's flag for this; your compiler may be different.)
>> >
>> > On Wed, Jan 13, 2010 at 12:20 PM, vikram 
>> wrote:
>> > > Hello Guys,
>> >
>> > > I am seeing that google protocol buffer is now supporting
>> > > unorderd_map with new modification in hash.h . But I am confused
>> where
>> > > exactly stl_hash.m4 looks for unordered_map by default . Can we
>> make
>> > > it to look in different directly as xlc compiler on AIX is
>> installed
>> > > under XYZ/vacpp/include which is different that default
>> /usr/include
>> > > directory?
>> >
>> > > I tried to run m4 with stl_hash.m4 as input and XYZ/vacpp/include
>> as
>> > > include directory but it failed. saying " end quote is not
>> provided"
>> > > Is there anyway I can make stl_hash.m4 to look into
>> > > different include file than /usr/include
>> >
>> > > Thanks & Regards,
>> > > Vikram
>> >
>> > > --
>> > > You received this message because you are subscribed to the Google
>> Groups
>> > > "Protocol Buffers" group.
>> > > To post to this group, send