Re: Sun Studio 11
I haven't heard from the patch author since my previous mail. :/
On Mon, Jun 22, 2009 at 4:06 PM, vikram patilvik...@gmail.com wrote:
Can somebody point out when this patch will be included in source?
I am trying to do compilation on Solaris 10 but right now I am getting
errors as mentioned in this post.
Vikram
On Jun 4, 1:42 pm, Kenton Varda ken...@google.com wrote:
Thanks for the patch!
It looks like you were using the examples as a test. Running make
check
in the top directory will run a much better suite of tests -- do they
pass?
Assuming it does work, can you re-send that patch as an attachment (it
looks
like it has been mangled), or even send it to me via
codereview.appspot.com?
Also, for me to submit it you'll need to sign the contributor license
agreement:
http://code.google.com/legal/individual-cla-v1.0.html-- if you own
copyright on this patchhttp://
code.google.com/legal/corporate-cla-v1.0.html-- if your employer
does
On Wed, Jun 3, 2009 at 10:53 PM, bmcarnes_proto...@oddren.com wrote:
Probably not. LibCstd isn't complete enough for protobuf.
I was able to get protobuf 2.1.0 to work well under Sun Studio 11,
using libCstd, with a few patches. Works well enough to compile and
run the add_person_cpp/list_people_cpp examples.
I include the build instructions and patch set below Also included
are patches to the examples makefile to test it out.
Apply below patchset first against protobuf 2.1.0
$ CXX=CC CC=cc ./configure --disable-shared
$ make
$ cd examples
$ make cpp
$ ./add_person_cpp addressbook.test
(enter a test record)
$ ./list_people_cpp addressbook.test
(test record is displayed correctly)
$ ldd list_people_cpp (or ldd add_person_cpp )
libpthread.so.1 = /lib/libpthread.so.1
libCstd.so.1 = /usr/lib/libCstd.so.1
libCrun.so.1 = /usr/lib/libCrun.so.1
libm.so.2 = /lib/libm.so.2
libc.so.1 = /lib/libc.so.1
Index: src/google/protobuf/repeated_field.h
===
--- src/google/protobuf/repeated_field.h(revision 2)
+++ src/google/protobuf/repeated_field.h(working copy)
@@ -69,7 +69,7 @@
class LIBPROTOBUF_EXPORT GenericRepeatedField {
public:
inline GenericRepeatedField() {}
-#if defined(__DECCXX) defined(__osf__)
+#if defined(__SUNPRO_CC) || defined(__DECCXX) defined(__osf__)
// HP C++ on Tru64 has trouble when this is not defined inline.
virtual ~GenericRepeatedField() {}
#else
@@ -548,7 +548,7 @@
current_size_ = 0;
}
-#if defined(__DECCXX) defined(__osf__)
+#if defined(__SUNPRO_CC) || defined(__DECCXX) defined(__osf__)
// HP C++ on Tru64 has trouble when this is not defined inline.
template
inline void RepeatedPtrFieldstring::Clear() {
Index: src/google/protobuf/descriptor_database.cc
===
--- src/google/protobuf/descriptor_database.cc (revision 2)
+++ src/google/protobuf/descriptor_database.cc (working copy)
@@ -127,7 +127,13 @@
// Insert the new symbol using the iterator as a hint, the new
entry will
// appear immediately before the one the iterator is pointing at.
+
+ // Sun Studio 11 needs a little help here
+#if defined(__SUNPRO_CC)
+ by_symbol_.insert(iter, make_pairconst string,Value(name,
value));
+#else
by_symbol_.insert(iter, make_pair(name, value));
+#endif
return true;
}
Index: src/google/protobuf/compiler/command_line_interface.cc
===
--- src/google/protobuf/compiler/command_line_interface.cc
(revision
2)
+++ src/google/protobuf/compiler/command_line_interface.cc
(working
copy)
@@ -474,7 +474,7 @@
// If no --proto_path was given, use the current working directory.
if (proto_path_.empty()) {
-proto_path_.push_back(make_pair(, .));
+proto_path_.push_back(make_pair(string(), string(.)));
}
// Check some errror cases.
Index: src/google/protobuf/repeated_field.cc
===
--- src/google/protobuf/repeated_field.cc (revision 2)
+++ src/google/protobuf/repeated_field.cc (working copy)
@@ -40,7 +40,7 @@
// HP C++ on Tru64 can't handle the stuff below being defined out-of-
line, so
// on that platform everything is defined in repeated_field.h. On
other
// platforms, we want these to be out-of-line to avoid code bloat.
-#if !defined(__DECCXX) || !defined(__osf__)
+#if !defined(__SUNPRO_CC) (!defined(__DECCXX) || !defined
(__osf__))
namespace internal {
--- Below changes the examples makefile to exercise protobuf under
solaris
Index: examples/Makefile
Re: valgrind issues
Did you call ShutdownProtobufLibrary() before checking for leaks? As it says, the memory in question is still reachable, so whether or not it is a leak is debatable. ShutdownProtobufLibrary() will go around and delete all the objects the library has allocated. It's a huge waste of time if you're about to call exit() anyway, but it should make valgrind shut up. Or maybe you can tell valgrind to ignore reachable memory? On Mon, Jun 22, 2009 at 11:23 PM, Monty Taylor mord...@inaugust.com wrote: Hey guys, We're valgrinding drizzle at the moment and see a lot of: # ==3378== 40 bytes in 1 blocks are still reachable in loss record 14 of 121 # ==3378==at 0x4A06D5C: operator new(unsigned long) (vg_replace_malloc.c:230) # ==3378==by 0x5894A8: drizzled::message::protobuf_BuildDesc_table_2eproto_AssignGlobalDescriptors(google::protobuf::FileDescriptor const*) (table.pb.cc:173) # ==3378==by 0x4C64F13: google::protobuf::DescriptorBuilder::BuildFile(google::protobuf::FileDescriptorProto const, void (*)(google::protobuf::FileDescriptor const*)) (descriptor.cc:2391) # ==3378==by 0x4C65BBF: google::protobuf::DescriptorPool::InternalBuildGeneratedFile(void const*, int, void (*)(google::protobuf::FileDescriptor const*)) (descriptor.cc:1962) # ==3378==by 0x5883B9: _GLOBAL__I_table.pb.cc (table.pb.cc:580) # ==3378==by 0x708675: (within /home/brian/merge/drizzled/drizzled) # ==3378==by 0x411BCA: (within /home/brian/merge/drizzled/drizzled) # ==3378==by 0x3171402B1F: (within /lib64/libpthread-2.9.so) # ==3378==by 0x708604: __libc_csu_init (in /home/brian/merge/drizzled/drizzled) # ==3378==by 0x3170C1E501: (below main) (libc-start.c:179) I'm not sure whether they are valid and something I should report to you, or whether they are invalid and something I should suppress. Any thoughts? Thanks! Monty --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: encoding of embedded messages and repeated elements
The advantage of writing the length is that a parser can skip the entire sub-message easily without having to parse its contents. Otherwise, we would probably use the group encoding for sub-messages, where a special end tag marks the end of the message. On Tue, Jun 23, 2009 at 9:06 AM, etorri e...@torri.be wrote: Hello, The length delimited encoding basically tells that the following N bytes belong to this field. Wouldn't it be easier to instead use the number of elements that belong to the embedded message (repeated element). Now (as far as I have understood) the message needs to be built from fragments and then collected together as the lengths are not known beforehand and it would be expensive to calculate the byte-length of the embedded message. Instead, it would be relatively inexpensive to calculate just the number of following elements that make the embedded message before starting to encode it. This would enable streaming of PB or encoding and sending the elements right as they are encoded. Sorry if I misunderstood something. I have just started looking at BP. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Deserializing a message for which I don't have the .proto (in Java)
You can use UnknownFieldSet, but be warned that the interface for that class is likely to change in a future version (because the current design is somewhat inefficient). If you just want to print the contents, you should be fine -- just parse into an UnknownFieldSet and then call its toString() method. Those parts won't change. On Tue, Jun 23, 2009 at 7:46 AM, Toph clo...@gmail.com wrote: Hi folks, I understand that protocol buffers messages are not fully self- describing. However, the message contains the field number, wire type, and value, right? In Java, how can I take a byte[] (array of bytes) that represents a message and deserialize it into a list of tuples that contain the field number, wire type, and value? I really just want to print out these tuples, even if the value is binary. Do any of the classes i the Java API provide a mechanism to do this? I know I could just take the documentation of the encoding and write code myself, but I am hoping that one of the API classes exists in a usable or near-usable form to do this for me. Will CodedInputStream do it? Can I use any of the parseFrom() or mergeFrom() methods? Note that I do not have the corresponding .proto file in source form, compiled form, or available to transmit along with any messages Thanks --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: New enum values
It will be treated like an unknown field. In C++ or Java, the value goes into the UnknownFieldSet, so if you then serialize the message, it will still be there. In Python it will just be discarded. On Wed, Jun 24, 2009 at 2:56 AM, Stuart Johnson stu...@stu.org.uk wrote: If a new enum value had been added to a enum type, and it's being deserialised by an application that is not yet aware of the new enum value, what should happen? Should the enum value be null, or should the deserialisation abort? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Sun Studio 11
If the tests pass, it should be good to go. Please do submit patches. On Wed, Jun 24, 2009 at 2:09 PM, vikram patilvik...@gmail.com wrote: I was finally able to compile it correctly and evaluate its correctness using tests. Kenton, Is there anything else I need to evaluate before using compiler and binaries. I can provide patch now if its needed. I am also going to try same procedure on AIX too . Is there any other precautions I needed to take before compiling on AIX? If it goes through without problem, I can provide patch which can include changes for both Solaris and AIX . Vikram On Jun 24, 1:40 pm, vikram patilvik...@gmail.com wrote: @Monty I followed your steps but make check still fails with following errors make check Making check in . make check-local Making lib/libgtest.a lib/libgtest_main.a in gtest `lib/libgtest.la' is up to date. `lib/libgtest_main.la' is up to date. Making check in src make check-am make protobuf-test protobuf-lazy-descriptor-test zcgzip zcgunzip source='google/protobuf/compiler/cpp/cpp_unittest.cc' object='protobuf_test-cpp_unittest.o' libtool=no \ DEPDIR=.deps depmode=none /bin/bash ../depcomp \ CC -DHAVE_CONFIG_H -I. -I.. -I../gtest/include -I../gtest/ include -D_REENTRANT -g -DNDEBUG -c -o protobuf_test-cpp_unittest.o `test -f 'google/protobuf/compiler/cpp/cpp_unittest.cc' || echo './'`google/protobuf/compiler/cpp/cpp_unittest.cc ./google/protobuf/descriptor.h, line 324: Warning: Identifier expected instead of }. ./google/protobuf/descriptor.h, line 343: Warning: Identifier expected instead of }. ./google/protobuf/descriptor.h, line 354: Warning: Identifier expected instead of }. ./google/protobuf/io/tokenizer.h, line 107: Warning: Identifier expected instead of }. google/protobuf/compiler/cpp/cpp_unittest.cc, line 120: Warning: A non-POD object of type std::string passed as a variable argument to function testing::internal::IsNullLiteralHelper(...). google/protobuf/compiler/cpp/cpp_unittest.cc, line 442: Error: TestAllTypes is not defined. google/protobuf/compiler/cpp/cpp_unittest.cc, line 457: Error: TestPackedTypes is not defined. google/protobuf/compiler/cpp/cpp_unittest.cc, line 475: Error: TestAllTypes is not defined. google/protobuf/compiler/cpp/cpp_unittest.cc, line 494: Error: TestPackedTypes is not defined. 4 Error(s) and 5 Warning(s) detected. *** Error code 4 make: Fatal error: Command failed for target `protobuf_test- cpp_unittest.o' Current working directory /work/vpatil/protobuf-2.1.0/src *** Error code 1 make: Fatal error: Command failed for target `check-am' Current working directory /work/vpatil/protobuf-2.1.0/src *** Error code 1 make: Fatal error: Command failed for target `check' Current working directory /work/vpatil/protobuf-2.1.0/src *** Error code 1 The following command caused the error: failcom='exit 1'; \ for f in x $MAKEFLAGS; do \ case $f in \ *=* | --[!k]*);; \ *k*) failcom='fail=yes';; \ esac; \ done; \ dot_seen=no; \ target=`echo check-recursive | sed s/-recursive//`; \ list='. src'; for subdir in $list; do \ echo Making $target in $subdir; \ if test $subdir = .; then \ dot_seen=yes; \ local_target=$target-am; \ else \ local_target=$target; \ fi; \ (cd $subdir make $local_target) \ || eval $failcom; \ done; \ if test $dot_seen = no; then \ make $target-am || exit 1; \ fi; test -z $fail make: Fatal error: Command failed for target `check-recursive' --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: encoding of embedded messages and repeated elements
Yes, groups are never going to fully go away. But we recommend against using them in new code. On Thu, Jun 25, 2009 at 9:13 AM, Piotr Findeisen piotr.findei...@gmail.comwrote: Hi! On Jun 24, 8:42 pm, Kenton Varda ken...@google.com wrote: The end-tag approach is more efficient than your idea -- it's faster (no need to count elements at all) and it takes no more space (no need to write a count, which makes up for the extra space taken by the end tag). But in any case, the encoding is not something we can change at this point, since protocol buffers is nothing without backwards-compatibility. As I read the code of C++ protobuf deserializer I found it supports end-tag approach using END_GROUP constant -- or I just misunderstood the code and/or this thread? From my experiments it looks like I can stream messages one by one separating them with END_GROUP tag, but -- again from comments in the code -- it's deprecated. If protocol buffers is nothing without backwards-compatibility, can I assume that existing and future implementation of C++ and also Java/Python deserializers will support this approach? best regards, Piotr --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Warnings with member variables in C++
BTW, this was also reported awhile ago as issue 86:
http://code.google.com/p/protobuf/issues/detail?id=86
On Thu, Jun 25, 2009 at 12:55 PM, Kenton Varda ken...@google.com wrote:
The problem is that GCC emits two or three copies of each constructor. In
order to keep generated code size down, we moved the constructor code into a
separate method which could then be shared by the multiple copies that GCC
emits.
http://gcc.gnu.org/bugs.html#nonbugs_cxx
There is nothing harmful about this -- all fields are still being
initialized properly. It's just happening somewhere other than the
constructor.
I recommend disabling the warning. Perhaps you can disable it just for
.pb.cc files?
On Thu, Jun 25, 2009 at 12:11 PM, dan.schmidt.valle
dan.schmidt.va...@gmail.com wrote:
Hi there,
I'm developing an app in C++ and I use PB for lots of things. I'm also
in an environment that requires me to make sure the code passes very
strict warning levels. I found out when I started using them that PB
didn't include the _unknown_fields_ or any of the repeated field
containers in the member initialisation list. This resulted in me
having warnings all over the place (of the **should be initialized in
the member initialization list** sort), so I amended the
cpp_message.cc method
void MessageGenerator::GenerateInitializerList(io::Printer* printer)
to include the following line: printer-Print(_unknown_fields_
(),\n);
and all the cpp_*_field.cc files to include initialisation for the
repeated files with:
void RepeatedEnumFieldGenerator::GenerateInitializer(io::Printer*
printer) const {
printer-Print(variables_, ,\n$name$_());
}
This was enough up to version 2.0.3. I now see that version 2.1.0 has
removed the initialisation of members from the constructor altogether.
I was wondering what the rationale for that had been, because I've
read that not doing that initialisation can be very harmful.
Thanks a lot.
Dan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: JavaScript
One problem with javascript and protobuf is that you need a lot of support code to parse the messages. Unless you end up sending quite a lot of stuff back and forth, making the user download a JS protobuf codec library may be a net loss. It may be better to use JSON or XML because browsers already have built-in support for those. That said, I think various people inside google have been playing with javascript + protocol buffers for awhile and if we end up with anything that works well enough, we'll release it. On Thu, Jun 25, 2009 at 3:23 PM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: Is there anything in the works that would allow us to output JavaScript along with C++, Java, and Python? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: ProtocolBuffer crash before program execution
The stack trace you provided looks like it is from v2.0.3. The initialization code changed drastically in 2.1.0 -- the method DescriptorPool::InternalBuildGeneratedFile no longer exists, for example. Can you provide a stack trace using 2.1.0? On Fri, Jun 26, 2009 at 8:07 AM, rodrigob rodrigo.benen...@gmail.comwrote: Hello there, I'm usually a linux developer, but I occasionally have to develop some Windows applications. I have a Gtk + Cairo + ProtocolBuffers that was ported to windows some months ago without any problem. Now I have a new one, with similar setup but a few more dependencies (CGAL, RSVG and others). After configuring the visual studio project and getting the code to compile and link correctly the executable crashes miserably on a ProtocolBuffers exception. The crash details can be inspected at http://files.getdropbox.com/u/185965/parking_planning/protobuf_crash.jpg the weird thing is that the application crashes before ever entering in the main, during some static elements initialization defined by Protocol Buffer. I used first protobuf 2.1.0 and then 2.0.3, both crash in the same way. Any idea of what is going on ? I know that Protobuf can work unders VisualStudio, but this time it is not working for me... what could cause this kind of crashes ? The problem seems similar to http://groups.google.com/group/protobuf/browse_thread/thread/f0f8108fdc0de614?pli=1# but the crash source is no the same. I'm using Windows Xp and Visual Studio 2005. Any help would be appreciated. Regards, rodrigob. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Compiling on AIX 5.3 using xlC 3.55 compiler
() from /work/vpatil/protobuf-2.1.0/src/.libs/libprotobuf.a (libprotobuf.so.4) #12 0xd1b39f88 in TryFindFileInFallbackDatabase__Q3_6google8protobuf14DescriptorPoolCFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ () from /work/vpatil/protobuf-2.1.0/src/.libs/libprotobuf.a (libprotobuf.so.4) #13 0xd1b3a2c8 in FindFileByName__Q3_6google8protobuf14DescriptorPoolCFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ () from /work/vpatil/protobuf-2.1.0/src/.libs/libprotobuf.a (libprotobuf.so.4) #14 0xd1d2a0c8 in Import__Q4_6google8protobuf8compiler8ImporterFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ () from /work/vpatil/protobuf-2.1.0/src/.libs/libprotobuf.a (libprotobuf.so.4) #15 0xd1755298 in google::protobuf::compiler::CommandLineInterface::Run () from /work/ vpatil/protobuf-2.1.0/src/.libs/libprotoc.a(libprotoc.so.4) #16 0x10002734 in main () #17 0x11c4 in __start () This one is what I found out when I debugged created binary using gdb. So code is stuck somewhere here. Vikram On Jun 26, 12:39 pm, vikram patilvik...@gmail.com wrote: Yes I am using v 2.1.0 I provided compiler option -qrtti=all . So compilation went smoothly but last step where protoc is uses proto files seems like running in infinite loop. I am currently looking at it Vikram On Jun 26, 12:32 pm, Kenton Varda ken...@google.com wrote: On Thu, Jun 25, 2009 at 6:59 PM, vikram patilvik...@gmail.com wrote: Hello guys, I am actually attempting to compile protocol buffer source on AIX 5.3 and xlC 3.55 . It seems that I am able to compile it correctly but when it tried to use created protoc compiler it fails. So following it the snapshot of what happens when I do make. basically it can not replace ./$^ with appropriate prerequisites Yeah, you need to use gmake. I created this simplest proto file to check whether it works fine. But when I tried tocompile it with create protoc binary ( .libs/lt- protoc ) it blocked in infinite loop. Can you run in a debugger, ctrl+c during the loop, and give me the stack trace? But with option uncommented it creates coredump ./src/.libs/lt-protoc src/google/protobuf/test.proto -I. --cpp_out=. Assertion failed: f == NULL || dynamic_castTo(f) != NULL, file ./ google/protobuf/stubs/common.h, line 282 IOT/Abort trap (core dumped) You are using v2.1.0, right? Does your compiler support RTTI? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Thread-safe messages
Simply adding a mutex lock in every accessor wouldn't really make them
thread-safe. Consider:
if (my_message.has_foo()) {
DoSomething(my_message.foo());
}
This is not thread-safe if my_message can be modified in a separate thread
*even if* each accessor locked a mutex, because foo could be cleared in
another thread between the call to has_foo() and the call to foo().
Here's a more dangerous example:
for (int i = 0; i my_message.foo_size(); i++) {
DoSomething(my_message.foo(i);
}
Here, if foo is cleared between the call to foo_size() and the call to
foo(i), then the program will crash. Having a mutex protecting each
individual accessor does not help.
Another problem is that strings are returned by reference. So even if the
accessor itself locks a mutex, the string's contents may be accessed after
the accessor returns, and if they are simultaneously modified in another
thread you'll get a crash.
What you really need to do is maintain your own mutex at a higher level.
Any thread which reads the message object needs to take out a reader lock
before it starts reading and hold that lock until it is completely done
accessing the object. Similarly, any thread writing to a message object
needs to take a writer lock on that mutex at a higher level.
The protobuf library can't really do anything to help here, since the
correct mutex usage depends entirely on your application code.
On Mon, Jun 29, 2009 at 10:01 AM, Jes damealpi...@hotmail.com wrote:
Hi everybody,
we are working on a distributed environment that uses PB, where
different threads will access to the contents of messages that can be
updated at any moment through the network.
I wonder if there is an easy way to transform the (derived) Messages
into a thread-safe version. Maybe the rough solution could be to
include a Mutex in the Message class and a MutexLock on each method of
the generated pb.h and pb.cc classes, but perhaps there are issues
that can break the safety of this approach (such as existing friends
or similar).
Could you have any suggestion on this? :-)
Thanks in advance!
Jes
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: ProtocolBuffer crash before program execution
Sorry, I don't see what the problem could be. Make sure that you are compiling libprotobuf and your project with the same C runtime version (e.g. debug vs. non-debug, DLL vs. static, single-threaded vs. multi-threaded). If that doesn't help, can you send me a minimal example program that demonstrates the problem, so I can debug it? On Mon, Jun 29, 2009 at 5:45 AM, rodrigob rodrigo.benen...@gmail.comwrote: I have uploaded the crash screenshot with Protobug 2.1.0 at http://files.getdropbox.com/u/185965/parking_planning/protobuf_crash_210.jpg as you can see the error is very similar. This problem is being a show stopper for us, any help will be greatly appreciated... Regards, rodrigob. On Jun 26, 9:25 pm, Kenton Varda ken...@google.com wrote: The stack trace you provided looks like it is from v2.0.3. The initialization code changed drastically in 2.1.0 -- the method DescriptorPool::InternalBuildGeneratedFile no longer exists, for example. Can you provide a stack trace using 2.1.0? On Fri, Jun 26, 2009 at 8:07 AM, rodrigob rodrigo.benen...@gmail.com wrote: Hello there, I'm usually a linux developer, but I occasionally have to develop some Windows applications. I have a Gtk + Cairo + ProtocolBuffers that was ported to windows some months ago without any problem. Now I have a new one, with similar setup but a few more dependencies (CGAL, RSVG and others). After configuring the visual studio project and getting the code to compile and link correctly the executable crashes miserably on a ProtocolBuffers exception. The crash details can be inspected at http://files.getdropbox.com/u/185965/parking_planning/protobuf_crash.jpg the weird thing is that the application crashes before ever entering in the main, during some static elements initialization defined by Protocol Buffer. I used first protobuf 2.1.0 and then 2.0.3, both crash in the same way. Any idea of what is going on ? I know that Protobuf can work unders VisualStudio, but this time it is not working for me... what could cause this kind of crashes ? The problem seems similar to http://groups.google.com/group/protobuf/browse_thread/thread/f0f8108f. .. but the crash source is no the same. I'm using Windows Xp and Visual Studio 2005. Any help would be appreciated. Regards, rodrigob. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Using CopyingOutputStreamAdaptor and CodedOutputStream
On Sun, Jun 28, 2009 at 7:30 AM, Alex Black a...@alexblack.ca wrote: Hi, I'm trying to use these to serialize messages to my own stream. I have a couple of questions: 1. If I use message.SerializeToCodedStream() do I still need to write the size of the message to the stream on my own first? Yes. 2. My new code looks like this: MyStream myStream; // this implements CopyingOutputStream CopyingOutputStreamAdaptor protoBufStream(myStream); CodedOutputStream codedOutputStream(protoBufStream); Then I write messages like this: message.SerializeToCodedStream(codedOutputStream); When the CodedOutputStream's destructor gets called, my program aborts hard.. Looks like there is still 5192 bytes in its buffer, and I think I need to call Flush() on it, but there is no flush? What is the error message? Stack trace? thx - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Using CopyingOutputStreamAdaptor and CodedOutputStream
No, that doesn't make sense.
I need to see the exact code, from the point where the CodedOutputStream is
constructed until it is destroyed.
On Mon, Jun 29, 2009 at 5:35 PM, Kenton Varda ken...@google.com wrote:
Odd. By any chance, does this only happen when writing a zero-length
message?
On Mon, Jun 29, 2009 at 5:22 PM, Alex Black a...@alexblack.ca wrote:
Error:
libprotobuf FATAL ..\src\google\protobuf\io\zero_copy_stream_impl.cc:377]
CHECK
failed: buffer_used_ == buffer_size_: BackUp() can only be called after
Next().
Stack trace:
msvcr90d.dll!_NMSG_WRITE(int rterrnum=10) Line 198 C
msvcr90d.dll!abort() Line 68 C
GraphPerf.exe!google::protobuf::internal::LogMessage::Finish() Line
173 C++
GraphPerf.exe!google::protobuf::internal::LogFinisher::operator=(google::protobuf::internal::LogMessage
other={...}) Line 177 C++
GraphPerf.exe!google::protobuf::io::CopyingOutputStreamAdaptor::BackUp(int
count=2202) Line 378 + 0xa0 bytes C++
GraphPerf.exe!google::protobuf::io::CodedOutputStream::~CodedOutputStream()
Line 526 C++
GraphPerf.exe!ProtoBufStreamAdaptor::~ProtoBufStreamAdaptor() Line 10 +
0x59 bytes C++
--
*From:* Kenton Varda [mailto:ken...@google.com]
*Sent:* Monday, June 29, 2009 7:08 PM
*To:* Alex Black
*Cc:* Protocol Buffers
*Subject:* Re: Using CopyingOutputStreamAdaptor and CodedOutputStream
On Sun, Jun 28, 2009 at 7:30 AM, Alex Black a...@alexblack.ca wrote:
Hi, I'm trying to use these to serialize messages to my own stream.
I have a couple of questions:
1. If I use message.SerializeToCodedStream() do I still need to write
the size of the message to the stream on my own first?
Yes.
2. My new code looks like this:
MyStream myStream; // this implements CopyingOutputStream
CopyingOutputStreamAdaptor protoBufStream(myStream);
CodedOutputStream codedOutputStream(protoBufStream);
Then I write messages like this:
message.SerializeToCodedStream(codedOutputStream);
When the CodedOutputStream's destructor gets called, my program aborts
hard.. Looks like there is still 5192 bytes in its buffer, and I think
I need to call Flush() on it, but there is no flush?
What is the error message? Stack trace?
thx
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: where are the examples
Protocol Buffers provides a way to convert between flat byte arrays and structured data. This is obviously useful for network communication, but protocol buffers does not provide any explicit networking support. It's up to you to take the byte array generated by protocol buffers and send it over the network. On Tue, Jun 30, 2009 at 12:45 PM, JavaSrvcs jvsr...@gmail.com wrote: I was led to believe that protocol buffers could be used for PC to PC communication, is there a way to do this, and if so, where can I find the examples? The example I am looking at provided writes to a file. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: CodedOuptputStream.PushLimit/PopLimit
Well, it's *supposed* to work as you say. But honestly, this code has not
been tested with blocking streams, since that's not how we use it in Google.
It's possible that there is a bug. But the code appears to be correct
AFAICT.
When CodedInputStream needs more data, it calls Refresh(), which looks like
this:
bool CodedInputStream::Refresh() {
DCHECK_EQ(buffer_size_, 0);
if (buffer_size_after_limit_ 0 || overflow_bytes_ 0 ||
total_bytes_read_ == current_limit_) {
// ...
return false;
}
// ...
}
In your case, total_bytes_read_ should be equal to current_limit_ -- both
should be 17. Therefore, it *should* be returning false, not trying to read
more.
What are you seeing? If you can't figure it out, can you write a small demo
program? Just make the demo program abort if CodedInputStream tries to read
too much. Then I can debug it.
On Tue, Jun 30, 2009 at 10:57 AM, Matthews heav...@gmail.com wrote:
I am writing a Java server and C++ client and trying to use Protocol
Buffers for encoding the socket communication. I have written the Java
server and tested it with a Java based client and all is well.
However, I am having some difficulty with the C++ portion.
I've wrapped the socket code in my own CopyingInputStream and
CopyingOutputStream sub-classes and am using those with the
CopyingInputStreamAdaptor and CopyingOutputStreamAdapdtor classes.
These classes are further wrapped in CodedInputStream and
CodedOutputStream instances for reading/writing the serialized data.
When I send messages I encode the message type and size as varints
then the message itself. When reading the server response in the C++
code I read the message type and size then pass the size to
CodedInputStream.PushLimit(). My server response is very small, 17
bytes so the initial read gets all the data (I verified that by
stepping through the code in the debugger).
In this case the type and size are each encoded as 1 byte so that
leaves 15 bytes for the message. What I am finding is that even after
the call to PushLimit() with the value 15 the CodedInputStream is
trying to read more bytes. Again, stepping through the code in the
debugger I can see that the 15 bytes are already buffered. I've poured
through the C++ docs and the description of limits states that
PushLimit will prevent CodedInputStream from reading any additional
bytes.
This is a problem as the socket is doing a blocking read so it is
waiting for bytes that will not be sent.
Am I mis-understanding the behavior of PushLimit?
Thanks,
Michael
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Service Factory?
Oh, you want something like DynamicMessage except for services? DynamicServiceStub, perhaps? There is no such thing included in the library, but it would be really trivial to write. All you need to do is implement the Service interface as a wrapper around the RpcChannel interface. Have GetRequestPrototype() and GetResponsePrototype() return DynamicMessages, and have CallMethod() just call the RpcChannel's CallMethod(). On Tue, Jun 30, 2009 at 2:03 PM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: Right, but I'm importing a .proto file at runtime using the Importer class. This ultimately leaves me with a ServiceDescriptor. I'd like to instantiate that Service from the ServiceDescriptor. The python code has a ServiceStubBuilder class. Where is there a ServiceStubBuilder in the C++ API? On Jun 30, 1:33 pm, Kenton Varda ken...@google.com wrote: Yes. In C++ the protocol compiler will generate a service interface and a stub implementation when given a .proto file containing a service definition. This is equivalent to what Python's service_reflection module does. But this is not an RPC implementation. You still have to add your own networking layer. On Tue, Jun 30, 2009 at 9:26 AM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: Is there a C++ equivalent to http://code.google.com/apis/protocolbuffers/docs/reference/python/goo. .. ? which is used to create protocol service and service stub classes from ServiceDescriptor objects at runtime. On Jun 29, 3:56 pm, Kenton Varda ken...@google.com wrote: Protocol Buffers does not include an RPC implementation, only abstract interfaces for one. ServiceDescriptor is useful for implementing your own RPC system on top of protocol buffers -- it allows you to define your services directly in the .proto file even though protocol buffers itself does not provide RPC. On Mon, Jun 29, 2009 at 10:15 AM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: What can you do with a ServiceDescriptor? Is there a DynamicServiceFactory similar to the DynamicMessageFactory where you can instantiate a service when all you have is a ServiceDescriptor, or some other equivalent way of doing it? Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: where are the examples
There are tons of resources on the internet and in books explaining how to do network programming in Java. Sorry, but this is really outside the scope of protocol buffers. On Tue, Jun 30, 2009 at 2:46 PM, JavaSrvcs jvsr...@gmail.com wrote: Is there a full compilable example (in Java) on how to do this ? What sort of service would I need to be running on a server to send data (and have the server receive and convert the flat byte array into a Java object). On Jun 30, 2:32 pm, Kenton Varda ken...@google.com wrote: Protocol Buffers provides a way to convert between flat byte arrays and structured data. This is obviously useful for network communication, but protocol buffers does not provide any explicit networking support. It's up to you to take the byte array generated by protocol buffers and send it over the network. On Tue, Jun 30, 2009 at 12:45 PM, JavaSrvcs jvsr...@gmail.com wrote: I was led to believe that protocol buffers could be used for PC to PC communication, is there a way to do this, and if so, where can I find the examples? The example I am looking at provided writes to a file. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Service Factory?
On Tue, Jun 30, 2009 at 4:28 PM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: I'm confused. Shouldn't I just create a class called DynamicServiceStub and have it inherit from class LIBPROTOBUF_EXPORT Service? And then implement a method like Service* DynamicServiceStub::New (ServiceDescriptor*, RpcChannel* ) ? Yes, that is what I was suggesting. If I get this working, I'd like to pass it back to you guys to perhaps use it in the next release. Honestly, I'm having some trouble thinking of use cases for this. How do you intend to use this class? Why not just call RpcChannel directly instead? On Jun 30, 2:08 pm, Kenton Varda ken...@google.com wrote: Oh, you want something like DynamicMessage except for services? DynamicServiceStub, perhaps? There is no such thing included in the library, but it would be really trivial to write. All you need to do is implement the Service interface as a wrapper around the RpcChannel interface. Have GetRequestPrototype() and GetResponsePrototype() return DynamicMessages, and have CallMethod() just call the RpcChannel's CallMethod(). On Tue, Jun 30, 2009 at 2:03 PM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: Right, but I'm importing a .proto file at runtime using the Importer class. This ultimately leaves me with a ServiceDescriptor. I'd like to instantiate that Service from the ServiceDescriptor. The python code has a ServiceStubBuilder class. Where is there a ServiceStubBuilder in the C++ API? On Jun 30, 1:33 pm, Kenton Varda ken...@google.com wrote: Yes. In C++ the protocol compiler will generate a service interface and a stub implementation when given a .proto file containing a service definition. This is equivalent to what Python's service_reflection module does. But this is not an RPC implementation. You still have to add your own networking layer. On Tue, Jun 30, 2009 at 9:26 AM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: Is there a C++ equivalent to http://code.google.com/apis/protocolbuffers/docs/reference/python/goo. .. ? which is used to create protocol service and service stub classes from ServiceDescriptor objects at runtime. On Jun 29, 3:56 pm, Kenton Varda ken...@google.com wrote: Protocol Buffers does not include an RPC implementation, only abstract interfaces for one. ServiceDescriptor is useful for implementing your own RPC system on top of protocol buffers -- it allows you to define your services directly in the .proto file even though protocol buffers itself does not provide RPC. On Mon, Jun 29, 2009 at 10:15 AM, rthompson.dtisoft@gmail.com rthompson.dtisoft@gmail.com wrote: What can you do with a ServiceDescriptor? Is there a DynamicServiceFactory similar to the DynamicMessageFactory where you can instantiate a service when all you have is a ServiceDescriptor, or some other equivalent way of doing it? Thanks! --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: where are the examples
You could look at one of the open source RPC implementations listed here: http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns#RPC_Implementations Network communication is non-trivial. Trying to demonstrate it with an example would make for a very big example of which only a few lines of code would have anything to do with protocol buffers. Worse, the example would probably only be a naive implementation not suitable for real-world use, which could lead people down the wrong path. A non-naive implementation would be big enough to be its own project, which is far too much work to put into an example. We could provide an example that integrates protocol buffers with some existing RPC implementation, but there are so many to choose from, and we would not be demonstrating much that is not already demonstrated by that RPC system's own documentation. We'd just be taking their examples and adding calls to protobuf parsing and serialization at the ends. It doesn't seem worthwhile. And yes, many people use protocol buffers for disk storage rather than networking. On Tue, Jun 30, 2009 at 3:25 PM, JavaSrvcs jvsr...@gmail.com wrote: I can't imagine any other use than using it to send data over the network and use on the other end. In fact is this not what it was invented for (communication). On 30 jun, 16:07, Kenton Varda ken...@google.com wrote: There are tons of resources on the internet and in books explaining how to do network programming in Java. Sorry, but this is really outside the scope of protocol buffers. On Tue, Jun 30, 2009 at 2:46 PM, JavaSrvcs jvsr...@gmail.com wrote: Is there a full compilable example (in Java) on how to do this ? What sort of service would I need to be running on a server to send data (and have the server receive and convert the flat byte array into a Java object). On Jun 30, 2:32 pm, Kenton Varda ken...@google.com wrote: Protocol Buffers provides a way to convert between flat byte arrays and structured data. This is obviously useful for network communication, but protocol buffers does not provide any explicit networking support. It's up to you to take the byte array generated by protocol buffers and send it over the network. On Tue, Jun 30, 2009 at 12:45 PM, JavaSrvcs jvsr...@gmail.com wrote: I was led to believe that protocol buffers could be used for PC to PC communication, is there a way to do this, and if so, where can I find the examples? The example I am looking at provided writes to a file.- Ocultar texto de la cita - - Mostrar texto de la cita - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Compiling on AIX 5.3 using xlC 3.55 compiler
: 1540-0836 (S) The #include file google/protobuf/generated_message_reflection.h is not found. /work/vpatil/protobuf-2.1.0/src/./google/protobuf/descriptor.pb.h, line 22.10: 1540-0836 (S) The #include file google/protobuf/repeated_field.h is not found. /work/vpatil/protobuf-2.1.0/src/./google/protobuf/descriptor.pb.h, line 23.10: 1540-0836 (S) The #include file google/protobuf/extension_set.h is not found. /work/vpatil/protobuf-2.1.0/src/./google/protobuf/extension_set.h, line 47.10: 1540-0836 (S) The #include file google/protobuf/stubs/common.h is not found. /work/vpatil/protobuf-2.1.0/src/./google/protobuf/extension_set.h, line 48.10: 1540-0836 (S) The #include file google/protobuf/message.h is not found. /work/vpatil/protobuf-2.1.0/src/./google/protobuf/message.h, line 139.10: 1540-0836 (S) The #include file google/protobuf/stubs/common.h is not found. /usr/vacpp/bin/xlC_r: 1501-240 Previous errors were found when compiling tempinc/_tree.C Please share some insights regarding this Thanks Regards, Vikram On Jun 26, 6:00 pm, Kenton Varda ken...@google.com wrote: This isn't telling me much. Can you interrupt and continue a few times to get a few different stack traces, in order to find out what the common part of the trace is? This should allow us to determine which loop is failing to terminate. On Fri, Jun 26, 2009 at 2:49 PM, vikram patilvik...@gmail.com wrote: @Kentan I built with debug info and got following stack trace . location of compiler is at /usr/vacpp/bin if its not by default. #0 0xd1afa248 in __lt__3stdHcQ2_3std11char_traitsXTc_Q2_3std9allocatorXTc__RCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__RCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc_ (_...@0xc, _...@0x2ff2221c) at /usr/vacpp/include/string:134 #1 0xd1afa374 in __cl__Q3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__RCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, a...@0xc, b...@0x2ff2221c) at ./google/protobuf/stubs/ hash.h:177 #2 0xd1d1bbc8 in _Lbound__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, _...@0x2ff2221c) at /usr/vacpp/include/xtree.t:376 #3 0xd1d22878 in lower_bound__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, __classretu...@0x2ff21e70, _...@0x2ff2221c) at /usr/ vacpp/include/xtree:377 #4 0xd1d28f34 in find__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, __classretu...@0x2ff21ed0, _...@0x2ff2221c) at /usr/ vacpp/include/xtree:365 #5 0xd1d2fd34 in ParseType__Q4_6google8protobuf8compiler6ParserFPQ3_6google8protobuf25FieldDescriptorProto_TypePQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0x2ff22268, type=0x2ff21f24, type_name=0x2ff21f28) at google/protobuf/compiler/ parser.cc:1000 #6 0xd1d31438 in ParseMessageField__Q4_6google8protobuf8compiler6ParserFPQ3_6google8protobuf20FieldDescriptorProtoPQ3_6google8protobuf16RepeatedPtrFieldXTQ3_6google8protobuf15DescriptorProto_ ( this=0x2ff22268, field=0x2000f538, messages=0x2000f458) at google/ protobuf/compiler/parser.cc:443 #7 0xd1d30f50 in google::protobuf::compiler::Parser::ParseMessageStatement (this=0x2ff22268, message=0x2000f3f8) at google/protobuf/compiler/ parser.cc:428 #8 0xd1d310c4 in google::protobuf::compiler::Parser::ParseMessageBlock
Re: where are the examples
We have our own system. While we would love to open source it in principle, it is currently tightly coupled to our internal datacenter and machine configurations, so we can't really release the code as-is. I personally hope that we manage to get it out eventually but I have no idea if or when it would happen. On Wed, Jul 1, 2009 at 10:18 AM, J.V. jvsr...@gmail.com wrote: thanks, which product(s) does Google use internally or find most beneficial? Kenton Varda wrote: You could look at one of the open source RPC implementations listed here: http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns#RPC_Implementations --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: ProtocolBuffer crash before program execution
Well, to be fair, Microsoft explicitly says that STL is not allowed to be used in library interfaces for this exact reason. However, protocol buffers was designed on a platform that had no such restriction, and we can't very well rewrite it now just because one platform -- that we don't even use internally -- has unusual requirements. So we did the best we could, and documented in the readme that you need to use matching runtimes. But yeah, I think it's a bad requirement and they could definitely do a better job detecting the resulting problems. On Wed, Jul 1, 2009 at 2:07 AM, rodrigob rodrigo.benen...@gmail.com wrote: Ok, using the debug version library with the debug version of the program fixed the crash. It is very disappointing to see that in VS20xx such miserable errors can occur, without any proper error message (incompatible libraries when linking, instead of random crash when executing). Thanks for the quicks answer. Regards, rodrigob. On Jun 30, 1:01 am, Kenton Varda ken...@google.com wrote: Sorry, I don't see what the problem could be. Make sure that you are compiling libprotobuf and your project with the same C runtime version (e.g. debug vs. non-debug, DLL vs. static, single-threaded vs. multi-threaded). If that doesn't help, can you send me a minimal example program that demonstrates the problem, so I can debug it? On Mon, Jun 29, 2009 at 5:45 AM, rodrigob rodrigo.benen...@gmail.com wrote: I have uploaded the crash screenshot with Protobug 2.1.0 at http://files.getdropbox.com/u/185965/parking_planning/protobuf_crash_. .. as you can see the error is very similar. This problem is being a show stopper for us, any help will be greatly appreciated... Regards, rodrigob. On Jun 26, 9:25 pm, Kenton Varda ken...@google.com wrote: The stack trace you provided looks like it is from v2.0.3. The initialization code changed drastically in 2.1.0 -- the method DescriptorPool::InternalBuildGeneratedFile no longer exists, for example. Can you provide a stack trace using 2.1.0? On Fri, Jun 26, 2009 at 8:07 AM, rodrigob rodrigo.benen...@gmail.com wrote: Hello there, I'm usually a linux developer, but I occasionally have to develop some Windows applications. I have a Gtk + Cairo + ProtocolBuffers that was ported to windows some months ago without any problem. Now I have a new one, with similar setup but a few more dependencies (CGAL, RSVG and others). After configuring the visual studio project and getting the code to compile and link correctly the executable crashes miserably on a ProtocolBuffers exception. The crash details can be inspected at http://files.getdropbox.com/u/185965/parking_planning/protobuf_crash.jpg the weird thing is that the application crashes before ever entering in the main, during some static elements initialization defined by Protocol Buffer. I used first protobuf 2.1.0 and then 2.0.3, both crash in the same way. Any idea of what is going on ? I know that Protobuf can work unders VisualStudio, but this time it is not working for me... what could cause this kind of crashes ? The problem seems similar to http://groups.google.com/group/protobuf/browse_thread/thread/f0f8108f. .. but the crash source is no the same. I'm using Windows Xp and Visual Studio 2005. Any help would be appreciated. Regards, rodrigob. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Java jar - downloadable? or only compilable?
In order to *use* the Java implementation, you need to build that same C++ code -- the protocol compiler. The Java protobuf runtime is mostly useless without it. However, you can get a prebuilt copy of protoc.exe for windows here: http://code.google.com/p/protobuf/downloads/detail?name=protoc-2.1.0-win32.zip You can then use it to build the Java code. On Wed, Jul 1, 2009 at 10:33 PM, J.V. jvsr...@gmail.com wrote: In order to get the Java implementation I have to compile some C++ code, but have limited resources on my machine at this time to install a compiler. Where can I get the Java jar to drop into my project for Win32? thanks J.V. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: output streams
You have to flush the CodedOutputStream.
You should just do this instead:
byte[] data = person.build().toByteArray();
On Thu, Jul 2, 2009 at 12:44 AM, J.V. jvsr...@gmail.com wrote:
I'm trying to get the following to work but nothing is being written to
the output stream.
any ideas?
private void testEncode() throws Exception {
OutputStream out = new ByteArrayOutputStream();
CodedOutputStream cos = CodedOutputStream.newInstance(out);
Person.person.Builder person = Person.person.newBuilder();
person.setFirstName(First);
person.setLastName(Last);
person.build().writeTo(cos);
}
When I try to decode cos it throws an exception
Exception in thread main
com.google.protobuf.InvalidProtocolBufferException: Protocol message
contained an invalid tag (zero).
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: encode to binary stream example
Just call writeTo():
void writeToStream(Message message, OutputStream out) {
message.writeTo(out);
}
On Thu, Jul 2, 2009 at 7:47 AM, J.V. jvsr...@gmail.com wrote:
Which class in the ProtoBuf Java api would be used to encode to a binary
stream?
Is there a short example or tutorial on how to encode (using Java) to a
binary stream for RCP?
a short code snippet on how to encode/decode would be super helpful.
thanks
jrv
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protoc and line endings
Protoc treats \r as plain whitespace, so it should have no problem with Windows line endings. I just tested this and sure enough, protoc works fine with .proto files that use Windows-style line endings. Mac pre-OSX line endings (\r with no \n) won't work if the file contains any comments. What kind of errors are you seeing? On Wed, Jul 1, 2009 at 11:48 PM, Marc Gravell marc.grav...@gmail.comwrote: I'm using protoc as the raw .proto parser for protobuf-net (I then process the compiled binary for code-generation); at the moment, it is very sensitive about line endings - if it isn't LF, it won't work. This creates a bit of a nag for Windows users, as you have to go out of your way to get the right line endings. I could patch the input files, but that gets very tricky if the user is importing multiple other files (with the wrong line endings) into their proto - and it means that their CRLF files wouldn't work fully interoperably. I have no idea how complex it would be, but is there any chance that protoc could be less choosy about line endings? For now, I've added some code to protobuf-net's protogen, so that if protoc reports failure it checks the *known* files for CR (and advises the user) - but this is still tricky for the import scenario. Thoughts? Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: output streams
Write to a stream:
message.writeTo(stream);
Parse from a stream:
message = MyType.parseFrom(stream);
If you need to read/write multiple messages on the same stream (or the
stream does not end immediately after the first message), use
writeDelimitedTo() and parseDelimitedFrom().
On Thu, Jul 2, 2009 at 10:56 AM, J.V. jvsr...@gmail.com wrote:
Thanks, for some reason I'm' getting the same error. If you have an
example of encoding to a stream and then decoding, it would be greatly
appreciated.
I will get it fully working and update the tutorial or post it somewhere
if desired.
thanks
jrv
Kenton Varda wrote:
You have to flush the CodedOutputStream.
You should just do this instead:
byte[] data = person.build().toByteArray();
On Thu, Jul 2, 2009 at 12:44 AM, J.V. jvsr...@gmail.com
mailto:jvsr...@gmail.com wrote:
I'm trying to get the following to work but nothing is being
written to
the output stream.
any ideas?
private void testEncode() throws Exception {
OutputStream out = new ByteArrayOutputStream();
CodedOutputStream cos = CodedOutputStream.newInstance(out);
Person.person.Builder person = Person.person.newBuilder();
person.setFirstName(First);
person.setLastName(Last);
person.build().writeTo(cos);
}
When I try to decode cos it throws an exception
Exception in thread main
com.google.protobuf.InvalidProtocolBufferException: Protocol message
contained an invalid tag (zero).
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protoc and line endings
protoc actually expects its input to be UTF-8 (though non-ASCII characters are only allowed in default values for string fields). It just doesn't like the BOM. On Thu, Jul 2, 2009 at 12:44 PM, Marc Gravell marc.grav...@gmail.comwrote: My bad... it isn't the line endings - it is the UTF8 BOM; when I switched one it switched the other. (which is annoying; encoding is much trickier than just cr/lf!) Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Protoc and line endings
If you want to write up a patch to recognize a UTF8 BOM and ignore it, go ahead. You can just modify the Tokenizer class to recognize and discard a BOM appearing at the beginning of the input. On Thu, Jul 2, 2009 at 1:31 PM, Marc Gravell marc.grav...@gmail.com wrote: OK... is there any way it /could/ silently ignore the BOM? ;-p I can try to advise the caller to use files without BOMs, but protoc reads UTF8 anyway it seems reasonable to accept a BOM? Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: ThirdPartyAddOns
Done and done. I didn't know protobuf-net had RPC or I would have listed it on the RPC implementations page some time ago. On Thu, Jul 2, 2009 at 1:47 PM, Marc Gravell marc.grav...@gmail.com wrote: I notice the replacement ThirdPartyAddOns page; can you please also include protobuf-net in the RPC section? It provides 2 separate RPC implementations (WCF-based and bespoke). Also; I wonder if it might help to clarify (at the top) which languages are provided in the official codebase; then the entire list of implementations would be visible on a single page (just a thought). Thanks, Marc Gravell --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: ThirdPartyAddOns
What's the difference between all these links? On Thu, Jul 2, 2009 at 4:08 PM, Chris Kuklewicz turingt...@gmail.comwrote: Kenton, Could you change the link on the http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns page for Haskell from http://darcs.haskell.org/packages/protocol-buffers/ to http://darcs.haskell.org/packages/protocol-buffers2/ And if there is room can you also add http://hackage.haskell.org/package/hprotoc ? -- Chris --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: error when using nested and packed in python
Yeah, I think it's broken.
In reflection.py there is a line:
return decoder.Position() - content_start
I think (untested) that it should actually be:
return decoder.Position() - initial_position
I have no idea how this got past testing and I have asked the people
responsible to fix it. Surprisingly you are the first to report this, so
maybe there's a mitigating factor that makes it unusual for the bug to harm
anything. Sorry for the trouble.
On Thu, Jul 2, 2009 at 5:14 PM, Dan danbr...@gmail.com wrote:
Hello protobuf gurus-
I'm just getting started with the python version of protobuf, using
v2.10. I want to store some lists of floats in a database's blob
field. When using a nested Message structure with the packed=true
option, I'm getting errors that look like this:
Traceback (most recent call last):
File stdin, line 1, in module
File build/bdist.linux-i686/egg/google/protobuf/message.py, line
160, in ParseFromString
File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
line 1215, in MergeFromString
File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
line 1075, in _DeserializeOneEntity
File build/bdist.linux-i686/egg/google/protobuf/reflection.py,
line 899, in _RecursivelyMerge
File build/bdist.linux-i686/egg/google/protobuf/internal/
decoder.py, line 181, in ReadMessageInto
google.protobuf.message.DecodeError: Submessage told to deserialize
from 14-byte encoding, but used only 12 bytes
If I use a flat (not nested) message structure, I do not get these
errors. If I use a nested, packed=false approach, I do not get these
errors. The error happens only when both nested and packed.
Here is some code to reproduce the problem. Note that the
deserialization seems to succeed in putting the correct values into
the list (at least in this test), in spite of the error message.
my.proto file contains:
message TestNest {
optional ListData list_data = 1;
message ListData {
repeated float float_packed = 1 [packed=true];
repeated float float_notpacked = 2 [packed=false];
repeated int32 int_packed = 3 [packed=true];
repeated int32 int_notpacked = 4 [packed=false];
}
}
and then I run this set of commands for each of the fields
[float_packed, float_notpacked, int_notpacked, int_packed] and see
that the error happens only for the two fields that use packed=true.
import my_pb2
a = my_pb2.TestNest()
a.list_data.float_notpacked.extend([1,2,3])
a_string = a.SerializeToString()
a_deserialized = my_pb2.TestNest()
a_deserialized.ParseFromString(a_string)
str(a_deserialized)
I can think of three options to proceed, 2 of which I'm able to do
myself:
1) don't use the packed=true option
2) catch the errors and ignore them
3) if this is actually a bug, ask you gurus to fix the protobuf code
What do you recommend?
Dan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: libprotobuf.lib, 29mb?
On Thu, Jul 2, 2009 at 5:33 PM, Alex Black a...@alexblack.ca wrote: I'm on windows, using Visual Studio 2008 SP1, compiling a Release x64 build of libprotobuf. The lib file is 29,570KB big, is that expected? I was going to say no way, but sure enough, the 32-bit release build is 18,154k, so 64-bit could easily be double that. However, when you actually link a binary with the library, the size is much smaller. protoc.exe, for example, is 1MB, and it includes both libprotobuf.lib and libprotoc.lib, the latter of which is 10MB. I don't know much about MSVC, but perhaps it is including some sort of debug info in this library? Cygwin produces a 14MB libprotobuf.a, but after stripping it is down to 968k. Note that I consider 1MB to be too big for many users, which is why the next version will have a lite mode that sheds all the descriptors and reflection features and should be very small. I'm using the provided visual studio projects, but I added the x64 configuration to them, perhaps I have some options set incorrectly or something. (The debug lib is 33,575KB) thanks! - Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: how to use protobuf with Google App Engine?
You'd have to ask the AppEngine people. Someone from that team submitted the input_stream.py change. The problem is that both protocol buffers and AppEngine place themselves in the google package. Protocol Buffers has, from day one, declared the package as a namespace package using setuptools, which theoretically allows sharing of the package. For things to work, though, AppEngine has to cooperate by doing the same thing in their own code. On Thu, Jul 2, 2009 at 6:46 PM, Dan danbr...@gmail.com wrote: Hello- I'm working with GAE and ran into this error when trying to use protocol buffers: from google.protobuf import descriptor ImportError: No module named protobuf I saw some mention of people using hacks to do this in the past, but also saw that the release notes for protocol buffers v 2.10 said this: * Changes to input_stream.py should make protobuf compatible with appengine. I'm not sure what is meant by compatible in that statement, but I'd like to know if GAE still requires the hackery described in these threads referenced below. Is there an up-to-date howto or something like that somewhere that I missed? http://groups.google.com/group/protobuf/browse_thread/thread/6430dc5d7ceb145a/b8634dddbe4425f3?lnk=gstq=gae#b8634dddbe4425f3 http://groups.google.com/group/google-appengine/browse_thread/thread/db910ed40ae495fc/f41873a476db744e Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Don't understand the purport of default values
The purpose of default values is to define what the getter methods for those fields return when the field is not set. This is really the only purpose. We do not serialize default values because it would be a waste of space. On Sat, Jul 4, 2009 at 1:47 PM, Zl D zlatko.zla...@gmail.com wrote: Hi, I define a message in .proto file with default values for all elements. After instantiation of generated class (C++), calling 'PrintDebugString()' or 'SerializeToOstream()' produces no output. If I set the any element manually via 'set_' method, then on output I get only that value. Calling getter for particular element produces output. Why cannot I get default values printed out using PrintDebugString() method (or serialized via 'SerializeToOstream()')? Regards, Zlatko --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Parser fails silently if first numeric field has value zero.
This is most likely a bug in your code -- you probably aren't transmitting
the message correctly. Please make sure that the bytes you give to the
protobuf parser on the receiving end are exactly the same bytes that were
produced by the serializer at the sending end -- both the size and contents
must be identical. If you are still having problems, please write a small
example program demonstrating the issue and send it to me to debug.
On Sun, Jul 5, 2009 at 11:30 PM, Vijay Mathew
vijay.the.sche...@gmail.comwrote:
Hello all,
I have a Message type EventStatus with the following definition:
message EventStatus {
required int32 reason = 1;
required string msg = 2;
}
This is part of a larger protocol where a C++ client sends an event to
a Java server where it is handled. The Java server returns an
EventStatus. If the reason field is 0 (success), protobuf parser on C+
+ side does not read the value of the msg field. If the reason field
is 0, then it works fine. As a workaround I reversed the order of
the two fields:
message EventStatus {
required string msg = 1;
required int32 reason = 2;
}
Now the msg field is read even when reason is zero.
Why this strange behavior? Is this a known bug is protobuf?
Thanks,
-- Vijay
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Protocol buffers extensions getting lost on the way?
The javadoc for ExtensionRegistry discusses the reasons why we didn't take
the auto-populated approach as was done in C++. In fact, I sort of wish we
had done something like ExtensionRegistry in C++ as well, as I get a lot of
messages from people who are confused why their program behaves differently
when they simply link in an extra library without actually using it. Also,
technically the C++ standard does not require a translation unit to be
initialized until it is first accessed (similar to Java), but it just
happens that most compilers do all initialization at startup time.
On Wed, Jul 8, 2009 at 9:38 AM, ProzakLord jamig...@gmail.com wrote:
Since I figured it out I might as well put the solution here.
The problem was on the java side.
The call to parse has to be modified so that it looks in the following
way(there may be a better option out there but this does it for me).
ExtensionsRegistry registry = ExtensionsRegistry.newInstance();
registry.add(Application.operation1);
registry.add(Application.operation2);
PbFrameworkMessage request = PbFrameworkMessage.parseFrom
(networkBuffer,registry);
Thats it.
Now the bonus points are is there a way in java for the
ExtensionsRegistry to autopopulate like in C++? is it already so?
Kind regards.
On 8 jul, 16:14, ProzakLord jamig...@gmail.com wrote:
Hi all,
I was wondering if anyone had run across the same issue.
The issue is the following. I have a bunch of protocl buffers that I
use for the communnication between client and server, the client is in
C++ the server in java. It was all running fine until I made the
following refactoring in the code:
I went from having something along the lines of
message PbApplicationMessage {
optional PbApplicationOP1Request operation1 = 1;
optional PbApplicationOP2Request operation2 = 2;
}
to more generic solution using extensions
Framework proto file
message PbFrameworkMessage {
required PbApplication Message appMessage = 1;
}
message PbApplicationMesasge{
extensions 1 to 18999;
}
and in my application proto file
#import framework.proto
extend PbApplicationMessage {
optional PbApplicationOP1Request operation1 = 1;
oprional PbApplicationOP2Request operation2 = 2;
}
I refactored all the code for what could be required. I compile the
protocol buffers protofile
so that both generated files are available on the same package for
both client and server.
The framework.proto is compile only once to be part of a library that
is statically linked to the client version.
Now the problem that I have is teh following.
I serialize in C++ the Framework Message after I have put the
extension in it by using the
request-mutable_appmessge().MutableExtenstion(operation1)
Just before calling the request-SerializeToString(string) I check
with
request-appmessage().HasExtension(operation1) and that returns true.
Until here all is fine. I then dump that to the wire
and when on the java side I pick up the protocolbuffer I do the
following
PbFrameworkMessage request = PbFrameworkMessage.parseFrom
(networkBuffer);
I then do a request.getAppMessage().hasExtension
(Application.operation1) which returns false
The network code that puts things to and from the wire has been
running without the extensions sending portocol buffers around with
absolutely no problems.
Am I missing something obvious that I ought to do before the serialize
to String on the client, whic is required by the enxtensions and is
not in the nromal version?
I am running protobuf 2.0.0beta on both sides.
Any help or pointer will be much appreciated.
Kind regards.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: zero_copy_stream_unittest.cc doesn't compile with msvc 7.1/2003
OK, I'll make the #if check the version number in the next release. On Tue, Jul 7, 2009 at 12:15 AM, Cosmin Cremarenco cosmin.cremare...@gmail.com wrote: It is 1310 On Jul 6, 7:49 pm, Kenton Varda ken...@google.com wrote: What's the _MSC_VER for that version? On Mon, Jul 6, 2009 at 10:37 AM, Cosmin Cremarenco cosmin.cremare...@gmail.com wrote: It is Visual Studio 2003 - Version 7.1.6030. I think the .1 minor comes from the installation of sp1 over 7.0 Cosmin On Jul 6, 7:33 pm, Kenton Varda ken...@google.com wrote: This is with MSVC 7.1? Which version of Visual Studio does that correspond to? I can never keep these straight. On Mon, Jul 6, 2009 at 7:27 AM, Cosmin Cremarenco cosmin.cremare...@gmail.com wrote: Hi When compiling zero_copy_stream_unittest.cc from the last version 2.1.0 msvc complains about the type _invalid_parameter_handler (I don't think it is supported in 7.1): Compiling... zero_copy_stream_unittest.cc \tmp\protobuf-2.1.0-vc71\src\google\protobuf\io \zero_copy_stream_unittest.cc(475) : error C2146: syntax error : missing ';' before identifier 'old_handler_' \tmp\protobuf-2.1.0-vc71\src\google\protobuf\io \zero_copy_stream_unittest.cc(475) : error C2501: 'google::protobuf::io::`anonymous- namespace'::MsvcDebugDisabler::_invalid_parameter_handler' : missing storage-class or type specifiers \tmp\protobuf-2.1.0-vc71\src\google\protobuf\io \zero_copy_stream_unittest.cc(475) : error C2501: 'google::protobuf::io::`anonymous- namespace'::MsvcDebugDisabler::old_handler_' : missing storage-class or type specifiers Changing the wrapping conditional compilation macro from #if _MSC_VER to #if 0 fixes it. Best regards, Cosmin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: intermittent issue with encode (version 2.0.3)
Run in a debugger and set a breakpoint at wire_format_inl.h:289. On Thu, Jul 9, 2009 at 1:56 PM, Rizzuto, Raymond raymond.rizz...@sig.comwrote: I think I have an error in my code (C++) that only occurs when I have multiple threads, and a lot of message volume. Even then, I can run the same test many times, but only get a failure on some runs. With 7 threads running on a 4 core machine, and generating 480384 google protocol buffer messages, I get 33 errors like this to stdout: libprotobuf ERROR /siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:289] Encountered string containing invalid UTF-8 data while serializing protocol buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes. I believe that the data is in error since I get similar errors decoding the messages: libprotobuf ERROR /siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:138] Encountered string containing invalid UTF-8 data while parsing protocol buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes. Is there any way that I can check for this at run time so that I can print out more context? I do call IsInitialized before serializing, but that doesn’t check for this case. I am running on SLES9SP4, using gcc 3.3.3 as the compiler. Ray -- Ray Rizzuto raymond.rizz...@sig.com Susquehanna International Group (610)747-2336 (W) (215)776-3780 (C) -- IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: intermittent issue with encode (version 2.0.3)
I suppose you could also temporarily edit the header file. On Thu, Jul 9, 2009 at 2:05 PM, Rizzuto, Raymond raymond.rizz...@sig.comwrote: I’m trying to, without success. Breakpoints in header files, at least with the version of tools I have, don’t work very well. -- *From:* Kenton Varda [mailto:ken...@google.com] *Sent:* Thursday, July 09, 2009 5:02 PM *To:* Rizzuto, Raymond *Cc:* protobuf@googlegroups.com *Subject:* Re: intermittent issue with encode (version 2.0.3) Run in a debugger and set a breakpoint at wire_format_inl.h:289. On Thu, Jul 9, 2009 at 1:56 PM, Rizzuto, Raymond raymond.rizz...@sig.com wrote: I think I have an error in my code (C++) that only occurs when I have multiple threads, and a lot of message volume. Even then, I can run the same test many times, but only get a failure on some runs. With 7 threads running on a 4 core machine, and generating 480384 google protocol buffer messages, I get 33 errors like this to stdout: libprotobuf ERROR /siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:289] Encountered string containing invalid UTF-8 data while serializing protocol buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes. I believe that the data is in error since I get similar errors decoding the messages: libprotobuf ERROR /siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:138] Encountered string containing invalid UTF-8 data while parsing protocol buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes. Is there any way that I can check for this at run time so that I can print out more context? I do call IsInitialized before serializing, but that doesn’t check for this case. I am running on SLES9SP4, using gcc 3.3.3 as the compiler. Ray -- Ray Rizzuto raymond.rizz...@sig.com Susquehanna International Group (610)747-2336 (W) (215)776-3780 (C) -- IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. -- IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: intermittent issue with encode (version 2.0.3)
This is something you can do in your own code -- just call your validation
function before serializing. If this were to be a feature of protocol
buffers, then we'd have to store a pointer to your validator function
somewhere. Storing it in the message object itself would harm performance
and memory usage, but storing it in a static location (such that it applies
to all instances of the type) would bring all the myriad problems commonly
associated with singletons. So I don't think there's any reasonable way for
the protobuf system to provide this.
On Thu, Jul 9, 2009 at 2:14 PM, Rizzuto, Raymond raymond.rizz...@sig.comwrote:
I’m going to try that. Since another group builds and packages the
libraries I use, it’ll take a bit to make a private copy with that change.
As an enhancement request, I wish there was a function I could call to
validate the message content before serialize, that would tell me about any
fields of the message that are in error. I.e. so I could catch that issue
similarly to catching uninitialized fields:
if (!m.IsInitialized())
{
std::string error = name + is missing fields: ;
std::vectorstd::string errors;
m.FindInitializationErrors(errors);
std::vectorstd::string::const_iterator it;
for(it = errors.begin(); it!= errors.end(); ++it)
{
if (it != errors.begin())
error += , ;
error += *it;
}
throw SPException(error.c_str());
}
It might not be something I’d do in production, but it sure would help
during development.
--
*From:* Kenton Varda [mailto:ken...@google.com]
*Sent:* Thursday, July 09, 2009 5:08 PM
*To:* Rizzuto, Raymond
*Cc:* protobuf@googlegroups.com
*Subject:* Re: intermittent issue with encode (version 2.0.3)
I suppose you could also temporarily edit the header file.
On Thu, Jul 9, 2009 at 2:05 PM, Rizzuto, Raymond raymond.rizz...@sig.com
wrote:
I’m trying to, without success. Breakpoints in header files, at least
with the version of tools I have, don’t work very well.
--
*From:* Kenton Varda [mailto:ken...@google.com]
*Sent:* Thursday, July 09, 2009 5:02 PM
*To:* Rizzuto, Raymond
*Cc:* protobuf@googlegroups.com
*Subject:* Re: intermittent issue with encode (version 2.0.3)
Run in a debugger and set a breakpoint at wire_format_inl.h:289.
On Thu, Jul 9, 2009 at 1:56 PM, Rizzuto, Raymond raymond.rizz...@sig.com
wrote:
I think I have an error in my code (C++) that only occurs when I have
multiple threads, and a lot of message volume. Even then, I can run the
same test many times, but only get a failure on some runs. With 7 threads
running on a 4 core machine, and generating 480384 google protocol buffer
messages, I get 33 errors like this to stdout:
libprotobuf ERROR
/siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:289]
Encountered string containing invalid UTF-8 data while serializing protocol
buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes.
I believe that the data is in error since I get similar errors decoding the
messages:
libprotobuf ERROR
/siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:138]
Encountered string containing invalid UTF-8 data while parsing protocol
buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes.
Is there any way that I can check for this at run time so that I can print
out more context? I do call IsInitialized before serializing, but that
doesn’t check for this case.
I am running on SLES9SP4, using gcc 3.3.3 as the compiler.
Ray
--
Ray Rizzuto
raymond.rizz...@sig.com
Susquehanna International Group
(610)747-2336 (W)
(215)776-3780 (C)
--
IMPORTANT: The information contained in this email and/or its attachments
is confidential. If you are not the intended recipient, please notify the
sender immediately by reply and immediately delete this message and all its
attachments. Any review, use, reproduction, disclosure or dissemination of
this message or any attachment by an unintended recipient is strictly
prohibited. Neither this message nor any attachment is intended as or should
be construed as an offer, solicitation or recommendation to buy or sell any
security or other financial instrument. Neither the sender, his or her
employer nor any of their respective affiliates makes any warranties as to
the completeness or accuracy of any of the information contained herein or
that this message or any of its attachments is free of viruses.
--
IMPORTANT: The information contained in this email and/or its attachments
is confidential
Re: intermittent issue with encode (version 2.0.3)
Sorry, I think I misread your message.
You just want there to me a method like IsInitialized() that you can call to
validate UTF-8 stuff. I'll think about that.
On Thu, Jul 9, 2009 at 2:32 PM, Kenton Varda ken...@google.com wrote:
This is something you can do in your own code -- just call your validation
function before serializing. If this were to be a feature of protocol
buffers, then we'd have to store a pointer to your validator function
somewhere. Storing it in the message object itself would harm performance
and memory usage, but storing it in a static location (such that it applies
to all instances of the type) would bring all the myriad problems commonly
associated with singletons. So I don't think there's any reasonable way for
the protobuf system to provide this.
On Thu, Jul 9, 2009 at 2:14 PM, Rizzuto, Raymond
raymond.rizz...@sig.comwrote:
I’m going to try that. Since another group builds and packages the
libraries I use, it’ll take a bit to make a private copy with that change.
As an enhancement request, I wish there was a function I could call to
validate the message content before serialize, that would tell me about any
fields of the message that are in error. I.e. so I could catch that issue
similarly to catching uninitialized fields:
if (!m.IsInitialized())
{
std::string error = name + is missing fields: ;
std::vectorstd::string errors;
m.FindInitializationErrors(errors);
std::vectorstd::string::const_iterator it;
for(it = errors.begin(); it!= errors.end(); ++it)
{
if (it != errors.begin())
error += , ;
error += *it;
}
throw SPException(error.c_str());
}
It might not be something I’d do in production, but it sure would help
during development.
--
*From:* Kenton Varda [mailto:ken...@google.com]
*Sent:* Thursday, July 09, 2009 5:08 PM
*To:* Rizzuto, Raymond
*Cc:* protobuf@googlegroups.com
*Subject:* Re: intermittent issue with encode (version 2.0.3)
I suppose you could also temporarily edit the header file.
On Thu, Jul 9, 2009 at 2:05 PM, Rizzuto, Raymond raymond.rizz...@sig.com
wrote:
I’m trying to, without success. Breakpoints in header files, at least
with the version of tools I have, don’t work very well.
--
*From:* Kenton Varda [mailto:ken...@google.com]
*Sent:* Thursday, July 09, 2009 5:02 PM
*To:* Rizzuto, Raymond
*Cc:* protobuf@googlegroups.com
*Subject:* Re: intermittent issue with encode (version 2.0.3)
Run in a debugger and set a breakpoint at wire_format_inl.h:289.
On Thu, Jul 9, 2009 at 1:56 PM, Rizzuto, Raymond raymond.rizz...@sig.com
wrote:
I think I have an error in my code (C++) that only occurs when I have
multiple threads, and a lot of message volume. Even then, I can run the
same test many times, but only get a failure on some runs. With 7 threads
running on a 4 core machine, and generating 480384 google protocol buffer
messages, I get 33 errors like this to stdout:
libprotobuf ERROR
/siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:289]
Encountered string containing invalid UTF-8 data while serializing protocol
buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes.
I believe that the data is in error since I get similar errors decoding
the messages:
libprotobuf ERROR
/siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:138]
Encountered string containing invalid UTF-8 data while parsing protocol
buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes.
Is there any way that I can check for this at run time so that I can print
out more context? I do call IsInitialized before serializing, but that
doesn’t check for this case.
I am running on SLES9SP4, using gcc 3.3.3 as the compiler.
Ray
--
Ray Rizzuto
raymond.rizz...@sig.com
Susquehanna International Group
(610)747-2336 (W)
(215)776-3780 (C)
--
IMPORTANT: The information contained in this email and/or its attachments
is confidential. If you are not the intended recipient, please notify the
sender immediately by reply and immediately delete this message and all its
attachments. Any review, use, reproduction, disclosure or dissemination of
this message or any attachment by an unintended recipient is strictly
prohibited. Neither this message nor any attachment is intended as or should
be construed as an offer, solicitation or recommendation to buy or sell any
security or other financial instrument. Neither the sender, his or her
employer nor any of their respective affiliates makes any warranties as to
the completeness or accuracy
Re: intermittent issue with encode (version 2.0.3)
As the comment says, the first call will always occur at startup time when there is only one thread anyway, so it's perfectly safe. The parenthetical about GCC4 is just an aside. On Thu, Jul 9, 2009 at 2:47 PM, Rizzuto, Raymond raymond.rizz...@sig.comwrote: I am a bit nervous about the GCC4 comment in GeneratedMessageFactory::singleton (message.cc): // No need for thread-safety here because this will be called at static // initialization time. (And GCC4 makes this thread-safe anyway.) I’m using gcc 3.3.3. The singleton object in GeneratedMessageFactory::singleton, is a local static of non-POD type. The C++ standard says: An implementation is permitted to perform early initialization of other local objects with static storage duration under the same conditions that an implementation is permitted to statically initialize an object with static storage duration in namespace scope (3.6.2). Otherwise such an object is initialized the first time control passes through its declaration; such an object is considered initialized upon the completion of its initialization. I don’t think the language standard addresses what “first time control passes through its declaration” means when two threads call the function simultaneously. Perhaps gcc4 provides features that make that safe. I don’t know if that is something that can be relied on in all compilers, however. Ray -- *From:* Kenton Varda [mailto:ken...@google.com] *Sent:* Thursday, July 09, 2009 5:08 PM *To:* Rizzuto, Raymond *Cc:* protobuf@googlegroups.com *Subject:* Re: intermittent issue with encode (version 2.0.3) I suppose you could also temporarily edit the header file. On Thu, Jul 9, 2009 at 2:05 PM, Rizzuto, Raymond raymond.rizz...@sig.com wrote: I’m trying to, without success. Breakpoints in header files, at least with the version of tools I have, don’t work very well. -- *From:* Kenton Varda [mailto:ken...@google.com] *Sent:* Thursday, July 09, 2009 5:02 PM *To:* Rizzuto, Raymond *Cc:* protobuf@googlegroups.com *Subject:* Re: intermittent issue with encode (version 2.0.3) Run in a debugger and set a breakpoint at wire_format_inl.h:289. On Thu, Jul 9, 2009 at 1:56 PM, Rizzuto, Raymond raymond.rizz...@sig.com wrote: I think I have an error in my code (C++) that only occurs when I have multiple threads, and a lot of message volume. Even then, I can run the same test many times, but only get a failure on some runs. With 7 threads running on a 4 core machine, and generating 480384 google protocol buffer messages, I get 33 errors like this to stdout: libprotobuf ERROR /siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:289] Encountered string containing invalid UTF-8 data while serializing protocol buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes. I believe that the data is in error since I get similar errors decoding the messages: libprotobuf ERROR /siglinux/tc/sles9sp4_gcc-3.3.3_i686/sig1/protobuf-2.0.3/include/google/protobuf/wire_format_inl.h:138] Encountered string containing invalid UTF-8 data while parsing protocol buffer. Strings must contain only UTF-8; use the 'bytes' type for raw bytes. Is there any way that I can check for this at run time so that I can print out more context? I do call IsInitialized before serializing, but that doesn’t check for this case. I am running on SLES9SP4, using gcc 3.3.3 as the compiler. Ray -- Ray Rizzuto raymond.rizz...@sig.com Susquehanna International Group (610)747-2336 (W) (215)776-3780 (C) -- IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses. -- IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment
Re: Howto build protobuf not as library, but directly linking object files?
On Fri, Jul 10, 2009 at 3:12 AM, bart van deenen bart.vandee...@gmail.comwrote: However, when I try to build my application with -Lpath/to/ libprotobuf.a I get hundreds of linker errors, complaining about undefined references to any of the google protocol functions. You need to use: -Lpath/to/lib/dir -lprotobuf The -L flag adds a directory which will be searched for libraries, whereas the -l flag says to link against a particular library. The compiler automatically prefixes the library name with lib and suffixes it with .a. Note that I test every release on MinGW. You can use MSYS to invoke the included configure script and makefiles, to build it just like you would on any Unix system. I'm using the same mingw compiler to build my Qt application, so that is not the problem. Anyway, I'm still struggling with this stuff, and since no one at my company knows anything about C++ on Windows, it's a hassle. So that's why I just try to circumvent the problem by just linking the object files directly. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: DescriptorPool in python
I don't think that's currently implemented in Python, unfortunately. On Mon, Jul 13, 2009 at 8:28 AM, Ferenc Szalai szf...@wsbricks.com wrote: Hi There is any plan to implement DescriptorPool class in python? Especially, I missing easy way to transform FileDescriptorProto to Descriptor in python. -- Regards, Ferenc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Compiling on AIX 5.3 using xlC 3.55 compiler
google/protobuf/stubs/hash.h already contains some hacks for hash_map. To support unordered_map, all we'd have to do is add another hack there which defines hash_map to be a subclass of unordered_map. Subclassing effectively functions as a template typedef here. I would rather not replace the identifier hash_map with unordered_map in the actual code until the Google style guide rules on the issue. I suspect that Google code will go on using hash_map with a similar hack because updating our entire code base is just not worth the effort. On Mon, Jul 13, 2009 at 5:13 PM, Monty Taylor mord...@inaugust.com wrote: vikram wrote: I have found out that with new xlC versions like 8.X onwards hash_map like functionality is supported but different name as unordered_map. So it there any way you can to use this container without modifying much of the code. In the code hash_map is used in many places. So it needs to be replaced with unordered_map with xlC compiler on AIX. Please provide some idea. I was trying to do template typedef but seems like I can not have all typenames while doing that. I was actually just working on making an update to the m4 to detect unordered_map in Drizzle. (We swiped the hash_map detection macro) unordered_map is the name it's apparently going to land in C++0x as, and is the name that it exists as in gcc 4.3 and 4.4. gcc still has hash_map as well, but it throws a deprecated warning. Might not be a terrible idea to go ahead and shift to unordered_map and then put in a mapping/typedef for hash_map if something doesn't have u_m? (Kenton - would you be interested in a patch doing that?) On Jul 1, 12:00 pm, Kenton Varda ken...@google.com wrote: Well, it looks like all of these are stuck in the same place -- in the same call to hash_map::find(). This would seem to indicate that your STL implementation is broken. It's also possible that the infinite loop is actually in protobuf code, and the only reason we see it always breaking in the same find() call is because that's the most expensive part of the loop. You could test this by breaking under gdb again, and then repeatedly typing finish to make it run to completion of the current function call. If it eventually gets back to protobuf code, then the problem is there, otherwise it's in the STL code. (Actually, I should have told you to do this originally, rather than the collect multiple stack traces idea...) On Tue, Jun 30, 2009 at 7:36 PM, vikram patilvik...@gmail.com wrote: Hey Kenton, This is compilation without STL implementation . I am assuming that if hash_map does not exist , google protocol buffer emulates hash_map. I am pasting 3-4 instances of stack where protoc is in infinite loop #0 0xd1cfdc60 in _Node::_Right__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__FPQ3_3std9_Tree_nodXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0 (_P=0xf04ca4e0) at /usr/vacpp/include/xtree:154 #1 0xd1d1bbdc in _Lbound__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, _...@0x2ffc) at /usr/vacpp/include/xtree.t:377 #2 0xd1d22878 in lower_bound__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, __classretu...@0x2ff21d70, _...@0x2ffc) at /usr/ vacpp/include/xtree:377 #3 0xd1d28f34
Re: Compiling on AIX 5.3 using xlC 3.55 compiler
And yes, I'd love a patch. :) On Mon, Jul 13, 2009 at 5:22 PM, Kenton Varda ken...@google.com wrote: google/protobuf/stubs/hash.h already contains some hacks for hash_map. To support unordered_map, all we'd have to do is add another hack there which defines hash_map to be a subclass of unordered_map. Subclassing effectively functions as a template typedef here. I would rather not replace the identifier hash_map with unordered_map in the actual code until the Google style guide rules on the issue. I suspect that Google code will go on using hash_map with a similar hack because updating our entire code base is just not worth the effort. On Mon, Jul 13, 2009 at 5:13 PM, Monty Taylor mord...@inaugust.comwrote: vikram wrote: I have found out that with new xlC versions like 8.X onwards hash_map like functionality is supported but different name as unordered_map. So it there any way you can to use this container without modifying much of the code. In the code hash_map is used in many places. So it needs to be replaced with unordered_map with xlC compiler on AIX. Please provide some idea. I was trying to do template typedef but seems like I can not have all typenames while doing that. I was actually just working on making an update to the m4 to detect unordered_map in Drizzle. (We swiped the hash_map detection macro) unordered_map is the name it's apparently going to land in C++0x as, and is the name that it exists as in gcc 4.3 and 4.4. gcc still has hash_map as well, but it throws a deprecated warning. Might not be a terrible idea to go ahead and shift to unordered_map and then put in a mapping/typedef for hash_map if something doesn't have u_m? (Kenton - would you be interested in a patch doing that?) On Jul 1, 12:00 pm, Kenton Varda ken...@google.com wrote: Well, it looks like all of these are stuck in the same place -- in the same call to hash_map::find(). This would seem to indicate that your STL implementation is broken. It's also possible that the infinite loop is actually in protobuf code, and the only reason we see it always breaking in the same find() call is because that's the most expensive part of the loop. You could test this by breaking under gdb again, and then repeatedly typing finish to make it run to completion of the current function call. If it eventually gets back to protobuf code, then the problem is there, otherwise it's in the STL code. (Actually, I should have told you to do this originally, rather than the collect multiple stack traces idea...) On Tue, Jun 30, 2009 at 7:36 PM, vikram patilvik...@gmail.com wrote: Hey Kenton, This is compilation without STL implementation . I am assuming that if hash_map does not exist , google protocol buffer emulates hash_map. I am pasting 3-4 instances of stack where protoc is in infinite loop #0 0xd1cfdc60 in _Node::_Right__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__FPQ3_3std9_Tree_nodXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0 (_P=0xf04ca4e0) at /usr/vacpp/include/xtree:154 #1 0xd1d1bbdc in _Lbound__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, _...@0x2ffc) at /usr/vacpp/include/xtree.t:377 #2 0xd1d22878 in lower_bound__Q2_3std5_TreeXTQ2_3std12_Tmap_traitsXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_TypeTQ3_6google8protobuf4hashXTQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc___TQ2_3std9allocatorXTQ2_3std4pairXTCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__TQ3_6google8protobuf25FieldDescriptorProto_Type__SP0__CFRCQ2_3std12basic_stringXTcTQ2_3std11char_traitsXTc_TQ2_3std9allocatorXTc__ (this=0xf04ca4e0, __classretu
Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
Speed varies a lot depending on the precise content. My benchmarks
generally show serialization performance somewhere between 100 MB/s and 1
GB/s, whereas you're seeing 33MB/s, but my benchmarks do not include any
kind of I/O. Maybe you could separate the serialization step from the I/O
(by serializing to one huge in-memory buffer) so that you can measure the
times separately?
On Mon, Jul 13, 2009 at 9:36 PM, Alex Black a...@alexblack.ca wrote:
I'm sending a message with about ~150k repeated items in it, total
size is about 3.3mb, and its taking me about 100ms to serialize it and
send it out.
Can I expect to do any better than this? What could I look into to
improve this?
- I have option optimize_for = SPEED; set in my proto file
- I'm compiling with -O3
- I'm sending my message in batches of 1000
- I'm using C++, on ubuntu, x64
- I'm testing all on one machine (e.g. client and server are on one
machine)
My message looks like:
message NodeWithNeighbors
{
required Id nodeId = 1;
repeated IdConfidence neighbors = 2;
}
message GetNeighborsResponse
{
repeated NodeWithNeighbors nodesWithNeighbors = 1;
}
message IdConfidence
{
required bytes id = 1;
required float confidence = 2;
}
Where bytes id is used to send 16byte IDs (uuids).
I'm writing each message (batch) out like this:
CodedOutputStream codedOutputStream(m_ProtoBufStream);
// Write out the size of the message
codedOutputStream.WriteVarint32(message.ByteSize());
// Ask the message to serialize itself to our stream adapter, which
ultimately calls Write on us
// which we then call Write on our composed stream
message.SerializeWithCachedSizes(codedOutputStream);
In my stream implementation I'm buffering every 16kb, and calling send
on the socket once i have 16kb.
Thanks!
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
Oh, I didn't even know you were including composition in there. My
benchmarks are only for serialization of already-composed messages.
But this still doesn't tell us how much time is spent on network I/O vs.
protobuf serialization. My guess is that once you factor that out, your
performance is pretty close to the benchmarks.
On Mon, Jul 13, 2009 at 10:11 PM, Alex Black a...@alexblack.ca wrote:
If I comment out the actual serialization and sending of the message
(so I am just composing messages, and clearing them each batch) then
the 100ms drops to about 50ms.
On Jul 14, 12:36 am, Alex Black a...@alexblack.ca wrote:
I'm sending a message with about ~150k repeated items in it, total
size is about 3.3mb, and its taking me about 100ms to serialize it and
send it out.
Can I expect to do any better than this? What could I look into to
improve this?
- I have option optimize_for = SPEED; set in my proto file
- I'm compiling with -O3
- I'm sending my message in batches of 1000
- I'm using C++, on ubuntu, x64
- I'm testing all on one machine (e.g. client and server are on one
machine)
My message looks like:
message NodeWithNeighbors
{
required Id nodeId = 1;
repeated IdConfidence neighbors = 2;
}
message GetNeighborsResponse
{
repeated NodeWithNeighbors nodesWithNeighbors = 1;
}
message IdConfidence
{
required bytes id = 1;
required float confidence = 2;
}
Where bytes id is used to send 16byte IDs (uuids).
I'm writing each message (batch) out like this:
CodedOutputStream codedOutputStream(m_ProtoBufStream);
// Write out the size of the message
codedOutputStream.WriteVarint32(message.ByteSize());
// Ask the message to serialize itself to our stream adapter,
which
ultimately calls Write on us
// which we then call Write on our composed stream
message.SerializeWithCachedSizes(codedOutputStream);
In my stream implementation I'm buffering every 16kb, and calling send
on the socket once i have 16kb.
Thanks!
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
OK. If your message composition (or parsing, on the receiving end) takes a
lot of time, you might look into how much of that is due to memory
allocation. Usually this is a pretty significant fraction. Two good ways
to improve that:
1) If your app builds many messages over time and most of them have roughly
the same shape (i.e. which fields are set, the size of repeated fields,
etc. are usually similar), then you should clear and reuse the same message
object rather than allocate a new one each time. This way it will reuse the
same memory, avoiding allocation.
2) Use tcmalloc:
http://google-perftools.googlecode.com
It is often faster than your system's malloc, particularly for
multi-threaded C++ apps. All C++ servers at Google use this.
On Mon, Jul 13, 2009 at 11:50 PM, Alex Black a...@alexblack.ca wrote:
Kenton: I made a mistake with these numbers - pls ignore them - I'll
revisit tomorrow.
Thx.
-Original Message-
From: protobuf@googlegroups.com [mailto:proto...@googlegroups.com] On
Behalf Of Alex Black
Sent: Tuesday, July 14, 2009 2:05 AM
To: Protocol Buffers
Subject: Re: Performance: Sending a message with ~150k items, approx 3.3mb,
can I do better than 100ms?
ok, I took I/O out of the picture by serializing each message into a
pre-allocated buffer, and this time I did a more through measurement.
Benchmark 1: Complete scenario
- average time 262ms (100 runs)
Benchmark 2: Same as # 1 but no IO
- average time 250ms (100 runs)
Benchmark 3: Same as 2 but with serialization commented out
- average time 251ms (100 runs)
Benchmark 4: Same as 3 but with message composition commented out too (no
protobuf calls)
- average time 185 ms (100 runs)
So from this I conclude:
- My initial #s were wrong
- My timings vary too much for each run to really get accurate averages
- IO takes about 10ms
- Serialization takes ~0ms
- Message composition and setting of fields takes ~66ms
My message composition is in a loop, the part in the loop looks like:
uuid_t relatedVertexId;
myProto::IdConfidence* neighborIdConfidence =
pNodeWithNeighbors-
add_neighbors();
// Set the vertex id
neighborIdConfidence-set_id((const void*)
relatedVertexId, 16);
// set the confidence
neighborIdConfidence-set_confidence( confidence );
currentBatchSize++;
if ( currentBatchSize == BatchSize )
{
// Flush out this batch
//stream getNeighborsResponse;
getNeighborsResponse.Clear();
currentBatchSize = 0;
}
On Jul 14, 1:27 am, Kenton Varda ken...@google.com wrote:
Oh, I didn't even know you were including composition in there. My
benchmarks are only for serialization of already-composed messages.
But this still doesn't tell us how much time is spent on network I/O vs.
protobuf serialization. My guess is that once you factor that out,
your performance is pretty close to the benchmarks.
On Mon, Jul 13, 2009 at 10:11 PM, Alex Black a...@alexblack.ca wrote:
If I comment out the actual serialization and sending of the message
(so I am just composing messages, and clearing them each batch) then
the 100ms drops to about 50ms.
On Jul 14, 12:36 am, Alex Black a...@alexblack.ca wrote:
I'm sending a message with about ~150k repeated items in it, total
size is about 3.3mb, and its taking me about 100ms to serialize it
and send it out.
Can I expect to do any better than this? What could I look into to
improve this?
- I have option optimize_for = SPEED; set in my proto file
- I'm compiling with -O3
- I'm sending my message in batches of 1000
- I'm using C++, on ubuntu, x64
- I'm testing all on one machine (e.g. client and server are on
one
machine)
My message looks like:
message NodeWithNeighbors
{
required Id nodeId = 1;
repeated IdConfidence neighbors = 2;
}
message GetNeighborsResponse
{
repeated NodeWithNeighbors nodesWithNeighbors = 1;
}
message IdConfidence
{
required bytes id = 1;
required float confidence = 2;
}
Where bytes id is used to send 16byte IDs (uuids).
I'm writing each message (batch) out like this:
CodedOutputStream codedOutputStream(m_ProtoBufStream);
// Write out the size of the message
codedOutputStream.WriteVarint32(message.ByteSize());
// Ask the message to serialize itself to our stream
adapter,
which
ultimately calls Write on us
// which we then call Write on our composed stream
message.SerializeWithCachedSizes(codedOutputStream
Re: Cannot resolve InternalDescriptorAssigner?
It looks like your protobuf library and protocol compiler binary are from different releases. This won't work. Please make sure both are upgraded to 2.1.0, the latest release. On Tue, Jul 14, 2009 at 8:52 AM, Michael Stapleton mike.staple...@echostar.com wrote: Hi All, I am using release protobuf-java-2.0.0beta.jar and when I generate the java code form my protobuf messages I get the error: com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner cannot be resolved to a type I get this in every Java file generated. I am using Eclipse IDE and generating the code from eclipse using a thread to execute the protoc compiler. I have searched the Internet and there doesn't seem to be anyone else with his error. Do I need a newer java protobuf version and if so where is it documented? Any help would be appreciated. Thanks, Mike --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner cannot be resolved to a type
Answered in the other thread -- you need to use matching protoc and protobuf library versions. On Tue, Jul 14, 2009 at 2:57 PM, Mike mike.staple...@echostar.com wrote: I am getting the error com.google.protobuf.Descriptors.FileDescriptor.InternalDescriptorAssigner cannot be resolved to a type when Eclipse compiles my generated Java code? I wrote a plugin that generates the protocol buffer messages from the ecore file of an EMF Model diagram. It creates a new project, puts the .proto files in a folder and then runs protoc java compiler against the .proto files to create the java code. The java files are put in another folder of the same project under a package file. I put the jar file protobuf- java-2.0.0beta.jar in the Java Build Path libraries of the project I placed the generated java files in. It removes all errors related to protocol buffer code except the one I listed. I have searched the Internet but have not seen this error anywhere? Anyone come across this before? Thanks, Mike --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Compiling on AIX 5.3 using xlC 3.55 compiler
] google.protobuf.DescriptorProto.field: .google.protobuf.FieldDescriptorProto is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.DescriptorProto.extension: .google.protobuf.FieldDescriptorProto is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.DescriptorProto.nested_type: .google.protobuf.DescriptorProto is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.DescriptorProto.enum_type: .google.protobuf.EnumDescriptorProto is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.DescriptorProto.extension_range: .google.protobuf.DescriptorProto.ExtensionRange is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.DescriptorProto.options: .google.protobuf.MessageOptions is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.FieldDescriptorProto.label: .google.protobuf.FieldDescriptorProto.Label is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.FieldDescriptorProto.type: .google.protobuf.FieldDescriptorProto.Type is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.FieldDescriptorProto.options: .google.protobuf.FieldOptions is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.EnumDescriptorProto.value: .google.protobuf.EnumValueDescriptorProto is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.EnumDescriptorProto.options: .google.protobuf.EnumOptions is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.EnumValueDescriptorProto.options: .google.protobuf.EnumValueOptions is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.ServiceDescriptorProto.method: .google.protobuf.MethodDescriptorProto is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.ServiceDescriptorProto.options: .google.protobuf.ServiceOptions is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.MethodDescriptorProto.options: .google.protobuf.MethodOptions is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.FileOptions.optimize_for: .google.protobuf.FileOptions.OptimizeMode is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.FileOptions.uninterpreted_option: .google.protobuf.UninterpretedOption is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.MessageOptions.uninterpreted_option: .google.protobuf.UninterpretedOption is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.FieldOptions.ctype: .google.protobuf.FieldOptions.CType is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.FieldOptions.uninterpreted_option: .google.protobuf.UninterpretedOption is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.EnumOptions.uninterpreted_option: .google.protobuf.UninterpretedOption is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.EnumValueOptions.uninterpreted_option: .google.protobuf.UninterpretedOption is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.ServiceOptions.uninterpreted_option: .google.protobuf.UninterpretedOption is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.MethodOptions.uninterpreted_option: .google.protobuf.UninterpretedOption is not defined. libprotobuf ERROR google/protobuf/descriptor.cc:2218] google.protobuf.UninterpretedOption.name: .google.protobuf.UninterpretedOption.NamePart is not defined. libprotobuf FATAL google/protobuf/descriptor.pb.cc:82] CHECK failed: file != NULL: gmake[2]: *** [unittest_proto_middleman] IOT/Abort trap (core dumped) Any suggestions? Thanks Regards, Vikram On Jul 13, 5:23 pm, Kenton Varda ken...@google.com wrote: And yes, I'd love a patch. :) On Mon, Jul 13, 2009 at 5:22 PM, Kenton Varda ken...@google.com wrote: google/protobuf/stubs/hash.h already contains some hacks for hash_map. To support unordered_map, all we'd have to do is add another hack there which defines hash_map to be a subclass of unordered_map. Subclassing effectively functions as a template typedef here. I would rather not replace the identifier hash_map with unordered_map in the actual code until the Google style guide rules on the issue. I suspect that Google code will go on using hash_map with a similar hack because updating our entire code base is just not worth the effort. On Mon, Jul 13, 2009 at 5:13 PM, Monty Taylor mord...@inaugust.com wrote: vikram wrote: I have found out that with new xlC versions like 8.X onwards hash_map like functionality is supported
Re: assertion crash with debug build on windows
Did you make sure to compile your debug build against a debug build of
libprotobuf? MSVC fails miserably when trying to mix and match debug vs.
release, apparently because the STL classes have different representations
between the two.
On Wed, Jul 15, 2009 at 7:38 AM, Mr Moose stephan.men...@gmail.com wrote:
Hi there,
I am experiencing crashes in a protocol buffers file using debug build
on windows (VC2005). On Linux, debug or not and on Windows release
everything is fine but when I try to run the code on debug it raises
an assertion to crash right afterwards. I'm using 2.1
here's what the stacktrace looks like:
msvcr80d.dll!_crt_debugger_hook(int _Reserved=) Line 62C
msvcr80d.dll!_invoke_watson(const wchar_t *
pszExpression=0x1de19fc0, const wchar_t * pszFunction=0x1de27140,
const wchar_t * pszFile=0x1de197a0, unsigned int nLine=147, unsigned
int pReserved=0) Line 181 + 0x7 bytes C++
msvcr80d.dll!_invalid_parameter(const wchar_t *
pszExpression=0x1de19fc0, const wchar_t * pszFunction=0x1de27140,
const wchar_t * pszFile=0x1de197a0, unsigned int nLine=147, unsigned
int pReserved=0) Line 88 + 0x19 bytes C++
rpc.dll!
std::_Vector_const_iteratorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
,std::allocatorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
::operator+=(int _Off=0) Line 147 + 0x55 bytesC++
rpc.dll!
std::_Vector_iteratorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
,std::allocatorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
::operator+=(int _Off=0) Line 363 C++
rpc.dll!
std::_Vector_iteratorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
,std::allocatorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
::operator+(int _Off=0) Line 369 + 0xc bytes C++
rpc.dll!
std::vectorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
,std::allocatorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
::insert
(std::_Vector_iteratorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
,std::allocatorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
_Where={_Bx={...} _Mysize=??? _Myres=??? }, const
std::basic_stringchar,std::char_traitschar,std::allocatorchar
_Val=MyService.proto) Line 855 + 0x38 bytes C++
my.dll!
std::vectorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
,std::allocatorstd::basic_stringchar,std::char_traitschar,std::allocatorchar
::push_back(const
std::basic_stringchar,std::char_traitschar,std::allocatorchar
_Val=MyService.proto) Line 800 + 0x2d bytes C++
my.dll!google::protobuf::DescriptorBuilder::BuildFile(const
google::protobuf::FileDescriptorProto proto={...}) Line 2715 C++
my.dll!google::protobuf::DescriptorPool::BuildFileFromDatabase(const
google::protobuf::FileDescriptorProto proto={...}) Line 2193 + 0x4a
bytes C++
my.dll!
google::protobuf::DescriptorPool::TryFindFileInFallbackDatabase(const
std::basic_stringchar,std::char_traitschar,std::allocatorchar
name=MyService.proto) Line 1231 + 0x3d bytes C++
my.dll!google::protobuf::DescriptorPool::FindFileByName(const
std::basic_stringchar,std::char_traitschar,std::allocatorchar
name=MyService.proto) Line 875 + 0xc bytes C++
my.dll!pwh::rpc::protobuf_AssignDesc_MyService_2eproto() Line 66 +
0x37 bytes C++
my.dll!google::protobuf::GoogleOnceType::Init(void (void)*
init_func=0x1dac3ddc) Line 73 + 0x5 bytes C++
my.dll!google::protobuf::GoogleOnceInit
(google::protobuf::GoogleOnceType * once=0x1dec2f74, void (void)*
init_func=0x1dac3ddc) Line 104 C++
my.dll!myns::`anonymous namespace'::protobuf_AssignDescriptorsOnce
() Line 292 + 0xf bytesC++
my.dll!myns::MyService::GetDescriptor() Line 3516 C++
It all happens here:
// If we have a fallback_database_, attempt to load all dependencies
now,
// before checkpointing tables_. This avoids confusion with
recursive
// checkpoints.
if (pool_-fallback_database_ != NULL) {
tables_-pending_files_.push_back(proto.name());
for (int i = 0; i proto.dependency_size(); i++) {
if (tables_-FindFile(proto.dependency(i)) == NULL
(pool_-underlay_ == NULL ||
pool_-underlay_-FindFileByName(proto.dependency(i)) ==
NULL)) {
// We don't care what this returns since we'll find out below
anyway.
pool_-TryFindFileInFallbackDatabase(proto.dependency(i));
}
}
tables_-pending_files_.pop_back();
}
So why actually any fallthrough? Is that supposed to happen or an
indication for misusage?
Using 2.0.3 this didn't happen.
Unfortunately I really need to run on debug now to find some other
problem in the code. Everything that comes to mind such as complete
rebuilds I have tried over and over again.
Maybe somebody here
Re: Writing a router tha can route messages
On Wed, Jul 15, 2009 at 10:56 AM, jasonh jas...@google.com wrote: Foo f = Foo.newBuilder().mergeFrom(...).setCookie(newCookieVal).build (); There's actually a shortcut for this: the toBuilder() method of the Message interface returns a Builder that is pre-initialized as a copy of that message. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: GzipOutputStream is slower than gziping the file by hand
Hmm, probably GzipOutputStream is not setting the same compression
parameters as gzip itself uses by default. I'm happy to accept a patch
fixing this. Does the interface (to GzipOutputStream) currently have a way
to control compression parameters? If not, it probably should.
On Wed, Jul 15, 2009 at 3:50 PM, Peter Keen peter.k...@gmail.com wrote:
Hi guys,
I'm playing around with protocol buffers for a project at work and I'm
coming across a possibly weird problem. I have the following setup in
my main():
std::cerr creating file std::endl;
int fd = open(blah.repo, O_WRONLY, O_CREAT);
if ( fd == -1 ) {
std::cerr ERROR: errno strerror(errno)
std::endl;
return 1;
}
ZeroCopyOutputStream* raw_output = new FileOutputStream(fd);
GzipOutputStream* gzip_output = new GzipOutputStream(raw_output,
GzipOutputStream::ZLIB);
CodedOutputStream* coded_output = new CodedOutputStream
(gzip_output);
// CodedOutputStream* coded_output = new CodedOutputStream
(raw_output);
This version takes, say, 8 seconds to create and serialize 100k simple
messages. If I flip it to not use the GzipOutputStream, it takes
roughly 1 second. Using gzip(1) to compress the resulting file takes
less than half a second.
Is there an option I need to be setting to bring it up to parity with
the command-line program or could there be a bug in GzipOutputStream?
For what it's worth, GzipInputStream is roughly on parity with a raw
CodedOutputStream.
Thanks,
Pete
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
On vacation through July 22nd.
Hi all, I will be on vacation and not checking e-mail through July 22nd. I've asked fellow protobuf dev Jason to help answer questions in the meantime, but if you see a question that isn't answered and you know the answer, please feel free to jump in. (That goes for when I'm not on vacation, too...) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Performance: Sending a message with ~150k items, approx 3.3mb, can I do better than 100ms?
If you can find a way to make it faster, please send a patch! :)
On Wed, Jul 15, 2009 at 4:46 PM, Alex Black a...@alexblack.ca wrote:
Thanks, yes performance seems really good, though I wouldn't mind seeing
the java deserialization faster.
--
*From:* Kenton Varda [mailto:ken...@google.com]
*Sent:* Tuesday, July 14, 2009 8:06 PM
*To:* Alex Black
*Cc:* protobuf@googlegroups.com
*Subject:* Re: Performance: Sending a message with ~150k items, approx
3.3mb, can I do better than 100ms?
So, 172 MB/s for composition + serialization. Sounds about right.
On Tue, Jul 14, 2009 at 10:46 AM, Alex Black a...@alexblack.ca wrote:
Thanks for those tips. I am using tcmalloc, and I'm re-using message
for each batch, e.g. I fill it up with say 500 items, send it out, clear it,
re-use it.
Here are my hopefully accurate timings, each done 100 times, averaged:
1. Baseline (just loops through the data on the server) no protobuf: 191ms
2. Compose messages, serialize them, no I/O or deserialization: 213ms
3. Same as #2 but with IO to a dum java client: 265ms
4. Same as #3 but add java protobuf deserialization: 323ms
So from this it looks like:
- composing and serializing the messages takes 22ms
- sending the data over sockets takes 52ms
- deserializing the data in java with protobuf takes 58ms
The amount of data being sent is: 3,959,368 bytes in 158,045 messages
(composed in batches of 1000).
- Alex
--
*From:* Kenton Varda [mailto:ken...@google.com]
*Sent:* Tuesday, July 14, 2009 3:26 AM
*To:* Alex Black
*Cc:* Protocol Buffers
*Subject:* Re: Performance: Sending a message with ~150k items, approx
3.3mb, can I do better than 100ms?
OK. If your message composition (or parsing, on the receiving end)
takes a lot of time, you might look into how much of that is due to memory
allocation. Usually this is a pretty significant fraction. Two good ways
to improve that:
1) If your app builds many messages over time and most of them have
roughly the same shape (i.e. which fields are set, the size of repeated
fields, etc. are usually similar), then you should clear and reuse the same
message object rather than allocate a new one each time. This way it will
reuse the same memory, avoiding allocation.
2) Use tcmalloc:
http://google-perftools.googlecode.com
It is often faster than your system's malloc, particularly for
multi-threaded C++ apps. All C++ servers at Google use this.
On Mon, Jul 13, 2009 at 11:50 PM, Alex Black a...@alexblack.ca wrote:
Kenton: I made a mistake with these numbers - pls ignore them - I'll
revisit tomorrow.
Thx.
-Original Message-
From: protobuf@googlegroups.com [mailto:proto...@googlegroups.com] On
Behalf Of Alex Black
Sent: Tuesday, July 14, 2009 2:05 AM
To: Protocol Buffers
Subject: Re: Performance: Sending a message with ~150k items, approx
3.3mb, can I do better than 100ms?
ok, I took I/O out of the picture by serializing each message into a
pre-allocated buffer, and this time I did a more through measurement.
Benchmark 1: Complete scenario
- average time 262ms (100 runs)
Benchmark 2: Same as # 1 but no IO
- average time 250ms (100 runs)
Benchmark 3: Same as 2 but with serialization commented out
- average time 251ms (100 runs)
Benchmark 4: Same as 3 but with message composition commented out too (no
protobuf calls)
- average time 185 ms (100 runs)
So from this I conclude:
- My initial #s were wrong
- My timings vary too much for each run to really get accurate averages
- IO takes about 10ms
- Serialization takes ~0ms
- Message composition and setting of fields takes ~66ms
My message composition is in a loop, the part in the loop looks like:
uuid_t relatedVertexId;
myProto::IdConfidence* neighborIdConfidence =
pNodeWithNeighbors-
add_neighbors();
// Set the vertex id
neighborIdConfidence-set_id((const void*)
relatedVertexId, 16);
// set the confidence
neighborIdConfidence-set_confidence( confidence
);
currentBatchSize++;
if ( currentBatchSize == BatchSize )
{
// Flush out this batch
//stream getNeighborsResponse;
getNeighborsResponse.Clear();
currentBatchSize = 0;
}
On Jul 14, 1:27 am, Kenton Varda ken...@google.com wrote:
Oh, I didn't even know you were including composition in there. My
benchmarks are only for serialization of already-composed messages.
But this still doesn't tell us how much time is spent on network I/O
vs.
protobuf serialization. My guess is that once you factor that out,
your performance is pretty close to the benchmarks
Re: Java deserialization - any best practices for performances?
On Fri, Jul 17, 2009 at 8:13 PM, Alex Black a...@alexblack.ca wrote:
When I write out messages using C++ I'm careful to clear messages and
re-use them, is there something equivalent on the java side when
reading those same messages in?
No. Sorry. This just doesn't fit at all with the Java library's design,
and even if it did, you cannot reuse Java String objects, which often
account for most of the memory usage. However, memory allocation is cheaper
in Java than in C++, so there's less to gain from it.
My code looks like:
CodedInputStream stream = CodedInputStream.newInstance(inputStream);
while ( !stream.isAtEnd() )
{
MyMessage.Builder builder = MyMessage.newBuilder();
stream.readMessage(builder, null);
MyMessage myMessage = builder.build();
for ( MessageValue messageValue : myMessage.getValuesList() )
{
..
}
}
I'm passing 150 messages each with 1000 items, so presumably memory is
allocated 150 times for each of the messages...
- Alex
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Compiling on AIX 5.3 using xlC 3.55 compiler
It looks like unordered_map or unordered_set is not behaving correctly (e.g.
not finding matching keys when they are present) but I can't really tell
just from that output.
On Mon, Jul 20, 2009 at 11:53 AM, vikram patilvik...@gmail.com wrote:
In previous attempt small test worked but was not able to work with
protocol buffer source code.
I tried one more thing as
#if deinfed (MISSING_HASH_MAP) defined (__xlC__)
#define hash_map std::tr1::unordered_map
google/protobuf/unittest.proto:85:37: Expected ;.
google/protobuf/unittest.proto:117:37: Expected ;.
google/protobuf/unittest.proto:133:55: Expected identifier.
google/protobuf/unittest.proto:134:55: Expected identifier.
google/protobuf/unittest.proto:135:55: Expected identifier.
google/protobuf/unittest.proto:136:55: Expected identifier.
google/protobuf/unittest.proto:137:54: Expected identifier.
google/protobuf/unittest.proto:138:55: Expected identifier.
google/protobuf/unittest.proto:139:55: Expected identifier.
google/protobuf/unittest.proto:140:55: Expected identifier.
google/protobuf/unittest.proto:141:55: Expected identifier.
google/protobuf/unittest.proto:142:54: Expected identifier.
google/protobuf/unittest.proto:143:55: Expected identifier.
google/protobuf/unittest.proto:144:55: Expected identifier.
google/protobuf/unittest.proto:146:54: Expected identifier.
google/protobuf/unittest.proto:147:54: Expected identifier.
google/protobuf/unittest.proto:154:73: Expected identifier.
google/protobuf/unittest.proto:155:57: Expected identifier.
google/protobuf/unittest.proto:192:47: Expected ;.
google/protobuf/unittest.proto:226:47: Expected ;.
google/protobuf/unittest.proto:244:65: Expected identifier.
google/protobuf/unittest.proto:245:65: Expected identifier.
google/protobuf/unittest.proto:246:65: Expected identifier.
google/protobuf/unittest.proto:247:65: Expected identifier.
google/protobuf/unittest.proto:248:64: Expected identifier.
google/protobuf/unittest.proto:249:65: Expected identifier.
google/protobuf/unittest.proto:250:65: Expected identifier.
google/protobuf/unittest.proto:251:65: Expected identifier.
google/protobuf/unittest.proto:252:65: Expected identifier.
google/protobuf/unittest.proto:253:64: Expected identifier.
google/protobuf/unittest.proto:254:65: Expected identifier.
google/protobuf/unittest.proto:255:65: Expected identifier.
google/protobuf/unittest.proto:257:64: Expected identifier.
google/protobuf/unittest.proto:258:64: Expected identifier.
google/protobuf/unittest.proto:268:64: Expected identifier.
google/protobuf/unittest.proto:269:68: Expected identifier.
google/protobuf/unittest.proto:276:42: Expected identifier.
google/protobuf/unittest.proto:379:26: Expected ;.
google/protobuf/unittest.proto:380:26: Expected ;.
google/protobuf/unittest.proto:451:47: Expected identifier.
google/protobuf/unittest.proto:452:47: Expected identifier.
google/protobuf/unittest.proto:453:47: Expected identifier.
google/protobuf/unittest.proto:454:47: Expected identifier.
google/protobuf/unittest.proto:455:47: Expected identifier.
google/protobuf/unittest.proto:460:46: Expected identifier.
gmake: *** [unittest_proto_middleman] Error 1
So I guess it wont even compile with unordered_map provided on Linux .
Please provide some inputs on this one.
Thanks Regards,
Vikram
On Jul 14, 6:35 pm, Kenton Varda ken...@google.com wrote:
It looks like your implementation of hash_map is not working correctly --
all lookups are failing. You might try writing a little test for
hash_map
itself that would be easier to debug.
On Tue, Jul 14, 2009 at 6:27 PM, vikram patilvik...@gmail.com wrote:
Kenton Monty,
I added hack as followes in the hash.h
// File changed .
#if defined(HAVE_HASH_MAP) defined(HAVE_HASH_SET)
#include HASH_MAP_H
#include HASH_SET_H
#elif defined (__xlC__)
#define MISSING_HASH
#include unordered_map
#include unordered_set
#else
#define MISSING_HASH
#include map
#include set
#endif
namespace google {
namespace protobuf {
#if defined(MISSING_HASH) defined(__xlC__)
//@TODO
//Inherit hash_map from unordered_map
template typename Key
struct hash : public std::tr1::hashKey {
};
template typename Key
struct hashconst Key* {
inline size_t operator()(const Key* key) const {
return reinterpret_castsize_t(key);
}
};
template typename Key, typename Data,
typename HashFcn = hashKey,
typename EqualKey = std::equal_toKey
class hash_map : public std::tr1::unordered_mapKey, Data, HashFcn ,
EqualKey {
};
template typename Key,
typename HashFcn = hashKey,
typename EqualKey = std::equal_toKey
class hash_set : public std::tr1::unordered_set
Key, HashFcn, EqualKey {
};
#elif defined(MISSING_HASH)
File continues as it is
Stack trace
pthread_kill(??, ??) at 0xd01246b4
_p_raise
Re: Java deserialization - any best practices for performances?
On Thu, Jul 23, 2009 at 12:32 AM, alopecoid alopec...@gmail.com wrote: Hi, I haven't actually used the Java protobuf API, but it seems to me from the quick occasional glance that this isn't entirely true. I mean, specifically in response to the code snippet posted in the original message, I would possibly: 1. Reuse the Builder object by calling its clear() method. This would save from the need to create a new Builder object for each iteration of the outermost loop. You can't continue to use a Builder after calling build(). Even if we made it so you could, it would be building an entirely new object, not reusing the old one. We can't make it reuse the old one because that would break the immutability guarantee of message objects. Reusing the actual builder object is not that useful since it's only a very small object containing a pointer to a message object. 2. Iterate over the repeated field using the get*Count() and get* (index) methods instead of the get*List() method. I'm not sure if this would save anything, but depending on how things are implemented in the generated code, this could save from allocating a new List object. Won't save anything; we still need a list object internally. But seriously, object allocation with a modern generational garbage collector is extremely cheap, especially for objects that don't stick around very long. So I don't think there's much to gain here. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Java deserialization - any best practices for performances?
On Thu, Jul 23, 2009 at 7:15 PM, alopecoid alopec...@gmail.com wrote: Hmm... that strikes me as strange. I understand that the Message objects are immutable, but the Builders are as well? I thought that they would work more along the lines of String and StringBuilder, where String is obviously immutable and StringBuilder is mutable/ reusable. The point is that it's the Message object that contains all the stuff allocated by the Builder, and therefore none of that stuff can actually be reused. (When you call build(), nothing is copied -- it just returns the object that it has been working on.) So reusing the builder itself is kind of useless, because it's just a trivial object containing one pointer (to the message object it is working on constructing). But while we're on the subject, I have been looking for some rough benchmarks comparing the performance of Protocol Buffers in Java versus C++. Do you (the collective you) have any [rough] idea as to how they compare performance wise? I am thinking more in terms of batch-style processing (disk I/O, parsing centric) rather than RPC centric usage patterns. Any experiences you can share would be great. I have some benchmarks that IIRC show that Java parsing and serialization is roughly half the speed of C++. As I recall a lot of the speed difference is from UTF-8 decoding/encoding -- in C++ we just leave the bytes encoded, but in Java we need to decode them in order to construct standard String objects. I've been planning to release these benchmarks publicly but it will take some work and there's a lot of higher-priority stuff to do. :/ (I think Jon Skeet did get the Java side of the benchmarks into SVN but there's no C++ equivalent yet.) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: java string literal too long when initializing java.lang.String descriptorData
How annoying. I'll make sure this or something like it gets into the next
release -- which I'm going to try to push next week.
On Wed, Jul 22, 2009 at 8:36 AM, anonymous eric.pe...@hp.com wrote:
Hello,
I was not able to compile a Java file generated by protoc 2.10 from a
rather big .proto file.
It seems I have hit the upper limit for a Java string literal
(65535???).
I slightly modified src/google/protobuf/compiler/java/java_file.cc so
that static initialization is performed from
an array of literal strings in the case CEscape(file_data).size()
65535.
Is this a real problem, or am I missing something ?
Here is the patch:
diff -r -u protobuf-2.1.0/src/google/protobuf/compiler/java/
java_file.cc protobuf-2.1.0.new/src/google/protobuf/compiler/java/
java_file.cc
--- protobuf-2.1.0/src/google/protobuf/compiler/java/java_file.cc
2009-05-13 16:36:30.0 -0400
+++ protobuf-2.1.0.new/src/google/protobuf/compiler/java/java_file.cc
2009-07-22 10:37:28.0 -0400
@@ -207,6 +207,9 @@
// This makes huge bytecode files and can easily hit the compiler's
internal
// code size limits (error code to large). String literals are
apparently
// embedded raw, which is what we want.
+ // In the case the FileDescriptorProto is too big for fitting into
a string
+ // literal, first creating ain array of string literals, then
concatenating
+ // them into the final FileDescriptorProto string.
FileDescriptorProto file_proto;
file_-CopyTo(file_proto);
string file_data;
@@ -218,22 +221,51 @@
return descriptor;\n
}\n
private static com.google.protobuf.Descriptors.FileDescriptor\n
-descriptor;\n
-static {\n
- java.lang.String descriptorData =\n);
- printer-Indent();
- printer-Indent();
+descriptor;\n);
- // Only write 40 bytes per line.
static const int kBytesPerLine = 40;
- for (int i = 0; i file_data.size(); i += kBytesPerLine) {
-if (i 0) printer-Print( +\n);
-printer-Print(\$data$\,
- data, CEscape(file_data.substr(i, kBytesPerLine)));
- }
- printer-Print(;\n);
- printer-Outdent();
+ // Limit for a Java literal string is 65535
+ bool stringTooLong = (CEscape(file_data).size() 65535);
+
+ if (stringTooLong) {
+printer-Print(static {\n
+java.lang.String descriptorDataArray[] = {\n);
+printer-Indent();
+printer-Indent();
+
+// Only write 40 bytes per line.
+for (int i = 0; i file_data.size(); i += kBytesPerLine) {
+ if (i 0) printer-Print(,\n);
+ printer-Print(\$data$\,
+data, CEscape(file_data.substr(i, kBytesPerLine)));
+}
+printer-Outdent();
+printer-Print(\n
+};\n\n);
+printer-Print(java.lang.String descriptorData = \\;\n);
+printer-Print(for (String data : descriptorDataArray) {\n);
+printer-Indent();
+printer-Print(descriptorData += data;\n);
+printer-Outdent();
+printer-Print(}\n\n);
+ } else {
+printer-Print(static {\n
+java.lang.String descriptorData =\n);
+printer-Indent();
+printer-Indent();
+
+// Only write 40 bytes per line.
+static const int kBytesPerLine = 40;
+for (int i = 0; i file_data.size(); i += kBytesPerLine) {
+ if (i 0) printer-Print( +\n);
+printer-Print(\$data$\,
+data, CEscape(file_data.substr(i, kBytesPerLine)));
+ }
+printer-Print(;\n);
+
+printer-Outdent();
+ }
//
-
// Create the InternalDescriptorAssigner.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Compiler for the Android Protocol Buffers implementation?
That implementation was written completely independently of the protocol buffers team at google, and is not maintained by us. I've heard that it is buggy, and it doesn't support many of the newer language features, though since I've never used it myself or looked at the code I'm not sure if that's true. In any case, as far as I know they have not released the compiler publicly -- but the Android people would know more about this than I would. Hopefully the lite mode in upcoming protobuf version 2.2.0 (which I hope to get out within the next couple weeks) will provide a better solution for Android developers. It no longer triggers this Dalvik VM bug, and is generally much better suited to mobile platforms, while still providing mostly the same API. On Sat, Jul 25, 2009 at 5:52 AM, Michael Elsdörfer elsdoer...@gmail.comwrote: Due to a Dalvik VM bug, you can apparently run into deadlocks when using Protocol Buffers on Android (http://groups.google.com/group/ android-developers/browse_thread/thread/c7b68765a02f74dfhttp://groups.google.com/group/%0Aandroid-developers/browse_thread/thread/c7b68765a02f74df ). A member of the Android team suggested (http://groups.google.com/group/ android-developers/msg/978cd88517a5fc25http://groups.google.com/group/%0Aandroid-developers/msg/978cd88517a5fc25) that one should possibly use a different Java implementation that Android also uses internally : http://android.git.kernel.org/?p=platform/external/protobuf.git;a=tree;f=src/com/google/common/io/protocol;h=44814a7cc40a99cf6a866e219793976de8f283a7;hb=HEAD However, it looks like the code that is generated by the official compiler isn't compatible with that implementation. Does a compatible compiler exist somewhere (in the open)? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: compile errors when building against protobuf headers
What compiler flags are you using? It looks like you have an excessively pedantic warning mode enabled, with -Werror to make warnings into errors, and it is applying even to system headers (usually warnings in system headers aren't reported). On Mon, Jul 27, 2009 at 6:27 PM, achap achappe...@gmail.com wrote: forgot to mention... I'm using protobuf version 2.1.0 On Jul 27, 6:25 pm, achap achappe...@gmail.com wrote: anyone else experience compile errors when building against the descriptor.h and wire_format.h files? error messages below: include/google/protobuf/descriptor.h:322: error: comma at end of enumerator list include/google/protobuf/descriptor.h:341: error: comma at end of enumerator list include/google/protobuf/descriptor.h:352: error: comma at end of enumerator list include/google/protobuf/descriptor.h:1202: error: extra ';' include/google/protobuf/descriptor.h:1219: error: extra ';' include/google/protobuf/descriptor.h:1239: error: extra ';' include/google/protobuf/descriptor.h:1245: error: extra ';' include/google/protobuf/descriptor.h:1253: error: extra ';' include/google/protobuf/descriptor.h:1260: error: extra ';' include/google/protobuf/descriptor.h:1270: error: extra ';' include/google/protobuf/wire_format.h:176: error: comma at end of enumerator list make[1]: *** [.obj/Linux/32/MyFile.o] Error 1 make: *** [all] Error 2 Im using gcc 4.1.2 on RedHat rhel 5.3 I can fix the headers to make them work, but I just figured I must be doing something incorrectly if I had to do so. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: compile errors when building against protobuf headers
Again, GCC will ignore warnings as long as it recognizes the protobuf headers as system headers. You can explicitly mark an include directory as containing system headers by using -isystem instead of -I. On Mon, Jul 27, 2009 at 7:02 PM, achap achappe...@gmail.com wrote: I am not using -Werror, but I am using -W -pedantic, which apparently is the problem. what is the likely hood that this will work with the pedantic warning enabled? My build system forces pendant... I can override it, but I'd rather not if I don't have too. Thanks. On Jul 27, 6:39 pm, Kenton Varda ken...@google.com wrote: What compiler flags are you using? It looks like you have an excessively pedantic warning mode enabled, with -Werror to make warnings into errors, and it is applying even to system headers (usually warnings in system headers aren't reported). On Mon, Jul 27, 2009 at 6:27 PM, achap achappe...@gmail.com wrote: forgot to mention... I'm using protobuf version 2.1.0 On Jul 27, 6:25 pm, achap achappe...@gmail.com wrote: anyone else experience compile errors when building against the descriptor.h and wire_format.h files? error messages below: include/google/protobuf/descriptor.h:322: error: comma at end of enumerator list include/google/protobuf/descriptor.h:341: error: comma at end of enumerator list include/google/protobuf/descriptor.h:352: error: comma at end of enumerator list include/google/protobuf/descriptor.h:1202: error: extra ';' include/google/protobuf/descriptor.h:1219: error: extra ';' include/google/protobuf/descriptor.h:1239: error: extra ';' include/google/protobuf/descriptor.h:1245: error: extra ';' include/google/protobuf/descriptor.h:1253: error: extra ';' include/google/protobuf/descriptor.h:1260: error: extra ';' include/google/protobuf/descriptor.h:1270: error: extra ';' include/google/protobuf/wire_format.h:176: error: comma at end of enumerator list make[1]: *** [.obj/Linux/32/MyFile.o] Error 1 make: *** [all] Error 2 Im using gcc 4.1.2 on RedHat rhel 5.3 I can fix the headers to make them work, but I just figured I must be doing something incorrectly if I had to do so. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Lite mode is in SVN -- mostly
I just committed the new lite mode refactoring for C++ and Java to SVN. To use the new feature, add this line to your .proto file: option optimize_for = LITE_RUNTIME; This will cause protoc to generate code which is just as fast as with optimize_for = SPEED, but lacks descriptors, reflection, UnknownFieldSet, and features which are tied to them. This means the code can be linked against a much smaller subset of the protocol buffers runtime library and will not spend time at start-up building descriptors. Currently, the C++ Makefile will compile libprotobuf and libprotobuf-lite as independent, stand-alone libraries, even though the latter is a subset of the former. Is this what we want? I'm not sure. If we make libprotobuf depend on libprotobuf-lite, then users who want the full library will have to specify *both* when linking their own apps, unless we provide a protobuf-config script or something like that. Complicating things further, if you app does not use extensions, then extension_set.cc can be removed from the lite library to make it even smaller, but having three separate libraries just seems excessive! For Java, I have not yet updated the Maven POM to separate the libraries. I'm not sure how -- Greg, can you work on this? Lite messages need only the following classes: AbstractMessageLite.java ByteString.java CodedInputStream.java CodedOutputStream.java ExtensionRegistryLite.java FieldSet.java GeneratedMessageLite.java InvalidProtocolBufferException.java Internal.java MessageLite.java UninitializedMessageException.java WireFormat.java *Numbers* C++, as measured by the size command on Linux: libprotobuf.so: 948k libprotobuf-lite.so: 148k libprotobuf-lite.so with extension_set.cc removed: 91k Java, measured by totalling the sizes of the .class files: full library: 904k lite library: 120k I suspect we can further reduce some of these numbers with some more work. For instance, an extension-less Java lite library might be as much as 50k smaller, though will require a bit more work than simply yanking some classes. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: How to sort the repeated element?
In C++, you can use the standard STL sort() function:
http://www.sgi.com/tech/stl/sort.html
http://www.sgi.com/tech/stl/sort.htmlRepeatedPtrField supports iterators,
so you can do something like:
#include algorithm
struct StudentOrdering {
inline operator()(const Student* a, const Student* b) {
return a-score() b-score();
}
}
void SortStudents(Class* cls) {
std::sort(cls-mutable_student()-begin(),
cls-mutable_student()-end(),
StudentOrdering());
}
On Tue, Jul 28, 2009 at 8:03 PM, 李海波 lihaibo2...@gmail.com wrote:
example:
==.proto file
message Class{
message Student{
required string name = 1;
required int32 score = 2;
}
repeated Student student = 1;
}
=
I want to sort students of the class by score,how can i do?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: upgrade to latest release
On Wed, Jul 29, 2009 at 12:00 PM, Sushil Shelly skshe...@gmail.com wrote: thanks for the heads up, upgrading to 2.1.0 noticed that the c++ example now requires -lpthread, once I compile my application it will need to know about pthread also. Technically this was always needed, but on some systems the pthread mutex functions happen to be in the C library, so you could get away without it. 2.1.0 also uses pthread_once, though, which is not typically in the C library. is there a set of release notes that i need to be aware of, Just CHANGES.txt. and also are there any patches against 2.1.0 that I may need, if so how do i find out what they are? No, if there were critical problems we'd do another release to fix them. will make uninstall take out all flavors of protocol buffer libraries (I used make install on top of a prior install and would like to remove the older version) I'm not completely sure. make install installs a versioned library, then symlinks libprotobuf.so to the version lib. Installing a new version would update the symlink. Uninistalling the old version might delete the symlink. But if you uninstall the old and then re-install the new (just re-run make install) then everything should be good. thanks On Wed, Jul 29, 2009 at 2:22 PM, Kenton Varda ken...@google.com wrote: On Wed, Jul 29, 2009 at 4:14 AM, Sushil Shelly skshe...@gmail.comwrote: I need to upgrade a linux installation of protocol buffers. I have 2.0.3 and need to move up to use 2.1.0 - is there a set procedure to do this? You can just install it normally. You may also want to recompile any apps based on it, but this is not strictly necessary. Although different versions of libprotobuf.so are not binary-compatible (C++ sadly makes compatibility too difficult to achieve), we use libtool's version numbering to make sure that binary-incompatible versions of libprotobuf do not overwrite each other. So, any binaries you compiled against the old version will continue to use the old version at runtime. If you want to uninstall the old version, then run make uninstall from the same directory where you you ran make install to install it. BTW, 2.2.0 will be out in the next week or two. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: upgrade to latest release
On Wed, Jul 29, 2009 at 11:49 AM, Monty Taylor mord...@inaugust.com wrote: A little off-topic - but do you have a cutoff date for patches to get in to that? I've still got some Sun Studio issues I've been working on a patch for that I'd love to be in 2.2.0... There's no official schedule, so I can wait. Can you get them to me in the next week? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Installation using cygwin fails
On Wed, Jul 29, 2009 at 11:56 AM, Javier espinosa.javie...@gmail.comwrote: Warning: This system can not link to static lib archive libprotobuf.la. i have the capability to make that librar automatically link in when you link to this library. But I cna only do htis if you have a shared versino of the library, which you do not appear to have. libtool: link: warning: undefined symbols not allowed in i686-pc- cygwin shared libraries These warnings are normal when compiling on Windows. You can probably avoid them by passing --disable-shared to configure, which won't hurt anything because libtool will refuse to build shared libraries anyway. eventually it says there is a libprotobuf.a file inside /.libs dir that cannot be found (no such file or directory) and then stops executing. The file IS there, I checked. I have not seen that in my testing. What exact arguments did you pass to configure and make? Can you send the full error log? (I doubt I'll be able to figure it out but maybe there's something there.) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: possible bug?
Probably you are treating the bytes as a NUL-terminated string somewhere,
and so the message is being cut short at the zero. Protocol messages can
contain NUL (zero-value) bytes, so you cannot pass an encoded protocol
message as a plain char* -- you must keep track of the size.
On Wed, Jul 29, 2009 at 4:48 PM, Sushil Shelly skshe...@gmail.com wrote:
I have a simple enumeration in my dot-proto as follows
message Bar {
required string name = 1;
required string device = 2;
}
enum msg State {
IDLE = 0;
BUSY = 1;
}
message Foo {
required State state = 1;
required Bar bar = 2;
}
now i set the required fields by using the helper methods provided by the
object;
the state is set to IDLE in the message.
On Receiving () - I de-serialize but nothing shows up in the msg, its as if
the required field was not set.
Then in another test case I set the state to Busy and send - nothing else
has changed,
this time I get the message and I am able to see all information.
Is this a known issue?
enum msg State {
DO_NOT_USE = 0;
IDLE = 1;
BUSY = 2;
}
IS THE WORK AROUND I AM USING
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: possible bug?
Also, in all likelihood the ParseFrom*() call is failing, but you aren't
checking the return value.
On Wed, Jul 29, 2009 at 4:54 PM, Kenton Varda ken...@google.com wrote:
Probably you are treating the bytes as a NUL-terminated string somewhere,
and so the message is being cut short at the zero. Protocol messages can
contain NUL (zero-value) bytes, so you cannot pass an encoded protocol
message as a plain char* -- you must keep track of the size.
On Wed, Jul 29, 2009 at 4:48 PM, Sushil Shelly skshe...@gmail.com wrote:
I have a simple enumeration in my dot-proto as follows
message Bar {
required string name = 1;
required string device = 2;
}
enum msg State {
IDLE = 0;
BUSY = 1;
}
message Foo {
required State state = 1;
required Bar bar = 2;
}
now i set the required fields by using the helper methods provided by the
object;
the state is set to IDLE in the message.
On Receiving () - I de-serialize but nothing shows up in the msg, its as
if the required field was not set.
Then in another test case I set the state to Busy and send - nothing else
has changed,
this time I get the message and I am able to see all information.
Is this a known issue?
enum msg State {
DO_NOT_USE = 0;
IDLE = 1;
BUSY = 2;
}
IS THE WORK AROUND I AM USING
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: ExtensionRegistry usage.
The Builder interface has a method setField() which takes a FieldDescriptor
and a value. You can pass the FieldDescriptor for an extension to set an
extension, and pass the message object as the value. You can get the
FieldDescriptor from ExtensionRegistry or from
GeneratedExtension.getDescriptor().
On Thu, Jul 30, 2009 at 3:29 AM, Brice Figureau
brice...@daysofwonder.combrice%2...@daysofwonder.com
wrote:
Hi,
I'm just starting using Protocol Buffers for Java for a network
protocol.
Here is how I designed my proto file:
request.proto:
message Message {
enum Type {
AUTH = 100;
CHAT = 101;
...
}
required Type type = 1;
optional int32 id = 2;
extensions 100 to max;
}
message AuthRequest {
extend Message {
optional AuthRequest auth_request = 100;
}
required string name = 1;
required string pass = 2;
}
message ChatRequest {
extend Message {
optional ChatRequest chat_request = 101;
}
required int32 player_id = 1;
optional string message_string = 2;
}
This is based on the Union Types technique described in the
documentation.
Now, all is well, I can do:
ChatRequest chat =
ChatRequest.newBuilder().setMessage(msg).setPlayerId(id).build();
Message request =
Message.newBuilder().setType(Type.CHAT).setExtension(ChatRequest.chatRequest,
fChatRequest).build();
and then serialize request to the wire.
What I'm trying to achieve now is to generalize this operation for all
my defined request types, using the descriptors and registry extension:
ExtensionRegistry registry = ExtensionRegistry.newInstance();
Request.registerAllExtensions(registry);
...
Then I can create generically a builder for a given type with:
public Builder getBuilder(Type type)
{
ExtensionRegistry.ExtensionInfo info =
registry.findExtensionByNumber(Message.getDescriptor(), type.getNumber());
return info.defaultInstance.newBuilderForType();
}
And on client call site:
ChatRequest chat = (ChatRequest.Builder)getBuilder(Type.CHAT);
chat.setPlayerId(id)...
Now, I'm trying to generically associate this ChatRequest to the correct
Extension field in the Message type, but unfortunately I can't find a
way of doing that, because setExtension() requires a
GeneratedMessage.GeneratedExtension which I can't have generically (ie
the extension registry even though generated with this information,
doesn't keep it).
What's worse is tha the Extension Registry has all the information, but
I can't recreate this specific generic type or add the extension from
the Registry information.
First question: how can I solve this?
Second question: if it's not solvable, then that means I'm not using PB
as it was intended. In this case, how should I use it?
Thanks,
--
Brice Figureau
My Blog: http://www.masterzen.fr/
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Add a protobuf-config script like the old gtk-config
On Wed, Jul 29, 2009 at 6:34 PM, Jeff Bailey jeffbai...@google.com wrote: *sigh* It looks like the version at appspot.com isn't GA+ enabled, so I sign in and it thinks I'm not signed in. Anyhow, a few comments: Since it's generated by configure.ac, do you need it in bin_SCRIPTS? I think that might cause it to get looked at twice. The purpose of putting it in bin_SCRIPTS is to make sure that it is installed, which configure is not going to do automatically. The automake docs say that bin_SCRIPTS are by default not included in the dist, which is what we want here (since configure generates it). You should pretty much always do a set -e at the top of a shell script to catch errors early on. Oops, fixed. *73* if test @prefix@ != /usr -a @prefix != / -a @prefix != ; then Should those all be @pre...@? Yes. :/ Also, I think test -a might be a bashism in this case. Changed to test. Same for this line: *79* if test $full_library = true -o $explicit_library = false; then Done. Also, I added --ldflags as a separate option since LDFLAGS and LIBS are traditionally separate. Not sure why gtk-config itself does not do this. Also also, I expanded the help text. Also^3, I made --version strip the suffix (-pre, rc1, etc.) since I doubt anyone will correctly parse it otherwise (since people will code against official releases which have no suffix). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Add a protobuf-config script like the old gtk-config
(New patch set uploaded.) On Thu, Jul 30, 2009 at 8:59 AM, Kenton Varda ken...@google.com wrote: On Wed, Jul 29, 2009 at 6:34 PM, Jeff Bailey jeffbai...@google.comwrote: *sigh* It looks like the version at appspot.com isn't GA+ enabled, so I sign in and it thinks I'm not signed in. Anyhow, a few comments: Since it's generated by configure.ac, do you need it in bin_SCRIPTS? I think that might cause it to get looked at twice. The purpose of putting it in bin_SCRIPTS is to make sure that it is installed, which configure is not going to do automatically. The automake docs say that bin_SCRIPTS are by default not included in the dist, which is what we want here (since configure generates it). You should pretty much always do a set -e at the top of a shell script to catch errors early on. Oops, fixed. *73* if test @prefix@ != /usr -a @prefix != / -a @prefix != ; then Should those all be @pre...@? Yes. :/ Also, I think test -a might be a bashism in this case. Changed to test. Same for this line: *79* if test $full_library = true -o $explicit_library = false; then Done. Also, I added --ldflags as a separate option since LDFLAGS and LIBS are traditionally separate. Not sure why gtk-config itself does not do this. Also also, I expanded the help text. Also^3, I made --version strip the suffix (-pre, rc1, etc.) since I doubt anyone will correctly parse it otherwise (since people will code against official releases which have no suffix). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Message type scope resolution
In other words, the scoping semantics are just like C++ namespaces, but we
use . rather than :: as the scope separator.
On Thu, Jul 30, 2009 at 1:50 AM, Constantinos Michael
cmmich...@gmail.comwrote:
Try doing this:
message foo {
message X { }
message bar {
message X { }
message abc {
*optional foo.X abc_x = 1;*
}
}
}
You can also prefix the type with a dot, e.g.
optional *.**foo.X* abc_x = 1;
That has the effect of fully qualifying the message, as you can enclose the
entire message foo above in another message foo, which also provides an X,
and have these fields in the innermost message:
optional *.**foo.X* x1 = 1; // outer foo
optional *.**foo.foo.X* x2 = 2; // inner foo
optional *foo.X* x3 = 3; // inner foo
-Constantinos
On Thu, Jul 30, 2009 at 5:22 AM, naderp paul.na...@gmail.com wrote:
Hi,
Does the protocol buffer language have a mechanism for resolving types
across scopes. Eg, given
message foo {
message X {}
message bar {
message X{}
message abc {
message foo:X--- This is not legal, but how
do you do it?
}
}
}
Thanks.
--
- Constantinos Michael
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Installation using cygwin fails
When you do it manually, does the program return an error code (even if it seems to work)? Maybe you could run it in gdb to find out why it is returning an error? (If you do this, be sure to use a protoc that you compiled yourself, not the prebuild one.) Unfortunately since I can't reproduce this locally, I can't debug it myself. On Thu, Jul 30, 2009 at 9:00 AM, Javier espinosa.javie...@gmail.com wrote: doing it manually seems to work. the problem is that those instructions appear to be part of the make and even if I run them manually when I run the make command again it still tries to execute them and fails. I have also tried to find out what make Error 127 means but I still haven't had any luck with that. And to think I did all of this on my Mac on the first try *sigh... On Jul 30, 10:14 am, Kenton Varda ken...@google.com wrote: I don't see an actual error message in the log. Did it not print one? If not, what happens when you run protoc manually? On Thu, Jul 30, 2009 at 7:17 AM, Javier espinosa.javie...@gmail.com wrote: So that fixed that. Sadly something else came up. c:/mingw/bin/mingw32-make all-recursive mingw32-make[1]: Entering directory `C:/appl/Cygwin/protobuf' Making all in . mingw32-make[2]: Entering directory `C:/appl/Cygwin/protobuf' mingw32-make[2]: Nothing to be done for `all-am'. mingw32-make[2]: Leaving directory `C:/appl/Cygwin/protobuf' Making all in src mingw32-make[2]: Entering directory `C:/appl/Cygwin/protobuf/src'./ protoc.exe google/protobuf/unittest.proto google/protobuf/ unittest_empty.proto google/protobuf/unittest_import.proto google/protobuf/ unittest_mset.proto google/protobuf/unittest_optimize_for.proto google/protobuf/unittest_embed_optimize_for.proto google/protobuf/ unittest_custom_options.proto google/protobuf/compiler/cpp/ cpp_test_bad_identifiers.proto -I. --cpp_out=. mingw32-make[2]: *** [unittest_proto_middleman] Error 127 mingw32-make[2]: Leaving directory `C:/appl/Cygwin/protobuf/src' mingw32-make[1]: *** [all-recursive] Error 1 mingw32-make[1]: Leaving directory `C:/appl/Cygwin/protobuf' mingw32-make: *** [all] Error 2 My guess is the protoc compiler is failing at some point but I'm not sure why. I even tried changing the protoc.exe file for the one that's available for download but it gave me the same error. On Jul 29, 3:28 pm, Kenton Varda ken...@google.com wrote: Odd. Here's a longshot idea: Try compiling from inside the Cygwin tree, e.g. create a /src directory and unpack the protobuf package there. This would avoid any possible confusion between C:/ vs. /cygdrive/c. But I don't know if it will actually help. On Wed, Jul 29, 2009 at 12:27 PM, Javier espinosa.javie...@gmail.com wrote: eventually it says there is a libprotobuf.a file inside /.libs dir that cannot be found (no such file or directory) and then stops executing. The file IS there, I checked. I have not seen that in my testing. What exact arguments did you pass to configure and make? Can you send the full error log? (I doubt I'll be able to figure it out but maybe there's something there.) I just did ./configure then make. no arguments. This is the error log (edited for readability): libtool: link: ar cru .libs/libprotoc.a code_generator.o command_line_interface.o cpp_enum.o cpp_enum_field.o cpp_extension.o cpp_field.o cpp_file.o cpp_generator.o cpp_helpers.o cpp_message.o cpp_message_field.o cpp_primitive_field.o cpp_service.o cpp_string_field.o java_enum.o java_enum_field.o java_extension.o java_field.o java_file.o java_generator.o java_helpers.o java_message.o java_message_field.o java_primitive_field.o java_service.o python_generator.o libtool: link: ranlib .libs/libprotoc.a libtool: link: ( cd .libs rm -f libprotoc.la ln -s ../ libprotoc.la libprotoc.la ) C:/appl/Cygwin/bin/sh.exe ../libtool --tag=CXX --mode=link g++ - Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -O2 -g - DNDEBUG -o protoc.exe main.o libprotobuf.la libprotoc.la libtool: link: g++ -Wall -Wwrite-strings -Woverloaded-virtual -Wno- sign-compare-O2 -g -DNDEBUG -o protoc.exe main.o ./.libs/ libprotobuf.a ./.libs/libprotoc.a /cygdrive/c/protobuf/src/.libs/libprotobuf.a g++.exe: /cygdrive/c/protobuf/src/.libs/libprotobuf.a: No such file or directory mingw32-make[2]: *** [protoc.exe] Error 1 mingw32-make[2]: Leaving directory `c:/protobuf/src' mingw32-make[1]: *** [all-recursive] Error 1 mingw32-make[1]: Leaving directory `c:/protobuf' mingw32-make: *** [all] Error 2 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email
Re: Add a protobuf-config script like the old gtk-config
Yeargh, I'm behind the times. pkg-config? I guess I should be integrating with that rather than writing my own script? On Thu, Jul 30, 2009 at 9:05 AM, Jeff Bailey jeffbai...@google.com wrote: lgtm 42 Packages which depend on Protocol Buffers should call this script automatically 43 as part of their own configure script. Provide an example with PKG_CONFIG or something like that. Otherwise, looks good. Thanks! Jeff Bailey | Google, Inc. | +1 514 670-8754 On Thu, Jul 30, 2009 at 12:00 PM, Kenton Varda ken...@google.com wrote: (New patch set uploaded.) On Thu, Jul 30, 2009 at 8:59 AM, Kenton Varda ken...@google.com wrote: On Wed, Jul 29, 2009 at 6:34 PM, Jeff Bailey jeffbai...@google.comwrote: *sigh* It looks like the version at appspot.com isn't GA+ enabled, so I sign in and it thinks I'm not signed in. Anyhow, a few comments: Since it's generated by configure.ac, do you need it in bin_SCRIPTS? I think that might cause it to get looked at twice. The purpose of putting it in bin_SCRIPTS is to make sure that it is installed, which configure is not going to do automatically. The automake docs say that bin_SCRIPTS are by default not included in the dist, which is what we want here (since configure generates it). You should pretty much always do a set -e at the top of a shell script to catch errors early on. Oops, fixed. *73* if test @prefix@ != /usr -a @prefix != / -a @prefix != ; then Should those all be @pre...@? Yes. :/ Also, I think test -a might be a bashism in this case. Changed to test. Same for this line: *79* if test $full_library = true -o $explicit_library = false; then Done. Also, I added --ldflags as a separate option since LDFLAGS and LIBS are traditionally separate. Not sure why gtk-config itself does not do this. Also also, I expanded the help text. Also^3, I made --version strip the suffix (-pre, rc1, etc.) since I doubt anyone will correctly parse it otherwise (since people will code against official releases which have no suffix). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: problem with the previous version
I need a lot more information to even begin speculating about this, such as what language you are using and any error messages that were printed when it crashed. A stack trace would also be nice. On Thu, Jul 30, 2009 at 9:46 AM, muka ly.pas...@gmail.com wrote: Hi, I have a problem with the previous version of google protocol buffer. I serialize an object of type X into a file A, then when trying to parse the file A with another object of type X, it crashs. The object of type X contains a lot of data, and repeated element. Is there some kind of possibility that it is linked to some security mechanisms ? cheers, pascal ly. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Add a protobuf-config script like the old gtk-config
New patch: Use pkg-config instead. Much simpler. On Thu, Jul 30, 2009 at 10:05 AM, Jeff Bailey jeffbai...@google.com wrote: I haven't done pkg-config stuff yet. Maybe check in on #autotools in freenode. I'm usually there, as are other people. Jeff Bailey | Google, Inc. | +1 514 670-8754 On Thu, Jul 30, 2009 at 12:12 PM, Kenton Varda ken...@google.com wrote: Yeargh, I'm behind the times. pkg-config? I guess I should be integrating with that rather than writing my own script? On Thu, Jul 30, 2009 at 9:05 AM, Jeff Bailey jeffbai...@google.comwrote: lgtm 42 Packages which depend on Protocol Buffers should call this script automatically 43 as part of their own configure script. Provide an example with PKG_CONFIG or something like that. Otherwise, looks good. Thanks! Jeff Bailey | Google, Inc. | +1 514 670-8754 On Thu, Jul 30, 2009 at 12:00 PM, Kenton Varda ken...@google.comwrote: (New patch set uploaded.) On Thu, Jul 30, 2009 at 8:59 AM, Kenton Varda ken...@google.comwrote: On Wed, Jul 29, 2009 at 6:34 PM, Jeff Bailey jeffbai...@google.comwrote: *sigh* It looks like the version at appspot.com isn't GA+ enabled, so I sign in and it thinks I'm not signed in. Anyhow, a few comments: Since it's generated by configure.ac, do you need it in bin_SCRIPTS? I think that might cause it to get looked at twice. The purpose of putting it in bin_SCRIPTS is to make sure that it is installed, which configure is not going to do automatically. The automake docs say that bin_SCRIPTS are by default not included in the dist, which is what we want here (since configure generates it). You should pretty much always do a set -e at the top of a shell script to catch errors early on. Oops, fixed. *73* if test @prefix@ != /usr -a @prefix != / -a @prefix != ; then Should those all be @pre...@? Yes. :/ Also, I think test -a might be a bashism in this case. Changed to test. Same for this line: *79* if test $full_library = true -o $explicit_library = false; then Done. Also, I added --ldflags as a separate option since LDFLAGS and LIBS are traditionally separate. Not sure why gtk-config itself does not do this. Also also, I expanded the help text. Also^3, I made --version strip the suffix (-pre, rc1, etc.) since I doubt anyone will correctly parse it otherwise (since people will code against official releases which have no suffix). --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Lite mode is in SVN -- mostly
On Wed, Jul 29, 2009 at 3:41 PM, ijuma ism...@juma.me.uk wrote: Is there a way to disable this behaviour? As you say, it doesn't seem particularly useful for sophisticated JVMs and unless I am missing something, the queue is currently unbounded so it could end up retaining quite a bit of memory for some usage patterns. There's no way to disable it at present, but I would be fine with adding a way. In fact I think this should probably be off by default. Best, Ismael --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Allow setting compression level on GzipOutputStream
Yeah, the test only compares default compression to no compression -- even with the small message, default compression manages to compress a little bit. Committed as rev 170. On Fri, Jul 31, 2009 at 5:08 PM, peter.k...@gmail.com wrote: On 2009/07/31 23:56:40, kenton wrote: This looks great, Kenton. Thanks for cleaning it up! The tests look pretty decent as well. With such a small golden message you're probably not going to see any difference between the different compression levels, and it is just passed through to zlib so there's probably no point in testing it here. http://codereview.appspot.com/96211 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: text relocation inside libprotoc.so.4.0.0
I think text relocation has something to do with the way a dynamic library is compiled and/or loaded. This is not something which Protocol Buffers asks for specifically -- either libtool, your linker, or your binary loader chose to use this feature without being asked for it. It seems that SELinux feels that text relocation is risky because it means that the library image in memory is both executable and writable, which presumably makes it easier to exploit buffer overruns -- but first a buffer overrun has to exist, independently. So this isn't a security hole in itself, it's just something that could hypothetically make some other security hole more exploitable. Possible solutions: * Compile protobuf with static linking only. (configure --disable-shared) * Figure out how to compile protobuf with text relocation disabled, and do that. (Maybe CFLAGS=-fPIC will do it? But note that this will make the code a bit slower.) * Tell your OS to allow text relocation for protoc because protoc does not process data from potentially malicious users and therefore is not a security risk. On Fri, Jul 31, 2009 at 6:43 PM, hw...@gear6.com hw...@gear6.com wrote: A question on the right behavior of libprotoc.so.4.0.0. On a CentOS 5.1, I downloaded and installed the latest protocol buffer 2.1. Make check runs fine. Then when I tried to do make under the ./ example directory, SELinux reports that protoc attempted to load /usr/ local/lib/libprotoc.so.4.0.0 which requires text relocation. This is a potential security problem. The command I run is simply: protoc --cpp_out=. addressbook.proto which reports from CLI that cannot restore segment proc after reloc: Permission Denied While I don't believe the code has malicious intention, just curious that is this text relocation inside libprotoc.so.4.0.0 a correct behavior? I hesitate to allow the this exception in SELinux before I get confirmation. Thanks, Hank --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Lite mode is in SVN -- mostly
On Sat, Aug 1, 2009 at 4:42 AM, Jon Skeet sk...@pobox.com wrote: Out of interest Kenton, does this make the bootstrapping code simpler? I'd imagine that can be built with just a lite version. It would be nice to get rid of some of the nastiness that's involved in C# just to get the PB-specific types to work :) In theory, descriptor.proto could become lite, and this might simplify bootstrapping. In practice, though, we actually use heavy features on the types in descriptor.proto, which is part of the reason why bootstrapping is so complicated. So, no, I don't think that would work. Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Field of Class type
Protocol message classes can only contain other protocol message classes, not arbitrary Java classes. Otherwise, what would happen when you compile the same .proto file in C++ or another language? On Mon, Aug 3, 2009 at 1:14 PM, Tai maitai.tru...@gmail.com wrote: Hi, I guess this is a simple question. I have a Java class with an attribute which is a class. How do I define that in a proto file? Maybe the full qualified class name? Thanks Tai --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Field of Class type
Ah. Sure, that sounds reasonable.
On Mon, Aug 3, 2009 at 2:15 PM, Tai maitai.tru...@gmail.com wrote:
Sorry but maybe my post is somehow confusing. I try to migrate my java
class into a protocal message class. Now I have a field like:
publiy MyClass {
Class anotherClass
}
I wonder how I should define that in a proto file (MyClassMessage).
Maybe like this:
message MyClass {
required string anotherClassName = 1;
}
Since I still have to use MyClass I just implement the writeObject()
and readObject() and use the the Class.forName():
private void writeObject(java.io.ObjectOutputStream out) throws
IOException {
Builder builder = MyClassMessage.MyClass.newBuilder();
builder.setAnotherClassName(anotherClassName.getName());
MyClassMessage.MyClass message = builder.build();
message.writeTo(out);
}
private void readObject(java.io.ObjectInputStream in)
throws ClassNotFoundException, IOException {
MyClassMessage.MyClass message =
MyClassMessage.MyClass.parseFrom(in);
anotherClassName= Class.forName(message.getAnotherClassName
());
}
On 3 Aug., 23:06, Kenton Varda ken...@google.com wrote:
Protocol message classes can only contain other protocol message classes,
not arbitrary Java classes. Otherwise, what would happen when you
compile
the same .proto file in C++ or another language?
On Mon, Aug 3, 2009 at 1:14 PM, Tai maitai.tru...@gmail.com wrote:
Hi,
I guess this is a simple question. I have a Java class with an
attribute which is a class. How do I define that in a proto file?
Maybe the full qualified class name?
Thanks Tai
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Subclass of a Proto Message
Your approach actually won't work (at least, as you've written the code)
because if you write two messages to a stream without any sort of delimiter,
it's impossible to figure out later where one message ends and the other
starts.
But I suggest doing this instead:
message A {
...
}
message B {
required A a = 1;
...
}
In other words, change inheritance to composition.
On Mon, Aug 3, 2009 at 2:00 PM, Tai maitai.tru...@gmail.com wrote:
Hi again,
I have two classes where B extends from A. Since both classes are
legacy and as far as I understood protocol buffers I write a proto
file for class A. For serialization and deserialization I just add the
methods writeObject() and readObject() to class A.
Now I wonder how to write the proto file for class B which is a
subclass of A? Is it like this that I define in the B proto file the
attributes defined in B and then in Class B's methods I do this:
private void writeObject(java.io.ObjectOutputStream out) throws
IOException {
Builder builder = A.AttributeA.newBuilder();
builder.setAttributeA(attributeA);
A.AttributeA message = builder.build();
message.writeTo(out);
Builder builder2 = B.AttributeB.newBuilder();
builder2.setAttributeB(attributeB);
B.AttributeB message2 = builder2.build();
message2.writeTo(out);
}
private void readObject(java.io.ObjectInputStream in)
throws ClassNotFoundException, IOException {
A.AttributeA message = A.AttributeA.parseFrom(in);
attributeA = message.getAttributeA();
B.AttributeB message2 = B.AttributeB.parseFrom(in);
attributeB = message2.getAttributeB();
}
Is this the standard approach?
Thanks Tai
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Changing {parse,merge}DelimitedFrom()
In Java I introduced methods called writeDelimitedTo(), parseDelimitedFrom(), and mergeDelimitedFrom() in version 2.1.0. These read and write messages which are prefixed with the message size, thus allowing multiple messages to be written to one stream without any external delimiter. As one or two people have pointed out, these aren't quite useful in their current form because the parse/merge methods do not give any way to detect EOF. So, I want to change the interface. I think the ideal interface would have mergeDelimitedFrom() returning a boolean which is false on EOF (currently it returns a Builder -- itself -- to allow chaining), and praseDelimitedFrom() should probably return NULL on EOF. The question is, should I just change these methods in-place or should I introduce new methods with the new behavior? If I change them in-place, there's a chance it will break someone. However, since the current methods aren't actually very useful without this change, I'm not sure if I need to worry about breaking anyone -- why would anyone be using a non-useful method anyway? Any opinions? Is anyone using these methods currently? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: 'Search this group' isn't very helpful
What were you searching for? Are you sure there should be results? On Tue, Aug 4, 2009 at 3:07 PM, yellen yel...@gmail.com wrote: I'm finding that the search function provided by the group is returning much less than I expect. Is anyone else having similar trouble? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: java class name different from proto file name
Yes, that's correct. If the Win32 API strips trailing whitespace from file names when opening them, that would explain why there was no error -- but that's very strange behavior IMO. On Tue, Aug 4, 2009 at 4:26 PM, Tai maitai.tru...@gmail.com wrote: This could be a windows problem where white spaces are accepted when doing a file lookup. I further assume when protoc takes the argument it uses the filename including the white space to generate the class name. Maybe that was the reason why the class was called MyClassProto and not MyClass. Tai On 4 Aug., 23:19, Kenton Varda ken...@google.com wrote: In that case I'm surprised it compiled -- I would have expected protoc to report file not found. Weird. On Tue, Aug 4, 2009 at 2:16 PM, Tai maitai.tru...@gmail.com wrote: Thanks. Your were pointing into the right direction. It wasn't the filename but the compile argument when using ant: project name=Create PB classes default=generate basedir=../../../.. target name=generate exec executable=D:/development/workspace/Platform/dev/jars/ protoc arg value=-I=Platform/dev/config/protocolbuffers / arg value=--java_out=Platform/src / arg value=Platform/dev/config/protocolbuffers/ ADMObjectHandleMessage.proto / arg value=Platform/dev/config/protocolbuffers/ ADMTimeMessage.proto / /exec /target /project The argument was: value=Platform/dev/config/protocolbuffers/ ADMObjectHandleMessage.proto and not: value=Platform/dev/config/protocolbuffers/ ADMObjectHandleMessage.proto Tai On 4 Aug., 22:12, Kenton Varda ken...@google.com wrote: It looks like your problem is that your file name actually has a space tacked on to the end of it. That is, the name is MyClass.proto - note the last space. You should get rid of that space. On Tue, Aug 4, 2009 at 9:54 AM, Tai maitai.tru...@gmail.com wrote: I got a strange behaviour when compiling a protofile (e.g. MyClass.proto). The generated java class is then called MyClassProto. All other proto files works fine. As a workaround I have added the following line to the proto file: option java_outer_classname = MyClass; Any idea what I am doing wrong? Thanks Tai --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: 'Search this group' isn't very helpful
Interesting. I'll file a bug against google groups. On Tue, Aug 4, 2009 at 3:25 PM, Adam yel...@gmail.com wrote: On Aug 4, 5:16 pm, Kenton Varda ken...@google.com wrote: What were you searching for? Are you sure there should be results? I originally searched for 'rpc' which I'm pretty sure in the past brought up over 3 pages of results. I've been archiving the emails locally from the group for over a year. To test the search I went back and found some keywords in subjects that I would think it should easily locate. For example: ECMAScript (2008-07-11) MacPorts (2008-07-13) Each time I get zero results. -Adam --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: list member containing sub classes
Can you just have the OParameter message contain optional fields of type string and int64, where only one of the two is ever set? Then don't have specific types corresponding to OStringParameter and OLongParameter. On Tue, Aug 4, 2009 at 5:07 PM, Tai maitai.tru...@gmail.com wrote: Hi, I have an (abstract) class: - OParameter with a member String key Now there are also two subclasses of OParameter: - OStringParameter with a member String value - OLongParameter with a member long value By using Protocol Buffers I define the proto files: - OParameterMessage.proto - OStringParameterMessage.proto - OLongParameterMessage.proto The files are independent from each other (no extension is used). As far as I understand this is how Protocol Buffers should be used. All classes (OParameter, OStringParameter and OLongParameter) then implements the writeObject() and readObject() and each class just uses their Protocol message and sets their members. Like: - Class OStringParameter.writeObject() builds a message and sets only its member (value) - The superclass OParameter.writeObject() builds a message and sets only its member (key) The same goes for the readObject(). Okay. So far so good. But now I have a class called OListParameter and its member (value) is a list of OParameter containing instances of both types: OStringParameter and OLongParameter. How do I describe this in a proto file? Thanks Tai --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: list member containing sub classes
Why do your protocol message types have to map 1:1 with your legacy classes?
Generally, attempting to map inheritance onto Protocol Buffers doesn't work
very well.
You are right that you can use extensions, but unless you have a large
number of subclasses of OParameter, this isn't helpful. If the only
subclasses of OParameter are OStringParameter, OLongParameter, and
OListParameter, then I highly recommend defining OParameter as:
message OParameter {
// Only one of the following will be filled in.
optional int64 long_value = 1;
optional string string_value = 2;
repeated OParameter list_value = 3;
}
If you really wanted to use extensions instead, you could do:
message OParameter {
// Only one extension will be filled in.
extensions 1 to max;
}
extend OParameter {
optional int64 oparameter_long_value = 1;
optional string oparameter_string_value = 2;
repeated OParameter oparameter_list_value = 3;
}
This is logically equivalent to the first definition. The only difference
is logistical: you can define each extension field in a different .proto
file, and your program only needs to compile in the definitions of
extensions that it actually uses. So you can do:
// oparameter.proto
message OParameter {
// Only one extension will be filled in.
extensions 1 to max;
}
// olongparameter.proto
import oparameter.proto;
extend OParameter {
optional int64 oparameter_long_value = 1;
}
// ostringparameter.proto
import oparameter.proto;
extend OParameter {
optional string oparameter_string_value = 2;
}
// olistparameter.proto
import oparameter.proto;
extend OParameter {
repeated OParameter oparameter_list_value = 3;
}
But again, if you only have these three subclasses, then this is overkill.
If you actually have a lot more (say, at least 10), then it might be
worthwhile.
Again, keep in mind that extensions are not inheritance -- they are a very
different concept which happens to be able to solve similar problems.
On Wed, Aug 5, 2009 at 1:31 AM, Tai maitai.tru...@gmail.com wrote:
Hi Kenton,
Sorry the parameter classes are legacy code and persisted to the DB as
well. So I have to figure out what's the best approach defining my
proto classes.
What I see is that there is the way of using extensions in the proto
files. As far as I understand it is an extension of members (an not
classes) in another proto file. Correct?
So in my case the files in OStringParameterMessage.proto and
OLongParameterMessage.proto extends OParameterMessage.proto with a
member value of type string and int64.
The OListParameterMessage.proto then extends it with a member value of
type List (containing OParameterMessage.OParameter instances).
In the OListParameter.writeObject() I build OParameter messages. But
wouldn't there be a problem since the value is either of type String
or Quantity (depending on the subclass of OParameter)? Probably I have
to define different member names.
Tai
On 5 Aug., 03:05, Kenton Varda ken...@google.com wrote:
Can you just have the OParameter message contain optional fields of type
string and int64, where only one of the two is ever set? Then don't have
specific types corresponding to OStringParameter and OLongParameter.
On Tue, Aug 4, 2009 at 5:07 PM, Tai maitai.tru...@gmail.com wrote:
Hi,
I have an (abstract) class:
- OParameter with a member String key
Now there are also two subclasses of OParameter:
- OStringParameter with a member String value
- OLongParameter with a member long value
By using Protocol Buffers I define the proto files:
- OParameterMessage.proto
- OStringParameterMessage.proto
- OLongParameterMessage.proto
The files are independent from each other (no extension is used). As
far as I understand this is how Protocol Buffers should be used. All
classes (OParameter, OStringParameter and OLongParameter) then
implements the writeObject() and readObject() and each class just uses
their Protocol message and sets their members. Like:
- Class OStringParameter.writeObject() builds a message and sets only
its member (value)
- The superclass OParameter.writeObject() builds a message and sets
only its member (key)
The same goes for the readObject().
Okay. So far so good. But now I have a class called OListParameter and
its member (value) is a list of OParameter containing instances of
both types: OStringParameter and OLongParameter.
How do I describe this in a proto file?
Thanks Tai
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: python custom options
Sorry, they are not implemented, but feel free to submit a patch. On Wed, Aug 5, 2009 at 6:13 AM, jacobgladish jacob.glad...@sig.com wrote: Does anyone know the current state of custom options with python? Are they functional? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Java: class visibility
Currently there's no way to do that. You could add an option to descriptor.proto, and update the Java code generator to honor it. Or you could just use a postprocessing step like: sed -i 's/^public class/class/' MyProto.java On Wed, Aug 5, 2009 at 6:41 AM, v6ak vit...@gmail.com wrote: How can I achieve package-private class visibility? Using of Protocol Buffers is just implementation detail only. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: TextFormat and optional fields.
Protobuf 2.1.0 throws NPE when you try to set a field to null. So, this has
been long fixed.
On Fri, Aug 7, 2009 at 4:30 AM, Brice Figureau
brice...@daysofwonder.combrice%2...@daysofwonder.com
wrote:
Hi,
Oops, sorry, I was plain wrong.
It turns out the code was setting the nonce parameter to null.
Anyway, maybe it should be necessary to change the TextFormat to better
deal with null fields?
On Fri, 2009-08-07 at 12:07 +0200, Brice Figureau wrote:
Hi,
I have the following message, under Protobuf 2.0.3:
message Message {
required string name = 1;
optional string nonce = 2;
}
When I call toString() on this message it throws a NullPointerException:
Exception in thread main java.lang.NullPointerException
at com.google.protobuf.ByteString.copyFromUtf8(Unknown Source)
at com.google.protobuf.TextFormat.escapeText(Unknown Source)
at com.google.protobuf.TextFormat.printFieldValue(Unknown Source)
at com.google.protobuf.TextFormat.printSingleField(Unknown Source)
at com.google.protobuf.TextFormat.printField(Unknown Source)
at com.google.protobuf.TextFormat.print(Unknown Source)
at com.google.protobuf.TextFormat.print(Unknown Source)
at com.google.protobuf.TextFormat.printToString(Unknown Source)
at com.google.protobuf.AbstractMessage.toString(Unknown Source)
Which means it is impossible to have a null default value in a message,
we have to always have a default value (which is sometimes not
possible).
Is that the expected behaviour?
--
Brice Figureau
My Blog: http://www.masterzen.fr/
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: 'Search this group' isn't very helpful
The groups people are working on fixing this. On Fri, Aug 7, 2009 at 12:51 PM, Scott Stafford scott.staff...@gmail.comwrote: I'm having zero results too, searching for DescriptorPool for instance. On Aug 4, 7:54 pm, Kenton Varda ken...@google.com wrote: Interesting. I'll file a bug against google groups. On Tue, Aug 4, 2009 at 3:25 PM, Adam yel...@gmail.com wrote: On Aug 4, 5:16 pm, Kenton Varda ken...@google.com wrote: What were you searching for? Are you sure there should be results? I originally searched for 'rpc' which I'm pretty sure in the past brought up over 3 pages of results. I've been archiving the emails locally from the group for over a year. To test the search I went back and found some keywords in subjects that I would think it should easily locate. For example: ECMAScript (2008-07-11) MacPorts (2008-07-13) Each time I get zero results. -Adam- Hide quoted text - - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
