valgrind issues
Hey guys, We're valgrinding drizzle at the moment and see a lot of: # ==3378== 40 bytes in 1 blocks are still reachable in loss record 14 of 121 # ==3378==at 0x4A06D5C: operator new(unsigned long) (vg_replace_malloc.c:230) # ==3378==by 0x5894A8: drizzled::message::protobuf_BuildDesc_table_2eproto_AssignGlobalDescriptors(google::protobuf::FileDescriptor const*) (table.pb.cc:173) # ==3378==by 0x4C64F13: google::protobuf::DescriptorBuilder::BuildFile(google::protobuf::FileDescriptorProto const&, void (*)(google::protobuf::FileDescriptor const*)) (descriptor.cc:2391) # ==3378==by 0x4C65BBF: google::protobuf::DescriptorPool::InternalBuildGeneratedFile(void const*, int, void (*)(google::protobuf::FileDescriptor const*)) (descriptor.cc:1962) # ==3378==by 0x5883B9: _GLOBAL__I_table.pb.cc (table.pb.cc:580) # ==3378==by 0x708675: (within /home/brian/merge/drizzled/drizzled) # ==3378==by 0x411BCA: (within /home/brian/merge/drizzled/drizzled) # ==3378==by 0x3171402B1F: (within /lib64/libpthread-2.9.so) # ==3378==by 0x708604: __libc_csu_init (in /home/brian/merge/drizzled/drizzled) # ==3378==by 0x3170C1E501: (below main) (libc-start.c:179) I'm not sure whether they are valid and something I should report to you, or whether they are invalid and something I should suppress. Any thoughts? Thanks! Monty --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Sun Studio 11
I am trying out what changes he suggested uptil now I am able to
compile but gtests are failing. I will update discussion once I am
able to find out whats going with gtests.
Vikram
On Jun 22, 4:34 pm, Kenton Varda wrote:
> I haven't heard from the patch author since my previous mail. :/
>
> On Mon, Jun 22, 2009 at 4:06 PM, vikram wrote:
>
> > Can somebody point out when this patch will be included in source?
> > I am trying to do compilation on Solaris 10 but right now I am getting
> > errors as mentioned in this post.
>
> > Vikram
>
> > On Jun 4, 1:42 pm, Kenton Varda wrote:
> > > Thanks for the patch!
> > > It looks like you were using the examples as a test. Running "make
> > check"
> > > in the top directory will run a much better suite of tests -- do they
> > pass?
>
> > > Assuming it does work, can you re-send that patch as an attachment (it
> > looks
> > > like it has been mangled), or even send it to me via
> > codereview.appspot.com?
> > > Also, for me to submit it you'll need to sign the contributor license
> > > agreement:
>
> > >http://code.google.com/legal/individual-cla-v1.0.html--if you own
> > > copyright on this patchhttp://
> > code.google.com/legal/corporate-cla-v1.0.html-- if your employer
> > > does
>
> > > On Wed, Jun 3, 2009 at 10:53 PM, wrote:
>
> > > > > Probably not. LibCstd isn't complete enough for protobuf.
>
> > > > I was able to get protobuf 2.1.0 to work well under Sun Studio 11,
> > > > using libCstd, with a few patches. Works well enough to compile and
> > > > run the add_person_cpp/list_people_cpp examples.
>
> > > > I include the build instructions and patch set below Also included
> > > > are patches to the examples makefile to test it out.
>
> > > >
> > > > $ CXX=CC CC=cc ./configure --disable-shared
> > > > $ make
> > > > $ cd examples
> > > > $ make cpp
> > > > $ ./add_person_cpp addressbook.test
> > > > (enter a test record)
> > > > $ ./list_people_cpp addressbook.test
> > > > (test record is displayed correctly)
> > > > $ ldd list_people_cpp (or ldd add_person_cpp )
> > > > libpthread.so.1 => /lib/libpthread.so.1
> > > > libCstd.so.1 => /usr/lib/libCstd.so.1
> > > > libCrun.so.1 => /usr/lib/libCrun.so.1
> > > > libm.so.2 => /lib/libm.so.2
> > > > libc.so.1 => /lib/libc.so.1
>
> > > > Index: src/google/protobuf/repeated_field.h
> > > > ===
> > > > --- src/google/protobuf/repeated_field.h (revision 2)
> > > > +++ src/google/protobuf/repeated_field.h (working copy)
> > > > @@ -69,7 +69,7 @@
> > > > class LIBPROTOBUF_EXPORT GenericRepeatedField {
> > > > public:
> > > > inline GenericRepeatedField() {}
> > > > -#if defined(__DECCXX) && defined(__osf__)
> > > > +#if defined(__SUNPRO_CC) || defined(__DECCXX) && defined(__osf__)
> > > > // HP C++ on Tru64 has trouble when this is not defined inline.
> > > > virtual ~GenericRepeatedField() {}
> > > > #else
> > > > @@ -548,7 +548,7 @@
> > > > current_size_ = 0;
> > > > }
>
> > > > -#if defined(__DECCXX) && defined(__osf__)
> > > > +#if defined(__SUNPRO_CC) || defined(__DECCXX) && defined(__osf__)
> > > > // HP C++ on Tru64 has trouble when this is not defined inline.
> > > > template <>
> > > > inline void RepeatedPtrField::Clear() {
> > > > Index: src/google/protobuf/descriptor_database.cc
> > > > ===
> > > > --- src/google/protobuf/descriptor_database.cc (revision 2)
> > > > +++ src/google/protobuf/descriptor_database.cc (working copy)
> > > > @@ -127,7 +127,13 @@
>
> > > > // Insert the new symbol using the iterator as a hint, the new
> > > > entry will
> > > > // appear immediately before the one the iterator is pointing at.
> > > > +
> > > > + // Sun Studio 11 needs a little help here
> > > > +#if defined(__SUNPRO_CC)
> > > > + by_symbol_.insert(iter, make_pair(name,
> > > > value));
> > > > +#else
> > > > by_symbol_.insert(iter, make_pair(name, value));
> > > > +#endif
>
> > > > return true;
> > > > }
> > > > Index: src/google/protobuf/compiler/command_line_interface.cc
> > > > ===
> > > > --- src/google/protobuf/compiler/command_line_interface.cc
> > (revision
> > > > 2)
> > > > +++ src/google/protobuf/compiler/command_line_interface.cc
> > (working
> > > > copy)
> > > > @@ -474,7 +474,7 @@
>
> > > > // If no --proto_path was given, use the current working directory.
> > > > if (proto_path_.empty()) {
> > > > - proto_path_.push_back(make_pair("", "."));
> > > > + proto_path_.push_back(make_pair(string(""), string(".")));
> > > > }
>
> > > > // Check some errror cases.
> > > > Index: src/google/protobuf/repeated_field.cc
> > > > ===
> > > > --- src/google/protobuf/repeated_field.cc (revision 2)
> > > > +++ src/google/protobuf/repeated_f
Re: Sun Studio 11
I haven't heard from the patch author since my previous mail. :/
On Mon, Jun 22, 2009 at 4:06 PM, vikram wrote:
>
> Can somebody point out when this patch will be included in source?
> I am trying to do compilation on Solaris 10 but right now I am getting
> errors as mentioned in this post.
>
> Vikram
>
>
> On Jun 4, 1:42 pm, Kenton Varda wrote:
> > Thanks for the patch!
> > It looks like you were using the examples as a test. Running "make
> check"
> > in the top directory will run a much better suite of tests -- do they
> pass?
> >
> > Assuming it does work, can you re-send that patch as an attachment (it
> looks
> > like it has been mangled), or even send it to me via
> codereview.appspot.com?
> > Also, for me to submit it you'll need to sign the contributor license
> > agreement:
> >
> > http://code.google.com/legal/individual-cla-v1.0.html-- if you own
> > copyright on this patchhttp://
> code.google.com/legal/corporate-cla-v1.0.html-- if your employer
> > does
> >
> > On Wed, Jun 3, 2009 at 10:53 PM, wrote:
> >
> > > > Probably not. LibCstd isn't complete enough for protobuf.
> >
> > > I was able to get protobuf 2.1.0 to work well under Sun Studio 11,
> > > using libCstd, with a few patches. Works well enough to compile and
> > > run the add_person_cpp/list_people_cpp examples.
> >
> > > I include the build instructions and patch set below Also included
> > > are patches to the examples makefile to test it out.
> >
> > >
> > > $ CXX=CC CC=cc ./configure --disable-shared
> > > $ make
> > > $ cd examples
> > > $ make cpp
> > > $ ./add_person_cpp addressbook.test
> > > (enter a test record)
> > > $ ./list_people_cpp addressbook.test
> > > (test record is displayed correctly)
> > > $ ldd list_people_cpp (or ldd add_person_cpp )
> > >libpthread.so.1 => /lib/libpthread.so.1
> > >libCstd.so.1 => /usr/lib/libCstd.so.1
> > >libCrun.so.1 => /usr/lib/libCrun.so.1
> > >libm.so.2 => /lib/libm.so.2
> > >libc.so.1 => /lib/libc.so.1
> >
> > > Index: src/google/protobuf/repeated_field.h
> > > ===
> > > --- src/google/protobuf/repeated_field.h(revision 2)
> > > +++ src/google/protobuf/repeated_field.h(working copy)
> > > @@ -69,7 +69,7 @@
> > > class LIBPROTOBUF_EXPORT GenericRepeatedField {
> > > public:
> > > inline GenericRepeatedField() {}
> > > -#if defined(__DECCXX) && defined(__osf__)
> > > +#if defined(__SUNPRO_CC) || defined(__DECCXX) && defined(__osf__)
> > > // HP C++ on Tru64 has trouble when this is not defined inline.
> > > virtual ~GenericRepeatedField() {}
> > > #else
> > > @@ -548,7 +548,7 @@
> > > current_size_ = 0;
> > > }
> >
> > > -#if defined(__DECCXX) && defined(__osf__)
> > > +#if defined(__SUNPRO_CC) || defined(__DECCXX) && defined(__osf__)
> > > // HP C++ on Tru64 has trouble when this is not defined inline.
> > > template <>
> > > inline void RepeatedPtrField::Clear() {
> > > Index: src/google/protobuf/descriptor_database.cc
> > > ===
> > > --- src/google/protobuf/descriptor_database.cc (revision 2)
> > > +++ src/google/protobuf/descriptor_database.cc (working copy)
> > > @@ -127,7 +127,13 @@
> >
> > > // Insert the new symbol using the iterator as a hint, the new
> > > entry will
> > > // appear immediately before the one the iterator is pointing at.
> > > +
> > > + // Sun Studio 11 needs a little help here
> > > +#if defined(__SUNPRO_CC)
> > > + by_symbol_.insert(iter, make_pair(name,
> > > value));
> > > +#else
> > > by_symbol_.insert(iter, make_pair(name, value));
> > > +#endif
> >
> > > return true;
> > > }
> > > Index: src/google/protobuf/compiler/command_line_interface.cc
> > > ===
> > > --- src/google/protobuf/compiler/command_line_interface.cc
> (revision
> > > 2)
> > > +++ src/google/protobuf/compiler/command_line_interface.cc
> (working
> > > copy)
> > > @@ -474,7 +474,7 @@
> >
> > > // If no --proto_path was given, use the current working directory.
> > > if (proto_path_.empty()) {
> > > -proto_path_.push_back(make_pair("", "."));
> > > +proto_path_.push_back(make_pair(string(""), string(".")));
> > > }
> >
> > > // Check some errror cases.
> > > Index: src/google/protobuf/repeated_field.cc
> > > ===
> > > --- src/google/protobuf/repeated_field.cc (revision 2)
> > > +++ src/google/protobuf/repeated_field.cc (working copy)
> > > @@ -40,7 +40,7 @@
> > > // HP C++ on Tru64 can't handle the stuff below being defined out-of-
> > > line, so
> > > // on that platform everything is defined in repeated_field.h. On
> > > other
> > > // platforms, we want these to be out-of-line to avoid code bloat.
> > > -#if !defined(__DECCXX) || !defined(__osf__)
> > > +#if !defined(__SUNPRO_CC) && (!defin
Re: Sun Studio 11
Can somebody point out when this patch will be included in source?
I am trying to do compilation on Solaris 10 but right now I am getting
errors as mentioned in this post.
Vikram
On Jun 4, 1:42 pm, Kenton Varda wrote:
> Thanks for the patch!
> It looks like you were using the examples as a test. Running "make check"
> in the top directory will run a much better suite of tests -- do they pass?
>
> Assuming it does work, can you re-send that patch as an attachment (it looks
> like it has been mangled), or even send it to me via codereview.appspot.com?
> Also, for me to submit it you'll need to sign the contributor license
> agreement:
>
> http://code.google.com/legal/individual-cla-v1.0.html-- if you own
> copyright on this patchhttp://code.google.com/legal/corporate-cla-v1.0.html--
> if your employer
> does
>
> On Wed, Jun 3, 2009 at 10:53 PM, wrote:
>
> > > Probably not. LibCstd isn't complete enough for protobuf.
>
> > I was able to get protobuf 2.1.0 to work well under Sun Studio 11,
> > using libCstd, with a few patches. Works well enough to compile and
> > run the add_person_cpp/list_people_cpp examples.
>
> > I include the build instructions and patch set below Also included
> > are patches to the examples makefile to test it out.
>
> >
> > $ CXX=CC CC=cc ./configure --disable-shared
> > $ make
> > $ cd examples
> > $ make cpp
> > $ ./add_person_cpp addressbook.test
> > (enter a test record)
> > $ ./list_people_cpp addressbook.test
> > (test record is displayed correctly)
> > $ ldd list_people_cpp (or ldd add_person_cpp )
> > libpthread.so.1 => /lib/libpthread.so.1
> > libCstd.so.1 => /usr/lib/libCstd.so.1
> > libCrun.so.1 => /usr/lib/libCrun.so.1
> > libm.so.2 => /lib/libm.so.2
> > libc.so.1 => /lib/libc.so.1
>
> > Index: src/google/protobuf/repeated_field.h
> > ===
> > --- src/google/protobuf/repeated_field.h (revision 2)
> > +++ src/google/protobuf/repeated_field.h (working copy)
> > @@ -69,7 +69,7 @@
> > class LIBPROTOBUF_EXPORT GenericRepeatedField {
> > public:
> > inline GenericRepeatedField() {}
> > -#if defined(__DECCXX) && defined(__osf__)
> > +#if defined(__SUNPRO_CC) || defined(__DECCXX) && defined(__osf__)
> > // HP C++ on Tru64 has trouble when this is not defined inline.
> > virtual ~GenericRepeatedField() {}
> > #else
> > @@ -548,7 +548,7 @@
> > current_size_ = 0;
> > }
>
> > -#if defined(__DECCXX) && defined(__osf__)
> > +#if defined(__SUNPRO_CC) || defined(__DECCXX) && defined(__osf__)
> > // HP C++ on Tru64 has trouble when this is not defined inline.
> > template <>
> > inline void RepeatedPtrField::Clear() {
> > Index: src/google/protobuf/descriptor_database.cc
> > ===
> > --- src/google/protobuf/descriptor_database.cc (revision 2)
> > +++ src/google/protobuf/descriptor_database.cc (working copy)
> > @@ -127,7 +127,13 @@
>
> > // Insert the new symbol using the iterator as a hint, the new
> > entry will
> > // appear immediately before the one the iterator is pointing at.
> > +
> > + // Sun Studio 11 needs a little help here
> > +#if defined(__SUNPRO_CC)
> > + by_symbol_.insert(iter, make_pair(name,
> > value));
> > +#else
> > by_symbol_.insert(iter, make_pair(name, value));
> > +#endif
>
> > return true;
> > }
> > Index: src/google/protobuf/compiler/command_line_interface.cc
> > ===
> > --- src/google/protobuf/compiler/command_line_interface.cc (revision
> > 2)
> > +++ src/google/protobuf/compiler/command_line_interface.cc (working
> > copy)
> > @@ -474,7 +474,7 @@
>
> > // If no --proto_path was given, use the current working directory.
> > if (proto_path_.empty()) {
> > - proto_path_.push_back(make_pair("", "."));
> > + proto_path_.push_back(make_pair(string(""), string(".")));
> > }
>
> > // Check some errror cases.
> > Index: src/google/protobuf/repeated_field.cc
> > ===
> > --- src/google/protobuf/repeated_field.cc (revision 2)
> > +++ src/google/protobuf/repeated_field.cc (working copy)
> > @@ -40,7 +40,7 @@
> > // HP C++ on Tru64 can't handle the stuff below being defined out-of-
> > line, so
> > // on that platform everything is defined in repeated_field.h. On
> > other
> > // platforms, we want these to be out-of-line to avoid code bloat.
> > -#if !defined(__DECCXX) || !defined(__osf__)
> > +#if !defined(__SUNPRO_CC) && (!defined(__DECCXX) || !defined
> > (__osf__))
>
> > namespace internal {
>
> > --- Below changes the examples makefile to exercise protobuf under
> > solaris
>
> > Index: examples/Makefile
> > ===
> > --- examples/Makefile (revision 2)
> > +++ examples/Makefile (working copy)
> > @@ -
Re: Making searches inside a message file
Sorry, Protocol Buffers do not provide support for searching or indexing. All protobuf does is parse and serialize data structures. Drizzle is actually building database functionality *on top* of protocol buffers -- they implement their own searching and indexing, using protocol buffers to implement the base serialization. On Fri, Jun 19, 2009 at 11:25 PM, green wrote: > > I am interested in using protocol buffers for the following scenario: > > I have a large number of small data points I need to serialize to a > file. Once this serialization process is complete, the data will be > strictly read only, accessed by multiple threads, until I need to > create a new version of the file. The application will access the file > at startup, and while executing will need to retrieve data points > based on a given parameter. Pretty much behaves like tabular data you > would find in a database table. I need to retrieve a "row", based on a > "primary key". Yet a traditional database is not a good solution in > this situation. > > My understanding is that it's not a good idea to have large messages, > but you can have a large number of small messages within a file, and > it shouldn't be a problem. I have seen the .proto file structure and > how messages are determined. What I don't understand (maybe I missed > that part in the documentation) is how to make searches for messages > within a file. If I use the "repeated" qualifier, it will let me have > more than one message in another one, but I'll just retrieve it as a > list. I can't specify what subset of messages I want. > > Can I sort messages based on a given field? > Can I request a subset of messages by index range, or some other > criteria? > Can I "browse" through a message file, given a particular search > parameter? > Can I have some sort of Map inside the .proto definition, where I > organize elements in key - value fashion? > > Alternatively, are my assumptions of what I should be able to do with > protocol buffers wrong from the start? I assumed this kind of thing > was possible since the drizzle devs are using protocol buffers for > their database implementation. Link below. > > http://drizzle.org/wiki/Table_Proto_Definition > > Thanks in advance. > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Making searches inside a message file
Hi Green, AFAICT, protocol buffers does not have inbuilt support for searching and querying, at least not in the public domain release. However, you should be able to use protocol buffers in your particular scenario. Since your data is more or less read-only once it's serialized, you should be able to serialize the wire format data such that your 'Row' messages are sorted on your 'primary key'; just keep the repeated rows field sorted on that key before serializing. Additionally, you should also be able to build indices on any other parameter to speed up serach/query on those parameters. The Drizzle link that you have given is interesting. I had quick look at it. If you have noticed, they are building their own indices and are planning to use PB to store those indices also. PS: I am a newbie to PB, so take what I said with a pinch of salt. green wrote: > I am interested in using protocol buffers for the following scenario: > > I have a large number of small data points I need to serialize to a > file. Once this serialization process is complete, the data will be > strictly read only, accessed by multiple threads, until I need to > create a new version of the file. The application will access the file > at startup, and while executing will need to retrieve data points > based on a given parameter. Pretty much behaves like tabular data you > would find in a database table. I need to retrieve a "row", based on a > "primary key". Yet a traditional database is not a good solution in > this situation. > > My understanding is that it's not a good idea to have large messages, > but you can have a large number of small messages within a file, and > it shouldn't be a problem. I have seen the .proto file structure and > how messages are determined. What I don't understand (maybe I missed > that part in the documentation) is how to make searches for messages > within a file. If I use the "repeated" qualifier, it will let me have > more than one message in another one, but I'll just retrieve it as a > list. I can't specify what subset of messages I want. > > Can I sort messages based on a given field? > Can I request a subset of messages by index range, or some other > criteria? > Can I "browse" through a message file, given a particular search > parameter? > Can I have some sort of Map inside the .proto definition, where I > organize elements in key - value fashion? > > Alternatively, are my assumptions of what I should be able to do with > protocol buffers wrong from the start? I assumed this kind of thing > was possible since the drizzle devs are using protocol buffers for > their database implementation. Link below. > > http://drizzle.org/wiki/Table_Proto_Definition > > Thanks in advance. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
