date:20170707

Re: [protobuf] Compiling protobuf with GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro

2017-07-07 Thread Brad Larson



On Friday, July 7, 2017 at 3:35:38 PM UTC-5, Feng Xiao wrote:
>
>
>
> On Fri, Jul 7, 2017 at 1:28 PM, Brad Larson  > wrote:
>
>>
>>
>> On Tuesday, July 19, 2016 at 10:15:29 AM UTC-5, Igor Gatis wrote:
>>>
>>> I'm using -DGOOGLE_PROTOBUF_NO_THREAD_SAFETY but I'm hitting the 
>>> following errors:
>>>
>>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 43: Error:  #20: 
>>> identifier "AtomicWord" is undefined
>>> AtomicWord GetNext() {
>>> ^
>>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 47: Error:  #20: 
>>> identifier "AtomicWord" is undefined
>>> AtomicWord word_;
>>> ^
>>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 44: Error:  #20: 
>>> identifier "NoBarrier_AtomicIncrement" is undefined
>>>   return NoBarrier_AtomicIncrement(&word_, 1) - 1;
>>>
>>>
>>> Any chance there is a GOOGLE_PROTOBUF_NO_THREAD_SAFETY check missing?
>>>
>>
>> I see these errors as well.
>>
>>  
>>
>>>
>>>
>>> On Monday, October 21, 2013 at 2:16:20 PM UTC-3, Safi Ali wrote:

 Hi,

 Thanks a lot Feng for your quick answer. Well at the moment, we are 
 only planning to use it in a single threaded application where we 
 write/read messages sequentially, instead of in parallel. So I guess we 
 are 
 safe for now.

 Regards,
 Safi

 On Monday, October 21, 2013 7:32:19 PM UTC+3, Feng Xiao wrote:
>
>
>
>
> On Mon, Oct 21, 2013 at 1:05 AM, Safi Ali  wrote:
>
>> Hi,
>>
>> I have been trying to compile google protocol buffers 2.5.0 on 
>> solaris (sparc) environment. It seems I have to use 
>> the GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro in order to make it compile 
>> properly. So I follow these steps to compile protobuf:-
>>
>> ./configure CPPFLAGS="-DGOOGLE_PROTOBUF_NO_THREAD_SAFETY"
>> make 
>> make check
>>
>> In 'make check', all tests pass. 
>> Can anyone shed some light on what are the caveats of using the 
>> no_thread_safety macro? What, if any, problems can I expect from 
>> protobufs 
>> with no thread safety. I have some apprehensions about it and it would 
>> be 
>> great if someone could clarify those for me:-
>>
>> 1. Is the thread safety only an issue during compilation of .proto 
>> files to java/c++ source files? or does protobuf also rely on thread 
>> safety 
>> during execution of compiled code?
>>
> Protobuf uses mutex/locks at runtime to protect certain data 
> structures in multi-threading environment.
>  
>
>> 2. If I dont use thread safety, does protobuf gracefully fall back to 
>> single threaded model where needed, or still try to use threads but in 
>> somewhat "unsafe" fashion which can lead to bugs such as deadlocks if im 
>> unlucky?
>>
> Protobuf doesn't create threads, but with no_thread_safety macro, all 
> mutex/locks will be turned into nop. That means you can only use protobuf 
> in a single threaded binary. If you try to use messages in multiple 
> threads, the code may break unexpectedly.
>

>> Can anyone confirm, is it acceptable to read/write different messages in 
>> different threads?  Or can we only have one thread make any protobuf calls 
>> at all?
>>
> If you are using GOOGLE_PROTOBUF_NO_THREAD_SAFETY, you cannot use 
> protobuf in multiple threads. Even using different message types in 
> different threads will be problematic because all message types share the 
> same global DescriptorPool/MessageFactory/etc. 
>
> If you are not using GOOGLE_PROTOBUF_NO_THREAD_SAFETY, protobuf supports 
> the same thread-safety semantics as with standard types. Reading/writing 
> different messages in different threads are fine. Reading the same message 
> in multiple threads is also fine. To write a message in multiple threads a 
> lock must be used.
>

Thank you Feng, this is very helpful.

I started trying to get things to compile with 
GOOGLE_PROTOBUF_NO_THREAD_SAFETY, but ran into some difficulties with the 
deterministic serialization feature that require atomics that are a little 
over my head, so for now I'll go back to supporting threads.  Hopefully 
someone with more knowledge on this feature can come along and fix the no 
thread support, as it is very desirable on embedded / non-pthread platforms 
like mine.
 

>  
>
>>  
>>
>>>  
>
>> 3. How is the performance affected while using thread unsafe code? if 
>> anyone has done some benchmarking, would be good to see the results.
>>
>> Regards,
>> Safi
>>
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, 
>> send an email to protobuf+u...@googlegroups.com.
>> To post to this group, send email to prot...@googlegroups.com.
>> Visit this group at http://groups.google.com/group/protobuf.
>

Re: [protobuf] Compiling protobuf with GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro

2017-07-07 Thread 'Feng Xiao' via Protocol Buffers

On Fri, Jul 7, 2017 at 1:28 PM, Brad Larson  wrote:

>
>
> On Tuesday, July 19, 2016 at 10:15:29 AM UTC-5, Igor Gatis wrote:
>>
>> I'm using -DGOOGLE_PROTOBUF_NO_THREAD_SAFETY but I'm hitting the
>> following errors:
>>
>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 43: Error:  #20:
>> identifier "AtomicWord" is undefined
>> AtomicWord GetNext() {
>> ^
>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 47: Error:  #20:
>> identifier "AtomicWord" is undefined
>> AtomicWord word_;
>> ^
>> "src/google/protobuf/stubs/atomic_sequence_num.h", line 44: Error:  #20:
>> identifier "NoBarrier_AtomicIncrement" is undefined
>>   return NoBarrier_AtomicIncrement(&word_, 1) - 1;
>>
>>
>> Any chance there is a GOOGLE_PROTOBUF_NO_THREAD_SAFETY check missing?
>>
>
> I see these errors as well.
>
>
>
>>
>>
>> On Monday, October 21, 2013 at 2:16:20 PM UTC-3, Safi Ali wrote:
>>>
>>> Hi,
>>>
>>> Thanks a lot Feng for your quick answer. Well at the moment, we are only
>>> planning to use it in a single threaded application where we write/read
>>> messages sequentially, instead of in parallel. So I guess we are safe for
>>> now.
>>>
>>> Regards,
>>> Safi
>>>
>>> On Monday, October 21, 2013 7:32:19 PM UTC+3, Feng Xiao wrote:




 On Mon, Oct 21, 2013 at 1:05 AM, Safi Ali  wrote:

> Hi,
>
> I have been trying to compile google protocol buffers 2.5.0 on solaris
> (sparc) environment. It seems I have to use the 
> GOOGLE_PROTOBUF_NO_THREAD_SAFETY
> macro in order to make it compile properly. So I follow these steps to
> compile protobuf:-
>
> ./configure CPPFLAGS="-DGOOGLE_PROTOBUF_NO_THREAD_SAFETY"
> make
> make check
>
> In 'make check', all tests pass.
> Can anyone shed some light on what are the caveats of using the
> no_thread_safety macro? What, if any, problems can I expect from protobufs
> with no thread safety. I have some apprehensions about it and it would be
> great if someone could clarify those for me:-
>
> 1. Is the thread safety only an issue during compilation of .proto
> files to java/c++ source files? or does protobuf also rely on thread 
> safety
> during execution of compiled code?
>
 Protobuf uses mutex/locks at runtime to protect certain data structures
 in multi-threading environment.


> 2. If I dont use thread safety, does protobuf gracefully fall back to
> single threaded model where needed, or still try to use threads but in
> somewhat "unsafe" fashion which can lead to bugs such as deadlocks if im
> unlucky?
>
 Protobuf doesn't create threads, but with no_thread_safety macro, all
 mutex/locks will be turned into nop. That means you can only use protobuf
 in a single threaded binary. If you try to use messages in multiple
 threads, the code may break unexpectedly.

>>>
> Can anyone confirm, is it acceptable to read/write different messages in
> different threads?  Or can we only have one thread make any protobuf calls
> at all?
>
If you are using GOOGLE_PROTOBUF_NO_THREAD_SAFETY, you cannot use protobuf
in multiple threads. Even using different message types in different
threads will be problematic because all message types share the same global
DescriptorPool/MessageFactory/etc.

If you are not using GOOGLE_PROTOBUF_NO_THREAD_SAFETY, protobuf supports
the same thread-safety semantics as with standard types. Reading/writing
different messages in different threads are fine. Reading the same message
in multiple threads is also fine. To write a message in multiple threads a
lock must be used.


>
>
>>

> 3. How is the performance affected while using thread unsafe code? if
> anyone has done some benchmarking, would be good to see the results.
>
> Regards,
> Safi
>
> --
> You received this message because you are subscribed to the Google
> Groups "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to protobuf+u...@googlegroups.com.
> To post to this group, send email to prot...@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/groups/opt_out.
>

 --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To

Re: [protobuf] Compiling protobuf with GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro

2017-07-07 Thread Brad Larson



On Tuesday, July 19, 2016 at 10:15:29 AM UTC-5, Igor Gatis wrote:
>
> I'm using -DGOOGLE_PROTOBUF_NO_THREAD_SAFETY but I'm hitting the following 
> errors:
>
> "src/google/protobuf/stubs/atomic_sequence_num.h", line 43: Error:  #20: 
> identifier "AtomicWord" is undefined
> AtomicWord GetNext() {
> ^
> "src/google/protobuf/stubs/atomic_sequence_num.h", line 47: Error:  #20: 
> identifier "AtomicWord" is undefined
> AtomicWord word_;
> ^
> "src/google/protobuf/stubs/atomic_sequence_num.h", line 44: Error:  #20: 
> identifier "NoBarrier_AtomicIncrement" is undefined
>   return NoBarrier_AtomicIncrement(&word_, 1) - 1;
>
>
> Any chance there is a GOOGLE_PROTOBUF_NO_THREAD_SAFETY check missing?
>

I see these errors as well.

 

>
>
> On Monday, October 21, 2013 at 2:16:20 PM UTC-3, Safi Ali wrote:
>>
>> Hi,
>>
>> Thanks a lot Feng for your quick answer. Well at the moment, we are only 
>> planning to use it in a single threaded application where we write/read 
>> messages sequentially, instead of in parallel. So I guess we are safe for 
>> now.
>>
>> Regards,
>> Safi
>>
>> On Monday, October 21, 2013 7:32:19 PM UTC+3, Feng Xiao wrote:
>>>
>>>
>>>
>>>
>>> On Mon, Oct 21, 2013 at 1:05 AM, Safi Ali  wrote:
>>>
 Hi,

 I have been trying to compile google protocol buffers 2.5.0 on solaris 
 (sparc) environment. It seems I have to use 
 the GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro in order to make it compile 
 properly. So I follow these steps to compile protobuf:-

 ./configure CPPFLAGS="-DGOOGLE_PROTOBUF_NO_THREAD_SAFETY"
 make 
 make check

 In 'make check', all tests pass. 
 Can anyone shed some light on what are the caveats of using the 
 no_thread_safety macro? What, if any, problems can I expect from protobufs 
 with no thread safety. I have some apprehensions about it and it would be 
 great if someone could clarify those for me:-

 1. Is the thread safety only an issue during compilation of .proto 
 files to java/c++ source files? or does protobuf also rely on thread 
 safety 
 during execution of compiled code?

>>> Protobuf uses mutex/locks at runtime to protect certain data structures 
>>> in multi-threading environment.
>>>  
>>>
 2. If I dont use thread safety, does protobuf gracefully fall back to 
 single threaded model where needed, or still try to use threads but in 
 somewhat "unsafe" fashion which can lead to bugs such as deadlocks if im 
 unlucky?

>>> Protobuf doesn't create threads, but with no_thread_safety macro, all 
>>> mutex/locks will be turned into nop. That means you can only use protobuf 
>>> in a single threaded binary. If you try to use messages in multiple 
>>> threads, the code may break unexpectedly.
>>>
>>
Can anyone confirm, is it acceptable to read/write different messages in 
different threads?  Or can we only have one thread make any protobuf calls 
at all?
 

>  
>>>
 3. How is the performance affected while using thread unsafe code? if 
 anyone has done some benchmarking, would be good to see the results.

 Regards,
 Safi

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Protocol Buffers" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/groups/opt_out.

>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] use of banned API functions on the ProtoBuf library

2017-07-07 Thread Michael Muriuki

Hi,

Am new to the ProtoBuf library and only use it as part of the Google's 
libraries. Recently our security team indicated that the library in iOS 
uses some of the banned 
 API functions 
listed h ere. Does 
anyone know why these have not been replaced with the safer alternatives 
and what measures are in place to ensure that the code is not susceptible 
to buffer overflow injection?

The functions *strlen, memcpy* and *memmove* are used in the following 
Protobuf code.

GPBCodedOutputStream.h
GPBCodedOutputStream.h
GPBDescriptor.h
GPBDescriptor.m
GPBMessage.h
GPBMessage.m
GPBRootObject.h
GPBRootObject.h

-- 


*Cellulant Group email disclaimer and confidentiality note*

Please go here 

 to 
read our email disclaimer and confidentiality note. 

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Re: Json object hash

2017-07-07 Thread alex

Thank you Arpit, your answer clarifies the issue for me!

I did go through the docs on struct and am giving it a go.

It would help tremendously if protobuf would have a clear guide on how 
these data types translate into different languages...

On Wednesday, July 5, 2017 at 7:50:34 PM UTC+2, Arpit Baldeva wrote:
>
> A list as a 'value type' in not supported for map. I have not worked with 
> JavaScript and protobuf but I believe in the other languages, it will be a 
> code generation error.
>
> In general, I have had the similar problems you have when you need to come 
> up with a particular json format which protobuf does not support. For 
> example, you can not have a list of list. The 'workaround' in this scenario 
> is to use
>
>
> https://github.com/google/protobuf/blob/master/src/google/protobuf/struct.proto
>  
>
> The library has special support the above proto file. I mention special 
> support because even if you make a verbatim copy of the above proto message 
> and use it, your json format would end up being different. 
>
> For your use case, you'd want to set the oneof in the Value protobuf to 
> the list. 
>
>
> On Wednesday, July 5, 2017 at 8:06:23 AM UTC-7, al...@nvent.nl wrote:
>>
>> Hi everyone,
>>
>> I'm trying to produce a specific response format and I'm struggling with 
>> the protobuf syntax to generate it. Worth mentioning, I'm new to protobuf 
>> and am struggling a bit with the docs, so it's likely that I missed 
>> something.
>>
>> The format I'm trying to produce, in json:
>>
>> {
>>   success: false
>>   errors: {
>> field1: ["error message 1", "error message 2"]
>> field2: []
>> ...fields
>>   }
>> }
>>
>> The proto syntax that I came up with looks like this:
>> message ResponseSubmit {
>>   bool success = 1
>>   map errors = 2
>> }
>>
>> message ErrorList {
>>   repeated string error = 1
>> }
>>
>> Then, using the generated javascript libraries:
>>
>> const response = new ResponseSubmit();
>> const errorsMap = response.getErrorsMap();
>> const valid = true;
>> const errors ={ field1: [], field2: ['Dummy error message'] };
>>
>> response.setSuccess(valid); // this works ok
>>
>> forEachObjIndexed((fieldErrors, fieldName) => {
>>   errorsMap.set(fieldName, fieldErrors); // this blows up 
>> });
>>
>> Adding a field to the map via the set method crashes with a TypeError: 
>> b.toArray is not a function.
>>
>> Can anyone lend a hand and point me to what I'm doing wrong here?
>>
>> Thank you!
>>
>> Alex
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Compiling protobuf with GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro

Re: [protobuf] Compiling protobuf with GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro

Re: [protobuf] Compiling protobuf with GOOGLE_PROTOBUF_NO_THREAD_SAFETY macro

[protobuf] use of banned API functions on the ProtoBuf library

[protobuf] Re: Json object hash

5 matches

Site Navigation

Mail list logo

Footer information