Our StringPiece type has been made obsolete by the std::string_view type introduced in C++17, so we will eventually get rid of StringPiece and replace it with std::string_view (or absl::string_view, which has the same API but is available in C++11). So if you are making a local modification to support allocating strings on arenas, it would probably be best to go straight to std::string_view and avoid our StringPiece type. To handle both the arena and non-arena case, the simplest solution would be to store both a std::string_view and a std::string. When arenas are used, you can have the string_view point into arena-allocated memory, and when arenas are not used, it can just point to the std::string's data.
On Thu, Sep 3, 2020 at 3:19 AM X Ah <xiaahui1...@gmail.com> wrote: > Hi Feng, > I have an API design problem, Since StringPiece doesn't own the data and > it's impossible to get memory from it, So how should the behavior be if I > do ParseFromString and the message is not in arena? I think the StringPiece > field should be empty if current message doesn't own an arena, but it is > strange for user. Could you introduce how Google internal use StringPiece > in protobuf? > Thanks! > On Saturday, January 16, 2016 at 3:32:39 AM UTC+8 xiao...@google.com > wrote: > >> On Thu, Jan 14, 2016 at 6:06 PM, Austin Schuh <aus...@peloton-tech.com> >> wrote: >> >>> Hi, >>> >>> I've got an application where I can't allocate memory while using >>> protobufs. Arenas have been awesome for doing that. I'm able to allocate >>> a big block of memory at startup time or stack allocate memory for the >>> arena, and then use that for allocating protobufs. Thanks! >>> >>> I'd like to be able to allocate strings in the arena. I'm willing to do >>> the implementation, and wouldn't mind up-streaming if my implementation is >>> complete enough and there is interest. It looks like I should start by >>> implementing ctype=STRING_PIECE and then allocate memory in the arena to >>> back it. The class in //src/google/protobuf:arenastring.h looks like the >>> place to do all the operations. It looks like I need to modify the >>> interface to provide setters and getters to support STRING_PIECE there. >>> >>> Is that the right place to start? Is there any more guidance that you >>> can give me? >>> >> Hi Austin, >> >> Thanks for contacting us and offering help! >> >> You are looking at the right direction. We actually already opensourced >> the StringPiece implementation not very long ago: >> >> https://github.com/google/protobuf/blob/master/src/google/protobuf/stubs/stringpiece.h >> >> It's intended to be used to implement "ctype = STRING_PIECE" for string >> fields and since it's merely a <const char*, size_t> pair, it can be >> directed at the buffer in the arena. Such features are implemented inside >> Google but unfortunately it's not opensourced due to dependency issues. We >> plan to get them out eventually but hasn't have enough time to work on it. >> Since we already have an internal version of it, we probably won't be able >> to accept your contributions. I can't give a concrete timeline about when >> we will get our implementation opensourced also. Sorry for that... >> >> If you need this soon, I suggest you try to implement it as simple as >> possible. Better to only support lite runtime with arena enabled. Some >> changes you want to make: >> 1. Make ArenaStringPtr work with StringPiece, or introduce an >> ArenaStringPiecePtr which might be easier to implement. >> 2. Update protocol compiler to use ArenaStringPtr/ArenaStringPiecePtr to >> store ctype=STRING_PIECE fields and expose a StringPiece API: >> // proto >> message Foo { >> string bar = 1 [ctype = STRING_PIECE]; >> } >> // generated C++ code >> message Foo { >> public: >> StringPiece bar() const; >> void set_bar(StringPiece value); // Note that we need to do a deep >> copy here because StringPiece doesn't own the underlying data. >> void set_alias_bar(StringPiece value); // Make the field point to the >> StringPiece data directly. Caller must make sure the underlying data >> outlives the Foo message. >> >> private: >> ArenaStringPiecePtr bar_; >> }; >> >> Look at the string_field.cc implementation in the compiler directory >> <https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/cpp/cpp_string_field.cc> >> and you can create a string_piece_field.cc implementation based on that. >> Most of the work will be done here, including not only the generated API >> but also all the parsing/serialization/copy/constructor/destructor support. >> >> That's pretty all that needed to support StringPiece in lite-runtime + >> arena. A lot more work will be needed to support other combinations >> (lite-runtime + no arena, full-runtime + arena, full-runtime + non-arena), >> but since you have a specific targeted platform and we will opensource the >> StringPiece support eventually, it's probably not worthwhile to invest time >> to support anything you don't actually need right now. >> >> Hope this helps. >> >> Regards, >> Feng >> >> >>> Thanks, >>> Austin >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to protobuf+u...@googlegroups.com. >>> To post to this group, send email to prot...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/protobuf/ccc70ecc-873a-4297-8102-8e2319ffd760n%40googlegroups.com > <https://groups.google.com/d/msgid/protobuf/ccc70ecc-873a-4297-8102-8e2319ffd760n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CADqAXr4AUyuqy_uPAXhau7OjPrfYtH8kwSgzhpWDxOM%3DCQjE5g%40mail.gmail.com.
