For a single Text or Data blob there is a hard limit of 512MB. You can, however, construct a message which contains multiple blobs, e.g. use `List(Text)`. Such a message can be up to 2^64 bytes.
If I were redesigning the encoding from scratch I'd probably allow for bigger individual blobs but there's no way to introduce them now without breaking compatibility, unfortunately. Protobuf theoretically supports 2GB messages but because the messages have to be parsed upfront in O(n) time, you won't have good results with messages of that size. Cap'n Proto, on the other hand, quite comfortably supports multi-GB messages since you can mmap() the message and randomly access it in O(1) time. -Kenton On Mon, Oct 31, 2022 at 2:21 PM Josemi <josemi....@gmail.com> wrote: > Hello. > > I need to work with an structured data that have atributes with undefined > lenght, some of them could have GB. > > I have been using Protocol Buffers for it until I see that exists a hard > limit of 2GB per message. Then I see this Stack Overflow solution > <https://stackoverflow.com/questions/34128872/google-protobuf-maximum-size>. > It says that Cap'n Proto "can support messages up to 2^64 bytes (2^32 > segments of 4GB each)" > > I reprogram the code and now it raise the error *capnp.lib.capnp.KjException: > capnp/layout.c++:1694: failed: text blob too big * trying to set a 0.8 > GB buffer to a Data type. > > On Cap'n Proto the atribute seems like that: > *file @1 :Data; # ptr[1],* > > And the code, in Python, is something like: > file = open('data_file', 'rb').read() > > > Whats wrong with that. > Cap'n Proto won't solve my problem?. > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "Cap'n Proto" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to capnproto+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/capnproto/8a23280b-b43b-483a-94db-7fd94bba93een%40googlegroups.com > <https://groups.google.com/d/msgid/capnproto/8a23280b-b43b-483a-94db-7fd94bba93een%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Cap'n Proto" group. To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/CAJouXQk0hybS%2Bc21E7NagDBQqNcCaXkTHWoPsmAT%3DiwNdHjfwg%40mail.gmail.com.
