As you might already know the FoundationDB has a number of limitations which influences the way we might store JSON documents. The limitations are:
| limitation |recommended value|recommended max|absolute max| |-------------------------|----------------------:|--------------------:|--------------:| | transaction duration | | | 5 sec | | transaction data size | | | 10 Mb | | key size | 32 bytes | 1 kB | 10 kB | | value size | | 10 kB | 100 kB | In order to fit the JSON document into 100kB we would have to partition it in some way. There are three ways of partitioning the document 1. store multiple binary blobs (parts) in different keys 2. flatten JSON structure and store every path leading to a scalar value under own key 3. measure the size of different branches of a tree representing the JSON document (while we parse) and use another key for the branch when we about to exceed the limit - The first approach is the simplest but it wouldn't allow us to access parts of the document. - The downsides of a second approach are: - flattened JSON structure would have long paths which means longer keys - the scalar value cannot be more than 100kb (unless we split it as well) - Third approach falls short in cases when the structure of the document doesn't allow a clean cut off branches: - complex rules to handle all corner cases The goals of this thread are: - to collect ideas on how to encode and store the JSON document - to comment on the collected ideas Non goals: - the storage of metadata for the document would be discussed elsewhere - thumb stones - edit conflicts - revisions Best regards, iilyak
