Since we’re on the subject of choice of serialization library, I’d like to bring up cap’n proto and flat buffers, both of which allow random reads into the serialized data without deserialization. There’s a (possibly biased) comparison here <https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-sbe.html>. I’m not sure how much of the time for loading data from disk in Quickstep is due to protobuf deserialization, but I’d guess that it’s quite a large fraction. Getting rid of that might speed up the first run of queries, and improve performance when we have to deal with spillover.
Of course, migrating from Protobuf v2 to v3 requires only minimal changes to the *.proto files, and I don’t know how much work it would be to migrate to one of these other serialization libraries. So it’s not clear if it would be worth it. > On Nov 30, 2016, at 09:31, Harshad Deshmukh <[email protected]> wrote: > > +1 for upgrade from me too. Having richer data structure support like maps > should be quite useful. > > Do you have some idea about the changes we need to do in the proto files? I > think Zuyu heavily uses proto for his work, so it will be useful to have his > opinion on the upgrade. > > > On 11/29/2016 05:03 PM, Marc Spehlmann wrote: >> +1 for the upgrade option >> >> On Tue, Nov 29, 2016 at 11:45 AM, Saket Saurabh <[email protected]> >> wrote: >> >>> Hi all, >>> I would like to start a discussion around how to fix the failing build >>> issue on OSX 10.12 due to deprecation of certain system APIs in the new OSX >>> version. >>> The Quickstep code does not rely on these deprecated APIs, but it is the >>> protobuf library which uses these deprecated APIs. [See: >>> https://github.com/google/protobuf/issues/2182] >>> This has already been resolved in the latest version of protobuf [Version >>>> = 3.0] [See: https://github.com/google/protobuf/pull/2337] >>> So, I foresee two possible ways to now fix this in Quickstep: >>> >>> 1. Use the compile-time time flag -Wno-deprecated-declarations to >>> ensure that the deprecated warnings do not cause build time failure. >>> >>> a. This may not be the best way to solve this. Turning off warnings >>> is always a bad idea, anyways. >>> >>> 2. Upgrade to Protobuf Version 3. >>> >>> a. This may require a one-time migration of our current .proto >>> files, but on the other hand, it will enable Quickstep to use the improved >>> Protobuf 3 library with richer APIs and features [e.g. maps, etc.]. >>> >>> For now, Option 2 seems like a favorable long-term approach to solve this >>> problem. Kindly raise if someone has a concern with this approach, or has a >>> better alternative in mind. >>> >>> Thanks! >>> Saket >>> > > -- > Thanks, > Harshad >
