Hi! I’m working on a new data format using Apache Drill as query engine. It has very nice scan speed, high compression ratio, and contains indexes to filter out irrelevant parts. Even more interesting is it supports extremely fast messages realtime ingestion. Currently it goes very well and we are planning to open sourced in one or two months.
Drill is very promising during our developing days. And we wish that it could support the Late Materialization(http://db.csail.mit.edu/projects/cstore/abadiicde2007.pdf <http://db.csail.mit.edu/projects/cstore/abadiicde2007.pdf>), which can make a very different when the filters are very effective and many columns are requested. Regards Flow Wei
