I just read the blog post [1] about Crail and file formats. (I have to declare my interests up front: I have been a huge supporter of Apache Arrow, and I am a PMC member. I’m speaking here as an Arrow contributor and enthusiast, not as a mentor of Crail.)
I am a bit troubled about the endorsement of Albis in a Crail blog post. For example, "we have developed a new file format called Albis”. Since the blog post is not signed, I take it that “We” means the authors of the paper [2] mentioned in the blog post. But I hope that “we” does not mean “we as Crail committers and PMC members". I know that there are different forces at play if you work for a corporation, or are a researcher, or are an idealistic open source. As a researcher, you need to invent new stuff and prove that it is better than everything that has been done before. But I’ve been through the file format wars — ORC vs Parquet — driven in large part by two competing vendors. It was sickening, and a huge waste of effort. Please, please don’t let this happen again. If you want to make Crail successful, you should make it absolutely clear to the Arrow, ORC and Parquet communities that you will help to make Crail work as well as it possibly can Also, on paper Albis looks very similar to Arrow, and the performance gap is fairly narrow. If you have found insights that would improve Arrow, I encourage you to share them and make Arrow better. It may be good research practice to accentuate the differences between the two, but it’s good open source practice to find consensus between technologies, and merge communities. There is a lot of work to be done, and too few people to do it. Lastly, I know I seem to be giving mixed messages here. I do believe that content about Crail will help drive engagement and build community (controversial content even more so). I am delighted that the Crail team is writing blog posts and posting them to Twitter. But be careful not to alienate communities that could help Crail gain widespread adoption. Julian [1] http://crail.incubator.apache.org/blog/2018/08/sql-p1.html <http://crail.incubator.apache.org/blog/2018/08/sql-p1.html> [2] https://www.usenix.org/conference/atc18/presentation/trivedi <https://www.usenix.org/conference/atc18/presentation/trivedi>
