Hi Oscar, I'd definitely be interested in your approach to containerization and running on Drill, Alluxio etc. on Kubernetes. A while ago I made an attempt at dockerizing Alluxio + Drill but hit a road block wrt. networking issues. Would be great if you have something you can share.
Regards, Ralph -----Original Message----- From: Oscar Morante [mailto:[email protected]] Sent: 28 September 2016 08:26 To: [email protected] Subject: Re: Support for Alluxio Hi there, I've been looking into this. It's almost working except for some issues when reading lots of parquet files, but I'm not sure if that is Alluxio's fault. I have a couple of WIP Docker images that's I'm using on Kubernetes. I can share if you want, but it's all very hackish right now and I'm still changing things very often. Cheers, On Wed, Sep 14, 2016 at 11:13:49AM -0700, Sudheesh Katkam wrote: >Based on the documentation [1], Alluxio not only uses HDFS as a underlying >storage system but also provides a “Hadoop API” to clients, and Drill uses >this API. So it should be possible. > >Create a storage plugin named “alluxio” whose contents match “dfs” plugin, and >then make changes. Not sure what exactly, but [2] should be helpful. See “S3 >Storage Plugin” as an example as well [3]. Once you get things to work, please >contribute by adding a section to the Drill documentation. > >Thank you, >Sudheesh > >[1] >http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api ><http://www.alluxio.org/docs/master/en/File-System-API.html#hadoop-api> >[2] http://www.alluxio.org/docs/master/en/Configuration-Settings.html ><http://www.alluxio.org/docs/master/en/Configuration-Settings.html> >[3] http://drill.apache.org/docs/s3-storage-plugin/ ><http://drill.apache.org/docs/s3-storage-plugin/> > >> On Sep 14, 2016, at 10:07 AM, Edmon Begoli <[email protected]> wrote: >> >> Is it possible to use Alluxio as a filesystem backend in Drill's >> storage formats, and if so how? >> >> Thanks. > -- Oscar Morante "Self-education is, I firmly believe, the only kind of education there is." -- Isaac Asimov.
