Re: Avro - Let's talk Avro again

2017-08-19 Thread Stefán Baxter
Thank you Saurabh, I was not really expecting a constructive reply to my previous email, was appreciated. I guess some old frustration got the better of me. All the best, -Stefan On Fri, Aug 18, 2017 at 10:55 PM, Saurabh Mahapatra < saurabhmahapatr...@gmail.com> wrote: > Thank you for this ca

Re: Drill schema handling [Was: Avro - Let's talk Avro again]

2017-08-19 Thread Stefán Baxter
Hello and thank you. Please also put this in the Avro context. That is quite reveling when thinking about Evolving schema support in Drill, not just for Avro but in general. Avro supports evolving schema and some might say it's designed around exactly that. The current handling of Avro in Drill

Drill schema handling [Was: Avro - Let's talk Avro again]

2017-08-19 Thread Paul Rogers
Hi All, Let’s also talk about the other core issue that Stefan raised: schema handling. Drill is very powerful in its use of JSON as its internal data format. Drill effortlessly handles maps and arrays as well as the more traditional scalar types. This can cause confusion, however. We all know

Re: Avro - Let's talk Avro again

2017-08-19 Thread Paul Rogers
Let’s talk a bit about Avro as an example of the whole class of “extended” storage plugins maintained by the community. Today, each storage plugin requires very tight integration into Drill. Each must implement its own support for implicit columns (such as “filename” or “dir0”), must support its

Re: Unable to SELECT from parquet file with Hadoop 2.7.4

2017-08-19 Thread Michele Lamarca
Hi Kunal, Just filed https://issues.apache.org/jira/browse/DRILL-5733 Kind regards, Michele On Fri, Aug 18, 2017 at 10:35 PM, Kunal Khatua wrote: > Hi Michele > > Could you also share the entire stack track in the Drillbit log when logging > the JIRA? > > I think you might have run this throu