Hi Stefan, thanks for this information - so it seems that there is currently no way of accessing nested rich objects with drill; I somehow got that wrong from the documentation...
Cheers, Johannes On Thu, Mar 24, 2016 at 2:14 PM, Stefán Baxter <ste...@activitystream.com> wrote: > FYI: flattening of embedded structures is not supported in Parquet either. > > Regards, > -Stefan > > On Wed, Mar 23, 2016 at 8:51 PM, Johannes Schulte < > johannes.schu...@gmail.com> wrote: > > > Hi Stefan, > > > > thanks for your response and the link to your udf repository, it's a good > > reference. I tried drill 1.6, the data is an array of complex objects > > though. I will try to setup a drill dev environment and see if i can > modify > > the tests to fail. > > > > Johannes > > > > On Wed, Mar 23, 2016 at 8:13 PM, Stefán Baxter < > ste...@activitystream.com> > > wrote: > > > > > FYI. this seems to be working in 1.6, at least on the Avro data that we > > > have. > > > > > > On Wed, Mar 23, 2016 at 6:59 PM, Stefán Baxter < > > ste...@activitystream.com> > > > wrote: > > > > > > > Hi again, > > > > > > > > What version of Drill are you using? > > > > > > > > Regards, > > > > - Stefán > > > > > > > > On Wed, Mar 23, 2016 at 4:49 PM, Stefán Baxter < > > > ste...@activitystream.com> > > > > wrote: > > > > > > > >> Hi Johannes, > > > >> > > > >> As great as Drill is the Avro plugin has been a source of > frustration > > > for > > > >> us @activitystream. > > > >> > > > >> We have a small UDF library [1] (apache licensed) which contains a > > > >> function can return an array (List<String>) from Avro as a CSV list. > > > >> > > > >> You could use that to roll your own or provide me with a small > sample > > > and > > > >> I can create a custom flatten function for you. > > > >> > > > >> The best would be to wait for a fix but this can potentially get you > > out > > > >> of a rough spot. > > > >> > > > >> [1] https://github.com/activitystream/asdrill > > > >> > > > >> Regards, > > > >> -Stefán > > > >> > > > >> On Wed, Mar 23, 2016 at 9:05 AM, Johannes Schulte < > > > >> johannes.schu...@gmail.com> wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> when trying to read simple avro arrays with select flatten(array) > > from > > > >>> dfs... i get the exception > > > >>> > > > >>> SQL Query Error: SYSTEM ERROR: ClassCastException: Cannot cast > > > >>> org.apache.drill.exec.vector.complex.MapVector to > > > >>> org.apache.drill.exec.vector.complex.RepeatedValueVector > > > >>> ^ > > > >>> > > > >>> The type of the array is said to be <UnknownType (2,002)> > > > >>> > > > >>> Is this the expected behaviour? The documentation mostly talsk > about > > > json > > > >>> and parquet complex types and i wonder if the avro storage plugin > > > behaves > > > >>> differently. > > > >>> > > > >>> Thanks, > > > >>> > > > >>> Johannes > > > >>> > > > >> > > > >> > > > > > > > > > >
