FYI: flattening of embedded structures is not supported in Parquet either. Regards, -Stefan
On Wed, Mar 23, 2016 at 8:51 PM, Johannes Schulte < johannes.schu...@gmail.com> wrote: > Hi Stefan, > > thanks for your response and the link to your udf repository, it's a good > reference. I tried drill 1.6, the data is an array of complex objects > though. I will try to setup a drill dev environment and see if i can modify > the tests to fail. > > Johannes > > On Wed, Mar 23, 2016 at 8:13 PM, Stefán Baxter <ste...@activitystream.com> > wrote: > > > FYI. this seems to be working in 1.6, at least on the Avro data that we > > have. > > > > On Wed, Mar 23, 2016 at 6:59 PM, Stefán Baxter < > ste...@activitystream.com> > > wrote: > > > > > Hi again, > > > > > > What version of Drill are you using? > > > > > > Regards, > > > - Stefán > > > > > > On Wed, Mar 23, 2016 at 4:49 PM, Stefán Baxter < > > ste...@activitystream.com> > > > wrote: > > > > > >> Hi Johannes, > > >> > > >> As great as Drill is the Avro plugin has been a source of frustration > > for > > >> us @activitystream. > > >> > > >> We have a small UDF library [1] (apache licensed) which contains a > > >> function can return an array (List<String>) from Avro as a CSV list. > > >> > > >> You could use that to roll your own or provide me with a small sample > > and > > >> I can create a custom flatten function for you. > > >> > > >> The best would be to wait for a fix but this can potentially get you > out > > >> of a rough spot. > > >> > > >> [1] https://github.com/activitystream/asdrill > > >> > > >> Regards, > > >> -Stefán > > >> > > >> On Wed, Mar 23, 2016 at 9:05 AM, Johannes Schulte < > > >> johannes.schu...@gmail.com> wrote: > > >> > > >>> Hi, > > >>> > > >>> when trying to read simple avro arrays with select flatten(array) > from > > >>> dfs... i get the exception > > >>> > > >>> SQL Query Error: SYSTEM ERROR: ClassCastException: Cannot cast > > >>> org.apache.drill.exec.vector.complex.MapVector to > > >>> org.apache.drill.exec.vector.complex.RepeatedValueVector > > >>> ^ > > >>> > > >>> The type of the array is said to be <UnknownType (2,002)> > > >>> > > >>> Is this the expected behaviour? The documentation mostly talsk about > > json > > >>> and parquet complex types and i wonder if the avro storage plugin > > behaves > > >>> differently. > > >>> > > >>> Thanks, > > >>> > > >>> Johannes > > >>> > > >> > > >> > > > > > >
