Re: Selecting column in dataframe created with incompatible schema causes AnalysisException

Thanks. Once you create the jira just reply to this email with the link. On Wednesday, March 2, 2016, Ewan Leith wrote: > Thanks, I'll create the JIRA for it. Happy to help contribute to a patch if > we can, not sure if my own scala skills will be up to it but

Re: Selecting column in dataframe created with incompatible schema causes AnalysisException

Thanks, I'll create the JIRA for it. Happy to help contribute to a patch if we can, not sure if my own scala skills will be up to it but perhaps one of my colleagues' will :) Ewan I don't think that exists right now, but it's definitely a good option to have. I myself have run into this issue

Re: Selecting column in dataframe created with incompatible schema causes AnalysisException

I don't think that exists right now, but it's definitely a good option to have. I myself have run into this issue a few times. Can you create a JIRA ticket so we can track it? Would be even better if you are interested in working on a patch! Thanks. On Wed, Mar 2, 2016 at 11:51 AM, Ewan Leith

Re: Selecting column in dataframe created with incompatible schema causes AnalysisException

Hi Reynold, yes that would be perfect for our use case. I assume it doesn't exist though, otherwise I really need to go re-read the docs! Thanks to both of you for replying by the way, I know you must be hugely busy. Ewan Are you looking for "relaxed" mode that simply return nulls for fields

Re: Selecting column in dataframe created with incompatible schema causes AnalysisException

Are you looking for "relaxed" mode that simply return nulls for fields that doesn't exist or have incompatible schema? On Wed, Mar 2, 2016 at 11:12 AM, Ewan Leith wrote: > Thanks Michael, it's not a great example really, as the data I'm working with > has some

Re: Selecting column in dataframe created with incompatible schema causes AnalysisException

Thanks Michael, it's not a great example really, as the data I'm working with has some source files that do fit the schema, and some that don't (out of millions that do work, perhaps 10 might not). In an ideal world for us the select would probably return the valid records only. We're trying

Re: Selecting column in dataframe created with incompatible schema causes AnalysisException

-dev +user StructType(StructField(data,ArrayType(StructType(StructField( > *stuff,ArrayType(*StructType(StructField(onetype,ArrayType(StructType(StructField(id,LongType,true), > StructField(name,StringType,true)),true),true), StructField(othertype, >