Re: [Spark 2.0.0] error when unioning to an empty dataset

All right, I looked at the schemas. There is one mismatching nullability, on a scala.Boolean. It looks like an empty Dataset with that *cannot* be nullable. However, when I run my code to generate the Dataset, the schema comes back with nullable = true. Effectively: scala> val empty =

Re: [Spark 2.0.0] error when unioning to an empty dataset

On 10/22/16 1:42 PM, Efe Selcuk wrote: Ah, looks similar. Next opportunity I get, I'm going to do a printSchema on the two datasets and see if they don't match up. I assume that unioning the underlying RDDs doesn't run into this problem because of less type checking or something along those

Re: [Spark 2.0.0] error when unioning to an empty dataset

Ah, looks similar. Next opportunity I get, I'm going to do a printSchema on the two datasets and see if they don't match up. I assume that unioning the underlying RDDs doesn't run into this problem because of less type checking or something along those lines? On Fri, Oct 21, 2016 at 3:39 PM

Re: [Spark 2.0.0] error when unioning to an empty dataset

Efe - You probably hit this bug: https://issues.apache.org/jira/browse/SPARK-18058 On 10/21/16 2:03 AM, Agraj Mangal wrote: I have seen this error sometimes when the elements in the schema have different nullabilities. Could you print the schema for data and for

Re: [Spark 2.0.0] error when unioning to an empty dataset

I have seen this error sometimes when the elements in the schema have different nullabilities. Could you print the schema for data and for someCode.thatReturnsADataset() and see if there is any difference between the two ? On Fri, Oct 21, 2016 at 9:14 AM, Efe Selcuk wrote: >

Re: [Spark 2.0.0] error when unioning to an empty dataset

Thanks for the response. What do you mean by "semantically" the same? They're both Datasets of the same type, which is a case class, so I would expect compile-time integrity of the data. Is there a situation where this wouldn't be the case? Interestingly enough, if I instead create an empty rdd

Re: [Spark 2.0.0] error when unioning to an empty dataset

I believe this normally comes when Spark is unable to perform union due to "difference" in schema of the operands. Can you check if the schema of both the datasets are semantically same ? On Tue, Oct 18, 2016 at 9:06 AM, Efe Selcuk wrote: > Bump! > > On Thu, Oct 13, 2016 at

Re: [Spark 2.0.0] error when unioning to an empty dataset

Bump! On Thu, Oct 13, 2016 at 8:25 PM Efe Selcuk wrote: > I have a use case where I want to build a dataset based off of > conditionally available data. I thought I'd do something like this: > > case class SomeData( ... ) // parameters are basic encodable types like >

[Spark 2.0.0] error when unioning to an empty dataset

I have a use case where I want to build a dataset based off of conditionally available data. I thought I'd do something like this: case class SomeData( ... ) // parameters are basic encodable types like strings and BigDecimals var data = spark.emptyDataset[SomeData] // loop, determining what