Re: [VOTE] Apache Spark 2.1.1 (RC3)

see https://issues.apache.org/jira/browse/SPARK-19611 On Mon, Apr 24, 2017 at 2:22 PM, Holden Karau wrote: > Whats the regression this fixed in 2.1 from 2.0? > > On Fri, Apr 21, 2017 at 7:45 PM, Wenchen Fan > wrote: > >> IIRC, the new

Re: [VOTE] Apache Spark 2.1.1 (RC3)

Yeah, I agree. -1 (binding) This vote fails, and I'll cut a new RC after #17749 is merged. On Mon, Apr 24, 2017 at 12:18 PM, Eric Liang wrote: > -1 (non-binding) > > I also agree with using NEVER_INFER for 2.1.1. The migration

Re: [VOTE] Apache Spark 2.1.1 (RC3)

Whoops, sorry finger slipped on that last message. It sounds like whatever we do is going to break some existing users (either with the tables by case sensitivity or with the unexpected scan). Personally I agree with Michael Allman on this, I believe we should use INFER_NEVER for 2.1.1. On Mon,

Re: [VOTE] Apache Spark 2.1.1 (RC3)

It On Mon, Apr 24, 2017 at 10:33 AM, Michael Allman wrote: > The trouble we ran into is that this upgrade was blocking access to our > tables, and we didn't know why. This sounds like a kind of migration > operation, but it was not apparent that this was the case. It took

Re: [VOTE] Apache Spark 2.1.1 (RC3)

The trouble we ran into is that this upgrade was blocking access to our tables, and we didn't know why. This sounds like a kind of migration operation, but it was not apparent that this was the case. It took an expert examining a stack trace and source code to figure this out. Would a more

Re: What is correct behavior for spark.task.maxFailures?

Looking at the code a bit more, it appears that blacklisting is disabled by default. To enable it, set spark.blacklist.enabled=true. The updates in 2.1.0 appear to provide much more fine-grained settings for this, like the number of tasks that can fail before an executor is blacklisted for a

Re: [VOTE] Apache Spark 2.1.1 (RC3)

Whats the regression this fixed in 2.1 from 2.0? On Fri, Apr 21, 2017 at 7:45 PM, Wenchen Fan wrote: > IIRC, the new "spark.sql.hive.caseSensitiveInferenceMode" stuff will only > scan all table files only once, and write back the inferred schema to > metastore so that we