Re: Does schema merge on keys with different types is allowed?

I have tested that issue manually and looked into the codes before. It seems it does not support to find a compatible type. https://github.com/apache/spark/blob/b914e1930fd5c5f2808f92d4958ec6fbeddf2e30/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L396-L465 2016-06-29

Re: Structured Streaming Sink in 2.0 collect/foreach restrictions added in SPARK-16020

Yeah, turning it into an RDD should preserve the incremental planning. On Tue, Jun 28, 2016 at 6:30 PM, Holden Karau wrote: > Ok, that makes sense (the JIRA where the restriction note was added didn't > have a lot of details). So for now, would converting to an RDD inside

Re: Structured Streaming Sink in 2.0 collect/foreach restrictions added in SPARK-16020

Ok, that makes sense (the JIRA where the restriction note was added didn't have a lot of details). So for now, would converting to an RDD inside of a custom Sink and then doing your operations on that be a reasonable work around? On Tuesday, June 28, 2016, Michael Armbrust

Re: Spark SQL PR looking for love...

I should briefly mention what the PR is about... This is a patch to address a problem where non-empty partitioned Hive metastore tables are never returned in a cache lookup in HiveMetastoreCatalog.getCached. Thanks, Michael > On Jun 28, 2016, at 3:27 PM, Michael Allman

Spark SQL PR looking for love...

Hello, Do any Spark SQL committers/experts have bandwidth to review a PR I submitted a week ago, https://github.com/apache/spark/pull/13818 ? The associated Jira ticket is https://issues.apache.org/jira/browse/SPARK-15968

Structured Streaming Sink in 2.0 collect/foreach restrictions added in SPARK-16020

Looking at the Sink in 2.0 there is a warning (added in SPARK-16020 without a lot of details) that says "Note: You cannot apply any operators on `data` except consuming it (e.g., `collect/foreach`)." but I'm wondering if this restriction is perhaps too broadly worded? Provided that we consume the

[build system] hanging procs on the jenkins master, emergency reboot

jenkins got itself in to another state and was killing the master. while poking around, i noticed lots of sleeping processes that were using a TON of cpu and had a bunch of log files open, but not writing to them. anyways, it looked like it needed a quick restart and that seems to have fixed the

Re: What's the meaning of Target Version/s in Spark's JIRA?

Hi, That makes sense. Thanks Dongjoon for the very prompt response! Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Tue, Jun 28, 2016 at 6:58 PM, Dongjoon Hyun

Re: What's the meaning of Target Version/s in Spark's JIRA?

Hi, 1.6.2 is just the result of back-porting of that patch. The patch was originally targeted and merged into 2.0.0. Warmly, Dongjoon. On Tue, Jun 28, 2016 at 10:54 AM, Jacek Laskowski wrote: > Hi, > > While reviewing the release notes for 1.6.2 I stumbled upon >

What's the meaning of Target Version/s in Spark's JIRA?

Hi, While reviewing the release notes for 1.6.2 I stumbled upon https://issues.apache.org/jira/browse/SPARK-13522. It's got Target Version/s: 2.0.0 with Fix Version/s: 1.6.2, 2.0.0. What's the meaning of Target Version/s in Spark? Pozdrawiam, Jacek Laskowski

UDTRegistration public

Hi, I have discovered class UDTRegistration on master. Will it be made public? If yes, when and under which version? https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/UDTRegistration.scala Thanks! Patrick. -- View this message in context:

Re: Please add an unsubscribe link to the footer of user list email

>From the ticket: Before going ahead I would like to see a much broader consensus of the changes being asked for. At least 3 +1s from PMC members would be ok. Can some PMC members chime in there please? Nick ​ On Tue, Jun 28, 2016 at 1:13 AM Reynold Xin wrote: > If

Re: [VOTE] Release Apache Spark 2.0.0 (RC1)

I take it there will be another RC due to some blockers and as there were no +1 votes anyway. FWIW, I cannot run python tests using "./python/run-tests". I'd be -1 for this reason (see https://github.com/apache/spark/pull/13737 / http://issues.apache.org/jira/browse/SPARK-15954) - does anyone