[jira] [Assigned] (SPARK-22644) Make ML testsuite support StructuredStreaming test

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22644: Assignee: (was: Apache Spark) > Make ML testsuite support StructuredStreaming test >

[jira] [Assigned] (SPARK-22644) Make ML testsuite support StructuredStreaming test

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22644: Assignee: Apache Spark > Make ML testsuite support StructuredStreaming test >

[jira] [Commented] (SPARK-22644) Make ML testsuite support StructuredStreaming test

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270325#comment-16270325 ] Apache Spark commented on SPARK-22644: -- User 'WeichenXu123' has created a pull request for this

[jira] [Created] (SPARK-22644) Make ML testsuite support StructuredStreaming test

2017-11-28 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-22644: -- Summary: Make ML testsuite support StructuredStreaming test Key: SPARK-22644 URL: https://issues.apache.org/jira/browse/SPARK-22644 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22643) ColumnarArray should be an immutable view

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270307#comment-16270307 ] Apache Spark commented on SPARK-22643: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22643) ColumnarArray should be an immutable view

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22643: Assignee: Apache Spark (was: Wenchen Fan) > ColumnarArray should be an immutable view >

[jira] [Assigned] (SPARK-22643) ColumnarArray should be an immutable view

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22643: Assignee: Wenchen Fan (was: Apache Spark) > ColumnarArray should be an immutable view >

[jira] [Created] (SPARK-22643) ColumnarArray should be an immutable view

2017-11-28 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-22643: --- Summary: ColumnarArray should be an immutable view Key: SPARK-22643 URL: https://issues.apache.org/jira/browse/SPARK-22643 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-28 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270259#comment-16270259 ] Wenchen Fan commented on SPARK-22472: - yea I think we should, this may turn a runnable query(with an

[jira] [Assigned] (SPARK-22642) the createdTempDir will not be deleted if an exception occurs

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22642: Assignee: (was: Apache Spark) > the createdTempDir will not be deleted if an

[jira] [Commented] (SPARK-22642) the createdTempDir will not be deleted if an exception occurs

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270228#comment-16270228 ] Apache Spark commented on SPARK-22642: -- User 'zuotingbing' has created a pull request for this

[jira] [Assigned] (SPARK-22642) the createdTempDir will not be deleted if an exception occurs

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22642: Assignee: Apache Spark > the createdTempDir will not be deleted if an exception occurs >

[jira] [Created] (SPARK-22642) the createdTempDir will not be deleted if an exception occurs

2017-11-28 Thread zuotingbing (JIRA)
zuotingbing created SPARK-22642: --- Summary: the createdTempDir will not be deleted if an exception occurs Key: SPARK-22642 URL: https://issues.apache.org/jira/browse/SPARK-22642 Project: Spark

[jira] [Commented] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270208#comment-16270208 ] Kent Yao commented on SPARK-22640: -- [~Jeff_Z] spark.yarn.appMasterEnv specified > Can not switch python

[jira] [Updated] (SPARK-22637) CatalogImpl.refresh() has quadratic complexity for a view

2017-11-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-22637: - Target Version/s: 2.2.2, 2.3.0 (was: 2.3.0) > CatalogImpl.refresh() has quadratic complexity

[jira] [Commented] (SPARK-22472) Datasets generate random values for null primitive types

2017-11-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270178#comment-16270178 ] Felix Cheung commented on SPARK-22472: -- This is going out in 2.2.1 - do we need a rel note on this

[jira] [Commented] (SPARK-22627) Fix formatting of headers in configuration.html page

2017-11-28 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270169#comment-16270169 ] Felix Cheung commented on SPARK-22627: -- quite possibly this is related to jekyll changes.. yap this

[jira] [Commented] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270139#comment-16270139 ] Jeff Zhang commented on SPARK-22640: you need to use spark.yarn.appMasterEnv since you are using yarn

[jira] [Commented] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270132#comment-16270132 ] Andrew Duffy commented on SPARK-22641: -- Query plan with the literal: {code} == Parsed Logical Plan

[jira] [Commented] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270128#comment-16270128 ] Andrew Duffy commented on SPARK-22641: -- So it seems this is only a problem when using literal

[jira] [Comment Edited] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270128#comment-16270128 ] Andrew Duffy edited comment on SPARK-22641 at 11/29/17 4:44 AM: So it

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Updated] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Duffy updated SPARK-22641: - Description: We seem to have found an issue with PySpark UDFs interacting with {{withColumn}}

[jira] [Created] (SPARK-22641) Pyspark UDF relying on column added with withColumn after distinct

2017-11-28 Thread Andrew Duffy (JIRA)
Andrew Duffy created SPARK-22641: Summary: Pyspark UDF relying on column added with withColumn after distinct Key: SPARK-22641 URL: https://issues.apache.org/jira/browse/SPARK-22641 Project: Spark

[jira] [Assigned] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22640: Assignee: Apache Spark > Can not switch python exec in executor side >

[jira] [Commented] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270067#comment-16270067 ] Apache Spark commented on SPARK-22640: -- User 'yaooqinn' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22640: Assignee: (was: Apache Spark) > Can not switch python exec in executor side >

[jira] [Created] (SPARK-22640) Can not switch python exec in executor side

2017-11-28 Thread Kent Yao (JIRA)
Kent Yao created SPARK-22640: Summary: Can not switch python exec in executor side Key: SPARK-22640 URL: https://issues.apache.org/jira/browse/SPARK-22640 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22373: Assignee: (was: Apache Spark) > Intermittent NullPointerException in >

[jira] [Assigned] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22373: Assignee: Apache Spark > Intermittent NullPointerException in >

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270053#comment-16270053 ] Apache Spark commented on SPARK-22373: -- User 'Victsm' has created a pull request for this issue:

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Min Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270055#comment-16270055 ] Min Shen commented on SPARK-22373: -- Created PR https://github.com/apache/spark/pull/19839 [~sowen]

[jira] [Commented] (SPARK-21918) HiveClient shouldn't share Hive object between different thread

2017-11-28 Thread Reid Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270040#comment-16270040 ] Reid Chan commented on SPARK-21918: --- Expecting > HiveClient shouldn't share Hive object between

[jira] [Updated] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Min Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-22373: - Attachment: generated.java CodeGeneratorTester.scala Attach the standalone testing

[jira] [Commented] (SPARK-19250) In security cluster, spark beeline connect to hive metastore failed

2017-11-28 Thread Reid Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270005#comment-16270005 ] Reid Chan commented on SPARK-19250: --- Hi [~meiyoula], may i ask how do you solve it? > In security

[jira] [Comment Edited] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269974#comment-16269974 ] Raghavendra edited comment on SPARK-22622 at 11/29/17 2:55 AM: --- [~srowen] I

[jira] [Comment Edited] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269974#comment-16269974 ] Raghavendra edited comment on SPARK-22622 at 11/29/17 2:37 AM: --- [~srowen] I

[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269974#comment-16269974 ] Raghavendra commented on SPARK-22622: - [~srowen] I am broadcasting all my large data strucures and im

[jira] [Comment Edited] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269974#comment-16269974 ] Raghavendra edited comment on SPARK-22622 at 11/29/17 2:36 AM: --- [~srowen] I

[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269935#comment-16269935 ] Sean Owen commented on SPARK-22622: --- The error isn't out of memory. It's that an array is too large,

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Min Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269924#comment-16269924 ] Min Shen commented on SPARK-22373: -- [~leigjklotz], I think bumping up Janino version to 3.0.7

[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269917#comment-16269917 ] Raghavendra commented on SPARK-22622: - The output of previous Stage is around 70GB and my driver

[jira] [Comment Edited] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269917#comment-16269917 ] Raghavendra edited comment on SPARK-22622 at 11/29/17 1:50 AM: --- The output

[jira] [Issue Comment Deleted] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavendra updated SPARK-22622: Comment: was deleted (was: The output of previous Stage is around 70GB and my driver memory is

[jira] [Updated] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavendra updated SPARK-22622: Summary: OutOfMemory thrown by Closure Serializer without proper failure propagation (was:

[jira] [Updated] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavendra updated SPARK-22622: Description: While moving from a Stage to another, the Closure serializer is trying to Serialize

[jira] [Updated] (SPARK-22622) OutOfMemory thrown by Closure Serializer

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavendra updated SPARK-22622: Description: While moving from a Stage to another, the Closure serializer is trying to Serialize

[jira] [Comment Edited] (SPARK-22622) OutOfMemory thrown by Closure Serializer

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269894#comment-16269894 ] Raghavendra edited comment on SPARK-22622 at 11/29/17 1:39 AM: --- The output

[jira] [Reopened] (SPARK-22622) OutOfMemory thrown by Closure Serializer

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavendra reopened SPARK-22622: - The output of previous Stage is around 70GB and my driver memory is 225GB. Why am i getting an OOM

[jira] [Issue Comment Deleted] (SPARK-22622) OutOfMemory thrown by Closure Serializer

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raghavendra updated SPARK-22622: Comment: was deleted (was: The output of previous Stage is around 70GB and my driver memory is

[jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer

2017-11-28 Thread Raghavendra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269894#comment-16269894 ] Raghavendra commented on SPARK-22622: - The output of previous Stage is around 70GB and my driver

[jira] [Created] (SPARK-22639) no rowcount estimation returned if groupby clause involves substring

2017-11-28 Thread ey-chih chow (JIRA)
ey-chih chow created SPARK-22639: Summary: no rowcount estimation returned if groupby clause involves substring Key: SPARK-22639 URL: https://issues.apache.org/jira/browse/SPARK-22639 Project: Spark

[jira] [Resolved] (SPARK-22637) CatalogImpl.refresh() has quadratic complexity for a view

2017-11-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22637. - Resolution: Fixed Assignee: Herman van Hovell Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.x final

2017-11-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269711#comment-16269711 ] Shixiong Zhu commented on SPARK-19552: -- [~wesmckinn] I will try. Looks like not a lot of work. >

[jira] [Updated] (SPARK-22636) row count not being set correctly (always 0) after Statement.executeUpdate()

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22636: -- Priority: Minor (was: Major) Issue Type: Improvement (was: Bug) That doesn't look like a bug,

[jira] [Assigned] (SPARK-22638) Use a separate query for StreamingQueryListenerBus

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22638: Assignee: Apache Spark (was: Shixiong Zhu) > Use a separate query for

[jira] [Assigned] (SPARK-22638) Use a separate query for StreamingQueryListenerBus

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22638: Assignee: Shixiong Zhu (was: Apache Spark) > Use a separate query for

[jira] [Commented] (SPARK-22638) Use a separate query for StreamingQueryListenerBus

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269557#comment-16269557 ] Apache Spark commented on SPARK-22638: -- User 'zsxwing' has created a pull request for this issue:

[jira] [Created] (SPARK-22638) Use a separate query for StreamingQueryListenerBus

2017-11-28 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-22638: Summary: Use a separate query for StreamingQueryListenerBus Key: SPARK-22638 URL: https://issues.apache.org/jira/browse/SPARK-22638 Project: Spark Issue

[jira] [Commented] (SPARK-22620) Deadlock in blacklisting when executor is removed due to a heartbeat timeout

2017-11-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269517#comment-16269517 ] Thomas Graves commented on SPARK-22620: --- Looking at this some more it seems that normally when the

[jira] [Commented] (SPARK-22637) CatalogImpl.refresh() has quadratic complexity for a view

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269472#comment-16269472 ] Apache Spark commented on SPARK-22637: -- User 'hvanhovell' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22637) CatalogImpl.refresh() has quadratic complexity for a view

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22637: Assignee: (was: Apache Spark) > CatalogImpl.refresh() has quadratic complexity for a

[jira] [Updated] (SPARK-22637) CatalogImpl.refresh() has quadratic complexity for a view

2017-11-28 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-22637: -- Description: {{org.apache.spark.sql.internal.CatalogImpl.refreshTable}} uses

[jira] [Assigned] (SPARK-22637) CatalogImpl.refresh() has quadratic complexity for a view

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22637: Assignee: Apache Spark > CatalogImpl.refresh() has quadratic complexity for a view >

[jira] [Resolved] (SPARK-22431) Creating Permanent view with illegal type

2017-11-28 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-22431. --- Resolution: Fixed Assignee: Sunitha Kambhampati Fix Version/s: 2.3.0

[jira] [Created] (SPARK-22637) CatalogImpl.refresh() has quadratic complexity for a view

2017-11-28 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-22637: - Summary: CatalogImpl.refresh() has quadratic complexity for a view Key: SPARK-22637 URL: https://issues.apache.org/jira/browse/SPARK-22637 Project: Spark

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269419#comment-16269419 ] Sean Owen commented on SPARK-22634: --- OK, so Spark pulls it in even with {{-Phadoop-provided}}? I can

[jira] [Assigned] (SPARK-22618) RDD.unpersist can cause fatal exception when used with dynamic allocation

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22618: Assignee: Apache Spark > RDD.unpersist can cause fatal exception when used with dynamic

[jira] [Commented] (SPARK-22618) RDD.unpersist can cause fatal exception when used with dynamic allocation

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269374#comment-16269374 ] Apache Spark commented on SPARK-22618: -- User 'brad-kaiser' has created a pull request for this

[jira] [Assigned] (SPARK-22618) RDD.unpersist can cause fatal exception when used with dynamic allocation

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22618: Assignee: (was: Apache Spark) > RDD.unpersist can cause fatal exception when used

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.x final

2017-11-28 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269357#comment-16269357 ] Wes McKinney commented on SPARK-19552: -- Is there anyone who might be able to help with upgrading

[jira] [Comment Edited] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Leigh Klotz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269354#comment-16269354 ] Leigh Klotz edited comment on SPARK-22373 at 11/28/17 7:51 PM: This

[jira] [Comment Edited] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Leigh Klotz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269354#comment-16269354 ] Leigh Klotz edited comment on SPARK-22373 at 11/28/17 7:51 PM: This

[jira] [Comment Edited] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Leigh Klotz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269354#comment-16269354 ] Leigh Klotz edited comment on SPARK-22373 at 11/28/17 7:50 PM: This

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Leigh Klotz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269354#comment-16269354 ] Leigh Klotz commented on SPARK-22373: -- This happens to me regularly enough using 2.1.1.20, Avro,

[jira] [Resolved] (SPARK-22515) Estimation relation size based on numRows * rowSize

2017-11-28 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22515. - Resolution: Fixed Assignee: Zhenhua Wang Fix Version/s: 2.3.0 > Estimation relation size

[jira] [Created] (SPARK-22636) row count not being set correctly (always 0) after Statement.executeUpdate()

2017-11-28 Thread Sergio Lob (JIRA)
Sergio Lob created SPARK-22636: -- Summary: row count not being set correctly (always 0) after Statement.executeUpdate() Key: SPARK-22636 URL: https://issues.apache.org/jira/browse/SPARK-22636 Project:

[jira] [Commented] (SPARK-22634) Update Bouncy castle dependency

2017-11-28 Thread Lior Regev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269308#comment-16269308 ] Lior Regev commented on SPARK-22634: Well, mainly there are 2 issues: # The included package in the

[jira] [Updated] (SPARK-19552) Upgrade Netty version to 4.1.x final

2017-11-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19552: - Summary: Upgrade Netty version to 4.1.x final (was: Upgrade Netty version to 4.1.8 final) >

[jira] [Updated] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-11-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19552: - Priority: Major (was: Minor) > Upgrade Netty version to 4.1.8 final >

[jira] [Reopened] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-11-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-19552: -- Netty is deprecating in https://github.com/netty/netty/issues/7439 Reopened this one to discuss

[jira] [Commented] (SPARK-22331) Make MLlib string params case-insensitive

2017-11-28 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269169#comment-16269169 ] yuhao yang commented on SPARK-22331: Thanks for the interests [~smurakozi]. I tried to support this

[jira] [Commented] (SPARK-15573) Backwards-compatible persistence for spark.ml

2017-11-28 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269140#comment-16269140 ] Ming Ma commented on SPARK-15573: - This is probably off topic as it isn't related to Spark version

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

2017-11-28 Thread Min Shen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269128#comment-16269128 ] Min Shen commented on SPARK-22373: -- Tried running the test application using 10 concurrent threads to

[jira] [Created] (SPARK-22635) FileNotFoundException again while reading ORC files containing special characters

2017-11-28 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-22635: --- Summary: FileNotFoundException again while reading ORC files containing special characters Key: SPARK-22635 URL: https://issues.apache.org/jira/browse/SPARK-22635

[jira] [Commented] (SPARK-21866) SPIP: Image support in Spark

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269026#comment-16269026 ] Apache Spark commented on SPARK-21866: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Comment Edited] (SPARK-22629) incorrect handling of calls to random in UDFs

2017-11-28 Thread Michael H (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268960#comment-16268960 ] Michael H edited comment on SPARK-22629 at 11/28/17 4:06 PM: - Hi Sean,

[jira] [Commented] (SPARK-22629) incorrect handling of calls to random in UDFs

2017-11-28 Thread Michael H (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268960#comment-16268960 ] Michael H commented on SPARK-22629: --- Hi Sean, Thanks for looking into this! I was worried this would

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-28 Thread Jakub Dubovsky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268892#comment-16268892 ] Jakub Dubovsky commented on SPARK-22585: I have created a PR where we can agree on a code to

[jira] [Commented] (SPARK-22585) Url encoding of jar path expected?

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1626#comment-1626 ] Apache Spark commented on SPARK-22585: -- User 'james64' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22585) Url encoding of jar path expected?

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22585: Assignee: (was: Apache Spark) > Url encoding of jar path expected? >

[jira] [Assigned] (SPARK-22585) Url encoding of jar path expected?

2017-11-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22585: Assignee: Apache Spark > Url encoding of jar path expected? >

[jira] [Updated] (SPARK-19106) Styling for the configuration docs is broken

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19106: -- Fix Version/s: (was: 2.2.1,2.1.1) 2.1.1 2.2.1 > Styling for

[jira] [Resolved] (SPARK-22627) Fix formatting of headers in configuration.html page

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22627. --- Resolution: Fixed Assignee: Felix Cheung Fix Version/s: 2.2.1 Hm, I wonder if the

[jira] [Updated] (SPARK-19106) Styling for the configuration docs is broken

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-19106: -- Fix Version/s: (was: 2.1.1) (was: 2.2.0) 2.2.1,2.1.1 >

[jira] [Commented] (SPARK-22630) Consolidate all configuration properties into one page

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268880#comment-16268880 ] Sean Owen commented on SPARK-22630: --- The flip-side is that you end up duplicating documentation.

[jira] [Commented] (SPARK-22629) incorrect handling of calls to random in UDFs

2017-11-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268875#comment-16268875 ] Sean Owen commented on SPARK-22629: --- This is expected behavior. If your source of data changes or is

[jira] [Commented] (SPARK-22627) Fix formatting of headers in configuration.html page

2017-11-28 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268869#comment-16268869 ] Marco Gaido commented on SPARK-22627: - [~srowen] the issue seems not present anymore on branch-2.2 (I

  1   2   >