[jira] [Commented] (SPARK-27541) Refresh class definitions for jars added via addJar()

2019-05-01 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831408#comment-16831408 ] Chakravarthi commented on SPARK-27541: -- [~navedalam] I would like to look into this  > Refresh

[jira] [Commented] (SPARK-27543) Support getRequiredJars and getRequiredFiles APIs for Hive UDFs

2019-05-01 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831407#comment-16831407 ] Chakravarthi commented on SPARK-27543: -- [~makagonov]  SparkContext.addJar could be used to include

[jira] [Assigned] (SPARK-27620) Update jetty to 9.4.18.v20190429

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27620: Assignee: (was: Apache Spark) > Update jetty to 9.4.18.v20190429 >

[jira] [Assigned] (SPARK-27620) Update jetty to 9.4.18.v20190429

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27620: Assignee: Apache Spark > Update jetty to 9.4.18.v20190429 >

[jira] [Created] (SPARK-27620) Update jetty to 9.4.18.v20190429

2019-05-01 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27620: --- Summary: Update jetty to 9.4.18.v20190429 Key: SPARK-27620 URL: https://issues.apache.org/jira/browse/SPARK-27620 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-01 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27619: -- Summary: MapType should be prohibited in hash expressions Key: SPARK-27619 URL: https://issues.apache.org/jira/browse/SPARK-27619 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27619: --- Description: Spark currently allows MapType expressions to be used as input to hash expressions,

[jira] [Updated] (SPARK-27619) MapType should be prohibited in hash expressions

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-27619: --- Description: Spark currently allows MapType expressions to be used as input to hash expressions,

[jira] [Assigned] (SPARK-26921) Fix CRAN hack as soon as Arrow is available on CRAN

2019-05-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26921: Assignee: Hyukjin Kwon > Fix CRAN hack as soon as Arrow is available on CRAN >

[jira] [Resolved] (SPARK-26921) Fix CRAN hack as soon as Arrow is available on CRAN

2019-05-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26921. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24506

[jira] [Assigned] (SPARK-24708) Document the default spark url of master in standalone is "spark://localhost:7070"

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24708: Assignee: (was: Apache Spark) > Document the default spark url of master in

[jira] [Assigned] (SPARK-24708) Document the default spark url of master in standalone is "spark://localhost:7070"

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24708: Assignee: Apache Spark > Document the default spark url of master in standalone is >

[jira] [Commented] (SPARK-24935) Problem with Executing Hive UDF's from Spark 2.2 Onwards

2019-05-01 Thread Reza Safi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831250#comment-16831250 ] Reza Safi commented on SPARK-24935: --- Is there any reason that this wasn't merged into 2.3 line? >

[jira] [Updated] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith Chacko updated SPARK-27617: -- Affects Version/s: 2.0.0 > Not able to specify LOCATION for internal table >

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831183#comment-16831183 ] Sujith Chacko edited comment on SPARK-27617 at 5/1/19 6:49 PM: --- cc

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831183#comment-16831183 ] Sujith Chacko edited comment on SPARK-27617 at 5/1/19 6:48 PM: --- cc

[jira] [Comment Edited] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831183#comment-16831183 ] Sujith Chacko edited comment on SPARK-27617 at 5/1/19 6:48 PM: --- cc

[jira] [Commented] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831183#comment-16831183 ] Sujith Chacko commented on SPARK-27617: --- cc [~dongjoon]    cc [~cloud_fan] I observed above

[jira] [Updated] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith Chacko updated SPARK-27617: -- Affects Version/s: 3.0.0 > Not able to specify LOCATION for internal table >

[jira] [Resolved] (SPARK-27618) Unnecessary access to externalCatalog

2019-05-01 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-27618. - Resolution: Fixed > Unnecessary access to externalCatalog > - > >

[jira] [Assigned] (SPARK-27618) Unnecessary access to externalCatalog

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27618: Assignee: Apache Spark > Unnecessary access to externalCatalog >

[jira] [Assigned] (SPARK-27618) Unnecessary access to externalCatalog

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27618: Assignee: (was: Apache Spark) > Unnecessary access to externalCatalog >

[jira] [Updated] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith Chacko updated SPARK-27617: -- Description: In spark whenever user specifies location uri in create table without external

[jira] [Commented] (SPARK-27618) Unnecessary access to externalCatalog

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831174#comment-16831174 ] Apache Spark commented on SPARK-27618: -- User 'OCaballero' has created a pull request for this

[jira] [Created] (SPARK-27618) Unnecessary access to externalCatalog

2019-05-01 Thread Xiao Li (JIRA)
Xiao Li created SPARK-27618: --- Summary: Unnecessary access to externalCatalog Key: SPARK-27618 URL: https://issues.apache.org/jira/browse/SPARK-27618 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27617) Not able to specify LOCATION for internal table

2019-05-01 Thread Sujith Chacko (JIRA)
Sujith Chacko created SPARK-27617: - Summary: Not able to specify LOCATION for internal table Key: SPARK-27617 URL: https://issues.apache.org/jira/browse/SPARK-27617 Project: Spark Issue

[jira] [Assigned] (SPARK-27557) Add copybutton to spark Python API docs for easier copying of code-blocks

2019-05-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27557: - Assignee: Sangram G > Add copybutton to spark Python API docs for easier copying of

[jira] [Resolved] (SPARK-27557) Add copybutton to spark Python API docs for easier copying of code-blocks

2019-05-01 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27557. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24456

[jira] [Commented] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-01 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831098#comment-16831098 ] Bryan Cutler commented on SPARK-27612: -- Also cc [~viirya] [~hyukjin.kwon], this is a little

[jira] [Commented] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831097#comment-16831097 ] Marco Gaido commented on SPARK-27612: - I don't have a python3 env, sorry... > Creating a DataFrame

[jira] [Updated] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-01 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27612: - Description: This seems to only affect Python 3. When creating a DataFrame with type

[jira] [Commented] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-01 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831092#comment-16831092 ] Bryan Cutler commented on SPARK-27612: -- Thanks [~mgaido], it seems like the problem does not happen

[jira] [Assigned] (SPARK-27611) Redundant javax.activation dependencies in the Maven build

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27611: Assignee: Apache Spark (was: Cheng Lian) > Redundant javax.activation dependencies in

[jira] [Assigned] (SPARK-27611) Redundant javax.activation dependencies in the Maven build

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27611: Assignee: Cheng Lian (was: Apache Spark) > Redundant javax.activation dependencies in

[jira] [Assigned] (SPARK-27611) Redundant javax.activation dependencies in the Maven build

2019-05-01 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian reassigned SPARK-27611: -- Assignee: Cheng Lian > Redundant javax.activation dependencies in the Maven build >

[jira] [Assigned] (SPARK-26921) Fix CRAN hack as soon as Arrow is available on CRAN

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26921: Assignee: (was: Apache Spark) > Fix CRAN hack as soon as Arrow is available on CRAN

[jira] [Assigned] (SPARK-26921) Fix CRAN hack as soon as Arrow is available on CRAN

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26921: Assignee: Apache Spark > Fix CRAN hack as soon as Arrow is available on CRAN >

[jira] [Assigned] (SPARK-27607) Improve performance of Row.toString()

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27607: Assignee: (was: Apache Spark) > Improve performance of Row.toString() >

[jira] [Assigned] (SPARK-27607) Improve performance of Row.toString()

2019-05-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27607: Assignee: Apache Spark > Improve performance of Row.toString() >

[jira] [Commented] (SPARK-17637) Packed scheduling for Spark tasks across executors

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831014#comment-16831014 ] Josh Rosen commented on SPARK-17637: I think this old feature suggestion is still very relevant and

[jira] [Commented] (SPARK-27607) Improve performance of Row.toString()

2019-05-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830997#comment-16830997 ] Josh Rosen commented on SPARK-27607: Feel free to take this. > Improve performance of

[jira] [Created] (SPARK-27616) Standalone cluster management user resource allocation

2019-05-01 Thread weDataSphere (JIRA)
weDataSphere created SPARK-27616: Summary: Standalone cluster management user resource allocation Key: SPARK-27616 URL: https://issues.apache.org/jira/browse/SPARK-27616 Project: Spark Issue

[jira] [Created] (SPARK-27615) Merge small files in the read stage

2019-05-01 Thread weDataSphere (JIRA)
weDataSphere created SPARK-27615: Summary: Merge small files in the read stage Key: SPARK-27615 URL: https://issues.apache.org/jira/browse/SPARK-27615 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-27614) Executor shuffle fetch hang

2019-05-01 Thread weDataSphere (JIRA)
weDataSphere created SPARK-27614: Summary: Executor shuffle fetch hang Key: SPARK-27614 URL: https://issues.apache.org/jira/browse/SPARK-27614 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27332) Filter Pushdown duplicates expensive ScalarSubquery (discarding result)

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830947#comment-16830947 ] Marco Gaido commented on SPARK-27332: - [~dzklip] actually Spark was not using the ScalarSubquery

[jira] [Commented] (SPARK-27612) Creating a DataFrame in PySpark with ArrayType produces some Rows with Arrays of None

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830945#comment-16830945 ] Marco Gaido commented on SPARK-27612: - I am not able to reproduce... {code} __ / __/__ ___

[jira] [Commented] (SPARK-27607) Improve performance of Row.toString()

2019-05-01 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830933#comment-16830933 ] Marco Gaido commented on SPARK-27607: - Hi [~joshrosen], are you working on it? If not I can take it.

[jira] [Commented] (SPARK-27597) RuntimeConfig should be serializable

2019-05-01 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830911#comment-16830911 ] Liang-Chi Hsieh commented on SPARK-27597: - I see. Please follow [~hyukjin.kwon]'s suggestion if

[jira] [Commented] (SPARK-27593) CSV Parser returns 2 DataFrame - Valid and Malformed DFs

2019-05-01 Thread Ladislav Jech (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830894#comment-16830894 ] Ladislav Jech commented on SPARK-27593: --- But you don't know if NULL is actually coming from data