[jira] [Commented] (SPARK-18451) Always set -XX:+HeapDumpOnOutOfMemoryError for Spark tests

[ https://issues.apache.org/jira/browse/SPARK-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223846#comment-16223846 ] Xin Lu commented on SPARK-18451: Pretty easy to do if we can get the jenkins job builder scripts changed

[jira] [Commented] (SPARK-22344) Prevent R CMD check from using /tmp

[ https://issues.apache.org/jira/browse/SPARK-22344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223844#comment-16223844 ] Shivaram Venkataraman commented on SPARK-22344: --- The ~/.cache is created by `install.spark`

[jira] [Commented] (SPARK-17788) RangePartitioner results in few very large tasks and many small to empty tasks

[ https://issues.apache.org/jira/browse/SPARK-17788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223837#comment-16223837 ] Apache Spark commented on SPARK-17788: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-17788) RangePartitioner results in few very large tasks and many small to empty tasks

[ https://issues.apache.org/jira/browse/SPARK-17788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17788: Assignee: (was: Apache Spark) > RangePartitioner results in few very large tasks and

[jira] [Assigned] (SPARK-17788) RangePartitioner results in few very large tasks and many small to empty tasks

[ https://issues.apache.org/jira/browse/SPARK-17788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17788: Assignee: Apache Spark > RangePartitioner results in few very large tasks and many small

[jira] [Commented] (SPARK-18394) Executing the same query twice in a row results in CodeGenerator cache misses

[ https://issues.apache.org/jira/browse/SPARK-18394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223835#comment-16223835 ] Wenchen Fan commented on SPARK-18394: - resolved by https://github.com/apache/spark/pull/18959 >

[jira] [Updated] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21033: Description: In `UnsafeInMemorySorter`, one record may take 32 bytes: 1 `long` for pointer, 1

[jira] [Updated] (SPARK-21033) fix the potential OOM in UnsafeExternalSorter

[ https://issues.apache.org/jira/browse/SPARK-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-21033: Description: ## What changes were proposed in this pull request? In `UnsafeInMemorySorter`, one

[jira] [Commented] (SPARK-22383) Generate code to directly get value of primitive type array from ColumnVector for table cache

[ https://issues.apache.org/jira/browse/SPARK-22383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223825#comment-16223825 ] Apache Spark commented on SPARK-22383: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22383) Generate code to directly get value of primitive type array from ColumnVector for table cache

[ https://issues.apache.org/jira/browse/SPARK-22383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22383: Assignee: (was: Apache Spark) > Generate code to directly get value of primitive type

[jira] [Assigned] (SPARK-22383) Generate code to directly get value of primitive type array from ColumnVector for table cache

[ https://issues.apache.org/jira/browse/SPARK-22383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22383: Assignee: Apache Spark > Generate code to directly get value of primitive type array from

[jira] [Created] (SPARK-22383) Generate code to directly get value of primitive type array from ColumnVector for table cache

Kazuaki Ishizaki created SPARK-22383: Summary: Generate code to directly get value of primitive type array from ColumnVector for table cache Key: SPARK-22383 URL:

[jira] [Commented] (SPARK-22277) Chi Square selector garbling Vector content.

[ https://issues.apache.org/jira/browse/SPARK-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223795#comment-16223795 ] Cheburakshu commented on SPARK-22277: - I actually meant VecotrIndexer or

[jira] [Updated] (SPARK-22382) Spark on mesos: doesn't support public IP setup for agent and master.

[ https://issues.apache.org/jira/browse/SPARK-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DUC LIEM NGUYEN updated SPARK-22382: Description: I've installed a system as followed: --mesos master private IP of 10.x.x.2 ,

[jira] [Updated] (SPARK-22382) Spark on mesos: doesn't support public IP setup for agent and master.

[ https://issues.apache.org/jira/browse/SPARK-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DUC LIEM NGUYEN updated SPARK-22382: Affects Version/s: (was: 2.1.1) 2.1.0 Description:

[jira] [Updated] (SPARK-22382) Spark on mesos: doesn't support public IP setup for agent and master.

[ https://issues.apache.org/jira/browse/SPARK-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DUC LIEM NGUYEN updated SPARK-22382: Description: I've installed a system as followed: --mesos master private IP of 10.x.x.2 ,

[jira] [Updated] (SPARK-22382) Spark on mesos: doesn't support public IP setup for agent and master.

[ https://issues.apache.org/jira/browse/SPARK-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DUC LIEM NGUYEN updated SPARK-22382: Description: I've installed a system as followed: --mesos master private IP of 10.x.x.2 ,

[jira] [Updated] (SPARK-22382) Spark on mesos: doesn't support public IP setup for agent and master.

[ https://issues.apache.org/jira/browse/SPARK-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] DUC LIEM NGUYEN updated SPARK-22382: Description: I've installed a system as followed: --mesos master private IP of 10.x.x.2 ,

[jira] [Created] (SPARK-22382) Spark on mesos: doesn't support public IP setup for agent and master.

DUC LIEM NGUYEN created SPARK-22382: --- Summary: Spark on mesos: doesn't support public IP setup for agent and master. Key: SPARK-22382 URL: https://issues.apache.org/jira/browse/SPARK-22382

[jira] [Assigned] (SPARK-22381) Add StringParam that supports valid options

[ https://issues.apache.org/jira/browse/SPARK-22381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22381: Assignee: (was: Apache Spark) > Add StringParam that supports valid options >

[jira] [Commented] (SPARK-22381) Add StringParam that supports valid options

[ https://issues.apache.org/jira/browse/SPARK-22381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223714#comment-16223714 ] Apache Spark commented on SPARK-22381: -- User 'hhbyyh' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22381) Add StringParam that supports valid options

[ https://issues.apache.org/jira/browse/SPARK-22381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22381: Assignee: Apache Spark > Add StringParam that supports valid options >

[jira] [Created] (SPARK-22381) Add StringParam that supports valid options

yuhao yang created SPARK-22381: -- Summary: Add StringParam that supports valid options Key: SPARK-22381 URL: https://issues.apache.org/jira/browse/SPARK-22381 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-22380) Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0

[ https://issues.apache.org/jira/browse/SPARK-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223700#comment-16223700 ] Maziyar PANAHI commented on SPARK-22380: Hi Sean, Tanks for your reply. In fact I am more

[jira] [Comment Edited] (SPARK-22055) Port release scripts

[ https://issues.apache.org/jira/browse/SPARK-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223229#comment-16223229 ] Xin Lu edited comment on SPARK-22055 at 10/28/17 6:30 PM: -- [~holdenk]

[jira] [Commented] (SPARK-22380) Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0

[ https://issues.apache.org/jira/browse/SPARK-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223688#comment-16223688 ] Sean Owen commented on SPARK-22380: --- The problem is that the tightly-coupled Hadoop dependencies depend

[jira] [Updated] (SPARK-22378) Redundant nullcheck is generated for extracting value in complex types

[ https://issues.apache.org/jira/browse/SPARK-22378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-22378: - Description: Redundant null check is generated in the code for extracting an element

[jira] [Assigned] (SPARK-22370) Config values should be captured in Driver.

[ https://issues.apache.org/jira/browse/SPARK-22370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22370: --- Assignee: Takuya Ueshin > Config values should be captured in Driver. >

[jira] [Resolved] (SPARK-22370) Config values should be captured in Driver.

[ https://issues.apache.org/jira/browse/SPARK-22370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22370. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19587

[jira] [Commented] (SPARK-21158) SparkSQL function SparkSession.Catalog.ListTables() does not handle spark setting for case-sensitivity

[ https://issues.apache.org/jira/browse/SPARK-21158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223667#comment-16223667 ] Wenchen Fan commented on SPARK-21158: - I think this is a reasonable feature request, i.e. making

[jira] [Assigned] (SPARK-22378) Eliminate redundant nullcheck in generated code for extracting value in complex type

[ https://issues.apache.org/jira/browse/SPARK-22378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22378: Assignee: Apache Spark > Eliminate redundant nullcheck in generated code for extracting

[jira] [Assigned] (SPARK-22378) Eliminate redundant nullcheck in generated code for extracting value in complex type

[ https://issues.apache.org/jira/browse/SPARK-22378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22378: Assignee: (was: Apache Spark) > Eliminate redundant nullcheck in generated code for

[jira] [Commented] (SPARK-22378) Eliminate redundant nullcheck in generated code for extracting value in complex type

[ https://issues.apache.org/jira/browse/SPARK-22378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223651#comment-16223651 ] Apache Spark commented on SPARK-22378: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22375) Test script can fail if eggs are installed by setup.py during test process

[ https://issues.apache.org/jira/browse/SPARK-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22375: Assignee: Apache Spark > Test script can fail if eggs are installed by setup.py during

[jira] [Assigned] (SPARK-22375) Test script can fail if eggs are installed by setup.py during test process

[ https://issues.apache.org/jira/browse/SPARK-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22375: Assignee: (was: Apache Spark) > Test script can fail if eggs are installed by

[jira] [Commented] (SPARK-22375) Test script can fail if eggs are installed by setup.py during test process

[ https://issues.apache.org/jira/browse/SPARK-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223634#comment-16223634 ] Apache Spark commented on SPARK-22375: -- User 'xynny' has created a pull request for this issue:

[jira] [Created] (SPARK-22380) Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0

Maziyar PANAHI created SPARK-22380: -- Summary: Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0 Key: SPARK-22380 URL: https://issues.apache.org/jira/browse/SPARK-22380 Project:

[jira] [Assigned] (SPARK-22369) PySpark: Document methods of spark.catalog interface

[ https://issues.apache.org/jira/browse/SPARK-22369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22369: Assignee: Apache Spark > PySpark: Document methods of spark.catalog interface >

[jira] [Assigned] (SPARK-22369) PySpark: Document methods of spark.catalog interface

[ https://issues.apache.org/jira/browse/SPARK-22369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22369: Assignee: (was: Apache Spark) > PySpark: Document methods of spark.catalog interface

[jira] [Commented] (SPARK-22369) PySpark: Document methods of spark.catalog interface

[ https://issues.apache.org/jira/browse/SPARK-22369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223591#comment-16223591 ] Apache Spark commented on SPARK-22369: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-22379) Reduce duplication setUpClass and tearDownClass in PySpark SQL tests

[ https://issues.apache.org/jira/browse/SPARK-22379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22379: Assignee: Apache Spark > Reduce duplication setUpClass and tearDownClass in PySpark SQL

[jira] [Commented] (SPARK-22379) Reduce duplication setUpClass and tearDownClass in PySpark SQL tests

[ https://issues.apache.org/jira/browse/SPARK-22379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223568#comment-16223568 ] Apache Spark commented on SPARK-22379: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-22379) Reduce duplication setUpClass and tearDownClass in PySpark SQL tests

[ https://issues.apache.org/jira/browse/SPARK-22379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22379: Assignee: (was: Apache Spark) > Reduce duplication setUpClass and tearDownClass in

[jira] [Created] (SPARK-22379) Reduce duplication setUpClass and tearDownClass in PySpark SQL tests

Hyukjin Kwon created SPARK-22379: Summary: Reduce duplication setUpClass and tearDownClass in PySpark SQL tests Key: SPARK-22379 URL: https://issues.apache.org/jira/browse/SPARK-22379 Project: Spark

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223564#comment-16223564 ] Hyukjin Kwon commented on SPARK-22240: -- I am sorry, I have been super busy these for few days. So,

[jira] [Resolved] (SPARK-22335) Union for DataSet uses column order instead of types for union

[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-22335. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19570

[jira] [Assigned] (SPARK-22335) Union for DataSet uses column order instead of types for union

[ https://issues.apache.org/jira/browse/SPARK-22335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-22335: Assignee: Liang-Chi Hsieh > Union for DataSet uses column order instead of types for

[jira] [Commented] (SPARK-22351) Support user-created custom Encoders for Datasets

[ https://issues.apache.org/jira/browse/SPARK-22351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223470#comment-16223470 ] Hyukjin Kwon commented on SPARK-22351: -- {quote} While this was possible in Spark 1.6 it's not longer

[jira] [Commented] (SPARK-22240) S3 CSV number of partitions incorrectly computed

[ https://issues.apache.org/jira/browse/SPARK-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223400#comment-16223400 ] Steve Loughran commented on SPARK-22240: so this partition calculation problem is independent of

[jira] [Created] (SPARK-22378) Eliminate redundant nullcheck in generated code for extracting value in complex type

Kazuaki Ishizaki created SPARK-22378: Summary: Eliminate redundant nullcheck in generated code for extracting value in complex type Key: SPARK-22378 URL: https://issues.apache.org/jira/browse/SPARK-22378

[jira] [Commented] (SPARK-22376) run-tests.py fails at exec-sbt if run with Python 3

[ https://issues.apache.org/jira/browse/SPARK-22376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223355#comment-16223355 ] Sean Owen commented on SPARK-22376: --- If the change is still compatible with Python 2, yes go ahead. >

[jira] [Commented] (SPARK-22373) Intermittent NullPointerException in org.codehaus.janino.IClass.isAssignableFrom

[ https://issues.apache.org/jira/browse/SPARK-22373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223348#comment-16223348 ] Sean Owen commented on SPARK-22373: --- It looks like a Janino problem, so not sure if it can be fixed in

[jira] [Commented] (SPARK-22375) Test script can fail if eggs are installed by setup.py during test process

[ https://issues.apache.org/jira/browse/SPARK-22375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223345#comment-16223345 ] Sean Owen commented on SPARK-22375: --- Certainly, please open a pull request. > Test script can fail if

[jira] [Resolved] (SPARK-21635) ACOS(2) and ASIN(2) should be null

[ https://issues.apache.org/jira/browse/SPARK-21635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21635. --- Resolution: Won't Fix > ACOS(2) and ASIN(2) should be null > -- > >

[jira] [Resolved] (SPARK-13085) Add scalastyle command used in build testing

[ https://issues.apache.org/jira/browse/SPARK-13085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-13085. --- Resolution: Duplicate > Add scalastyle command used in build testing >

[jira] [Resolved] (SPARK-20731) Add ability to change or omit .csv file extension in CSV Data Source

[ https://issues.apache.org/jira/browse/SPARK-20731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20731. -- Resolution: Won't Fix > Add ability to change or omit .csv file extension in CSV Data Source >

[jira] [Resolved] (SPARK-21619) Fail the execution of canonicalized plans explicitly

[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21619. - Resolution: Fixed Fix Version/s: 2.3.0 > Fail the execution of canonicalized plans explicitly >

[jira] [Resolved] (SPARK-22333) ColumnReference should get higher priority than timeFunctionCall(CURRENT_DATE, CURRENT_TIMESTAMP)

[ https://issues.apache.org/jira/browse/SPARK-22333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22333. - Resolution: Fixed Assignee: Feng Zhu Fix Version/s: 2.3.0 > ColumnReference should get