[jira] [Commented] (SPARK-22883) ML test for StructuredStreaming: spark.ml.feature, A-M

2018-03-01 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383242#comment-16383242 ] Joseph K. Bradley commented on SPARK-22883: --- Merged part 1 of 2 to master and branch-2.3 > ML

[jira] [Commented] (SPARK-21209) Implement Incremental PCA algorithm for ML

2018-03-01 Thread Sandeep Kumar Choudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383228#comment-16383228 ] Sandeep Kumar Choudhary commented on SPARK-21209: - Hi Ben St. Clair. I have implemented

[jira] [Assigned] (SPARK-22883) ML test for StructuredStreaming: spark.ml.feature, A-M

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22883: Assignee: Joseph K. Bradley (was: Apache Spark) > ML test for StructuredStreaming:

[jira] [Assigned] (SPARK-22883) ML test for StructuredStreaming: spark.ml.feature, A-M

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22883: Assignee: Apache Spark (was: Joseph K. Bradley) > ML test for StructuredStreaming:

[jira] [Reopened] (SPARK-22883) ML test for StructuredStreaming: spark.ml.feature, A-M

2018-03-01 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reopened SPARK-22883: --- > ML test for StructuredStreaming: spark.ml.feature, A-M >

[jira] [Updated] (SPARK-22883) ML test for StructuredStreaming: spark.ml.feature, A-M

2018-03-01 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-22883: -- Fix Version/s: (was: 2.4.0) > ML test for StructuredStreaming: spark.ml.feature,

[jira] [Assigned] (SPARK-22883) ML test for StructuredStreaming: spark.ml.feature, A-M

2018-03-01 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-22883: - Assignee: Joseph K. Bradley > ML test for StructuredStreaming:

[jira] [Resolved] (SPARK-22883) ML test for StructuredStreaming: spark.ml.feature, A-M

2018-03-01 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-22883. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20111

[jira] [Commented] (SPARK-23434) Spark should not warn `metadata directory` for a HDFS file path

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383157#comment-16383157 ] Apache Spark commented on SPARK-23434: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-23457) Register task completion listeners first for ParquetFileFormat

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383156#comment-16383156 ] Apache Spark commented on SPARK-23457: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-23434) Spark should not warn `metadata directory` for a HDFS file path

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383153#comment-16383153 ] Apache Spark commented on SPARK-23434: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-23554) Hive's textinputformat.record.delimiter equivalent in Spark

2018-03-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383146#comment-16383146 ] Hyukjin Kwon commented on SPARK-23554: -- I think it's a duplicate of SPARK-21289? > Hive's

[jira] [Assigned] (SPARK-23563) make the size fo cache in CodeGenerator configable

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23563: Assignee: (was: Apache Spark) > make the size fo cache in CodeGenerator configable >

[jira] [Assigned] (SPARK-23563) make the size fo cache in CodeGenerator configable

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23563: Assignee: Apache Spark > make the size fo cache in CodeGenerator configable >

[jira] [Commented] (SPARK-23563) make the size fo cache in CodeGenerator configable

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383129#comment-16383129 ] Apache Spark commented on SPARK-23563: -- User 'passionke' has created a pull request for this issue:

[jira] [Updated] (SPARK-23542) The exists action shoule be further optimized in logical plan

2018-03-01 Thread KaiXinXIaoLei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXinXIaoLei updated SPARK-23542: -- Description: The optimized logical plan of query '*select * from tt1 where exists (select * 

[jira] [Updated] (SPARK-23542) The exists action shoule be further optimized in logical plan

2018-03-01 Thread KaiXinXIaoLei (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXinXIaoLei updated SPARK-23542: -- Summary: The exists action shoule be further optimized in logical plan (was: The `where

[jira] [Commented] (SPARK-23563) make the size fo cache in CodeGenerator configable

2018-03-01 Thread kejiqing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383005#comment-16383005 ] kejiqing commented on SPARK-23563: -- a long term spark sql task, the meta space in driver will increase

[jira] [Commented] (SPARK-23552) Dataset.withColumn does not allow overriding of a struct field

2018-03-01 Thread David Capwell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383002#comment-16383002 ] David Capwell commented on SPARK-23552: --- This appears to also be a issue with drop. >

[jira] [Updated] (SPARK-23563) make the size fo cache in CodeGenerator configable

2018-03-01 Thread kejiqing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kejiqing updated SPARK-23563: - Summary: make the size fo cache in CodeGenerator configable (was: make size fo cache in CodeGenerator

[jira] [Created] (SPARK-23563) make size fo cache in CodeGenerator configable

2018-03-01 Thread kejiqing (JIRA)
kejiqing created SPARK-23563: Summary: make size fo cache in CodeGenerator configable Key: SPARK-23563 URL: https://issues.apache.org/jira/browse/SPARK-23563 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-23551. Resolution: Fixed Fix Version/s: 2.3.1 2.4.0 Issue resolved by

[jira] [Created] (SPARK-23562) RFormula handleInvalid should handle invalid values in non-string columns.

2018-03-01 Thread Bago Amirbekian (JIRA)
Bago Amirbekian created SPARK-23562: --- Summary: RFormula handleInvalid should handle invalid values in non-string columns. Key: SPARK-23562 URL: https://issues.apache.org/jira/browse/SPARK-23562

[jira] [Assigned] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-23551: -- Assignee: Dongjoon Hyun > Exclude `hadoop-mapreduce-client-core` dependency from

[jira] [Commented] (SPARK-19181) SparkListenerSuite.local metrics fails when average executorDeserializeTime is too short.

2018-03-01 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382963#comment-16382963 ] Marcelo Vanzin commented on SPARK-19181: Another failure (after quite some time):

[jira] [Commented] (SPARK-23534) Spark run on Hadoop 3.0.0

2018-03-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382932#comment-16382932 ] Saisai Shao commented on SPARK-23534: - One issue is Hive 1.2.1.spark2 rejects Hadoop 3 (SPARK-18673)

[jira] [Commented] (SPARK-18673) Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

2018-03-01 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382929#comment-16382929 ] Saisai Shao commented on SPARK-18673: - Spark itself uses it own hive (1.2.1.spark2), I think we need

[jira] [Updated] (SPARK-23498) Accuracy problem in comparison with string and integer

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23498: -- Target Version/s: (was: 2.3.1) > Accuracy problem in comparison with string and integer >

[jira] [Comment Edited] (SPARK-23498) Accuracy problem in comparison with string and integer

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382912#comment-16382912 ] Dongjoon Hyun edited comment on SPARK-23498 at 3/2/18 12:12 AM:

[jira] [Assigned] (SPARK-23559) add epoch ID to data writer factory

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23559: Assignee: (was: Apache Spark) > add epoch ID to data writer factory >

[jira] [Assigned] (SPARK-23559) add epoch ID to data writer factory

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23559: Assignee: Apache Spark > add epoch ID to data writer factory >

[jira] [Commented] (SPARK-23498) Accuracy problem in comparison with string and integer

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382912#comment-16382912 ] Dongjoon Hyun commented on SPARK-23498: --- [~KevinZwx]. Did you see HIVE-17186? Hive doesn't give you

[jira] [Commented] (SPARK-23559) add epoch ID to data writer factory

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382911#comment-16382911 ] Apache Spark commented on SPARK-23559: -- User 'jose-torres' has created a pull request for this

[jira] [Created] (SPARK-23561) make StreamWriter not a DataSourceWriter subclass

2018-03-01 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23561: --- Summary: make StreamWriter not a DataSourceWriter subclass Key: SPARK-23561 URL: https://issues.apache.org/jira/browse/SPARK-23561 Project: Spark Issue Type:

[jira] [Created] (SPARK-23560) A joinWith followed by groupBy requires extra shuffle

2018-03-01 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-23560: - Summary: A joinWith followed by groupBy requires extra shuffle Key: SPARK-23560 URL: https://issues.apache.org/jira/browse/SPARK-23560 Project: Spark

[jira] [Commented] (SPARK-18630) PySpark ML memory leak

2018-03-01 Thread yogesh garg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382886#comment-16382886 ] yogesh garg commented on SPARK-18630: - After some discussion, I think it makes sense to move just the

[jira] [Created] (SPARK-23559) add epoch ID to data writer factory

2018-03-01 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23559: --- Summary: add epoch ID to data writer factory Key: SPARK-23559 URL: https://issues.apache.org/jira/browse/SPARK-23559 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-23558) clean up StreamWriter factory lifecycle

2018-03-01 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23558: --- Summary: clean up StreamWriter factory lifecycle Key: SPARK-23558 URL: https://issues.apache.org/jira/browse/SPARK-23558 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-23557) design doc for read side

2018-03-01 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23557: --- Summary: design doc for read side Key: SPARK-23557 URL: https://issues.apache.org/jira/browse/SPARK-23557 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-23556) design doc for write side

2018-03-01 Thread Jose Torres (JIRA)
Jose Torres created SPARK-23556: --- Summary: design doc for write side Key: SPARK-23556 URL: https://issues.apache.org/jira/browse/SPARK-23556 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-18844) Add more binary classification metrics to BinaryClassificationMetrics

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382719#comment-16382719 ] Apache Spark commented on SPARK-18844: -- User 'sandecho' has created a pull request for this issue:

[jira] [Commented] (SPARK-23555) Add BinaryType support for Arrow in PySpark

2018-03-01 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382715#comment-16382715 ] Bryan Cutler commented on SPARK-23555: -- I'm working on it > Add BinaryType support for Arrow in

[jira] [Created] (SPARK-23555) Add BinaryType support for Arrow in PySpark

2018-03-01 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-23555: Summary: Add BinaryType support for Arrow in PySpark Key: SPARK-23555 URL: https://issues.apache.org/jira/browse/SPARK-23555 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-23520) Add support for MapType fields in JSON schema inference

2018-03-01 Thread David Courtinot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382649#comment-16382649 ] David Courtinot edited comment on SPARK-23520 at 3/1/18 9:26 PM: - Good

[jira] [Comment Edited] (SPARK-23520) Add support for MapType fields in JSON schema inference

2018-03-01 Thread David Courtinot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382649#comment-16382649 ] David Courtinot edited comment on SPARK-23520 at 3/1/18 9:24 PM: - Good

[jira] [Comment Edited] (SPARK-23520) Add support for MapType fields in JSON schema inference

2018-03-01 Thread David Courtinot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382649#comment-16382649 ] David Courtinot edited comment on SPARK-23520 at 3/1/18 9:22 PM: - Good

[jira] [Comment Edited] (SPARK-23520) Add support for MapType fields in JSON schema inference

2018-03-01 Thread David Courtinot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382649#comment-16382649 ] David Courtinot edited comment on SPARK-23520 at 3/1/18 9:22 PM: - Good

[jira] [Commented] (SPARK-23520) Add support for MapType fields in JSON schema inference

2018-03-01 Thread David Courtinot (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382649#comment-16382649 ] David Courtinot commented on SPARK-23520: - Good catch. I think the issue is very similar indeed.

[jira] [Created] (SPARK-23554) Hive's textinputformat.record.delimiter equivalent in Spark

2018-03-01 Thread Ruslan Dautkhanov (JIRA)
Ruslan Dautkhanov created SPARK-23554: - Summary: Hive's textinputformat.record.delimiter equivalent in Spark Key: SPARK-23554 URL: https://issues.apache.org/jira/browse/SPARK-23554 Project: Spark

[jira] [Commented] (SPARK-21209) Implement Incremental PCA algorithm for ML

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382537#comment-16382537 ] Apache Spark commented on SPARK-21209: -- User 'sandecho' has created a pull request for this issue:

[jira] [Updated] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23551: -- Priority: Minor (was: Major) > Exclude `hadoop-mapreduce-client-core` dependency from

[jira] [Assigned] (SPARK-21209) Implement Incremental PCA algorithm for ML

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21209: Assignee: (was: Apache Spark) > Implement Incremental PCA algorithm for ML >

[jira] [Commented] (SPARK-21209) Implement Incremental PCA algorithm for ML

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382501#comment-16382501 ] Apache Spark commented on SPARK-21209: -- User 'sandecho' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21209) Implement Incremental PCA algorithm for ML

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21209: Assignee: Apache Spark > Implement Incremental PCA algorithm for ML >

[jira] [Assigned] (SPARK-23550) Cleanup unused / redundant methods in Utils object

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23550: Assignee: Apache Spark > Cleanup unused / redundant methods in Utils object >

[jira] [Commented] (SPARK-23550) Cleanup unused / redundant methods in Utils object

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382478#comment-16382478 ] Apache Spark commented on SPARK-23550: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23550) Cleanup unused / redundant methods in Utils object

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23550: Assignee: (was: Apache Spark) > Cleanup unused / redundant methods in Utils object >

[jira] [Assigned] (SPARK-23553) Tests should not assume the default value of `spark.sql.sources.default`

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23553: Assignee: (was: Apache Spark) > Tests should not assume the default value of

[jira] [Commented] (SPARK-23553) Tests should not assume the default value of `spark.sql.sources.default`

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382448#comment-16382448 ] Apache Spark commented on SPARK-23553: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-23553) Tests should not assume the default value of `spark.sql.sources.default`

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23553: Assignee: Apache Spark > Tests should not assume the default value of

[jira] [Updated] (SPARK-23553) Tests should not assume the default value of `spark.sql.sources.default`

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23553: -- Description: Currently, some tests have an assumption that

[jira] [Updated] (SPARK-23553) Tests should not assume the default value of `spark.sql.sources.default`

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23553: -- Description: Currently, some tests have an assumption that

[jira] [Created] (SPARK-23553) Tests should not assume the default value of `spark.sql.sources.default`

2018-03-01 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-23553: - Summary: Tests should not assume the default value of `spark.sql.sources.default` Key: SPARK-23553 URL: https://issues.apache.org/jira/browse/SPARK-23553 Project:

[jira] [Updated] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23551: -- Description: This issue aims to prevent `orc-mapreduce` dependency from making IDEs and maven

[jira] [Created] (SPARK-23552) Dataset.withColumn does not allow overriding of a struct field

2018-03-01 Thread David Capwell (JIRA)
David Capwell created SPARK-23552: - Summary: Dataset.withColumn does not allow overriding of a struct field Key: SPARK-23552 URL: https://issues.apache.org/jira/browse/SPARK-23552 Project: Spark

[jira] [Commented] (SPARK-23543) Automatic Module creation fails in Java 9

2018-03-01 Thread Brian D Chambers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382399#comment-16382399 ] Brian D Chambers commented on SPARK-23543: -- Note  this is not an implementation of java 9

[jira] [Updated] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-23551: -- Description: This issue aims to prevent `orc-mapreduce` dependency makes IDEs and maven

[jira] [Assigned] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23551: Assignee: Apache Spark > Exclude `hadoop-mapreduce-client-core` dependency from

[jira] [Assigned] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23551: Assignee: (was: Apache Spark) > Exclude `hadoop-mapreduce-client-core` dependency

[jira] [Commented] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382392#comment-16382392 ] Apache Spark commented on SPARK-23551: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Resolved] (SPARK-23471) RandomForestClassificationModel save() - incorrect metadata

2018-03-01 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-23471. --- Resolution: Cannot Reproduce I'll close this for now, but please say if it's

[jira] [Created] (SPARK-23551) Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce`

2018-03-01 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-23551: - Summary: Exclude `hadoop-mapreduce-client-core` dependency from `orc-mapreduce` Key: SPARK-23551 URL: https://issues.apache.org/jira/browse/SPARK-23551 Project:

[jira] [Created] (SPARK-23550) Cleanup unused / redundant methods in Utils object

2018-03-01 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-23550: -- Summary: Cleanup unused / redundant methods in Utils object Key: SPARK-23550 URL: https://issues.apache.org/jira/browse/SPARK-23550 Project: Spark Issue

[jira] [Issue Comment Deleted] (SPARK-10908) ClassCastException in HadoopRDD.getJobConf

2018-03-01 Thread Riccardo Vincelli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riccardo Vincelli updated SPARK-10908: -- Comment: was deleted (was: Hi, I am encountering this as well, running two local Spark

[jira] [Comment Edited] (SPARK-10908) ClassCastException in HadoopRDD.getJobConf

2018-03-01 Thread Riccardo Vincelli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382241#comment-16382241 ] Riccardo Vincelli edited comment on SPARK-10908 at 3/1/18 4:19 PM: --- Hi,

[jira] [Comment Edited] (SPARK-10908) ClassCastException in HadoopRDD.getJobConf

2018-03-01 Thread Riccardo Vincelli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382241#comment-16382241 ] Riccardo Vincelli edited comment on SPARK-10908 at 3/1/18 4:19 PM: --- Hi,

[jira] [Commented] (SPARK-10908) ClassCastException in HadoopRDD.getJobConf

2018-03-01 Thread Riccardo Vincelli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382241#comment-16382241 ] Riccardo Vincelli commented on SPARK-10908: --- Hi, I am encountering this as well, running two

[jira] [Assigned] (SPARK-23010) Add integration testing for Kubernetes backend into the apache/spark repository

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23010: Assignee: (was: Apache Spark) > Add integration testing for Kubernetes backend into

[jira] [Assigned] (SPARK-23010) Add integration testing for Kubernetes backend into the apache/spark repository

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23010: Assignee: Apache Spark > Add integration testing for Kubernetes backend into the

[jira] [Commented] (SPARK-23010) Add integration testing for Kubernetes backend into the apache/spark repository

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382233#comment-16382233 ] Apache Spark commented on SPARK-23010: -- User 'ssuchter' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23405) The task will hang up when a small table left semi join a big table

2018-03-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23405: --- Assignee: KaiXinXIaoLei > The task will hang up when a small table left semi join a big

[jira] [Resolved] (SPARK-23405) The task will hang up when a small table left semi join a big table

2018-03-01 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23405. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20670

[jira] [Assigned] (SPARK-19185) ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19185: Assignee: (was: Apache Spark) > ConcurrentModificationExceptions with

[jira] [Commented] (SPARK-19185) ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382119#comment-16382119 ] Apache Spark commented on SPARK-19185: -- User 'gaborgsomogyi' has created a pull request for this

[jira] [Assigned] (SPARK-19185) ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-19185: Assignee: Apache Spark > ConcurrentModificationExceptions with CachedKafkaConsumers when

[jira] [Commented] (SPARK-23443) Spark with Glue as external catalog

2018-03-01 Thread Ameen Tayyebi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382117#comment-16382117 ] Ameen Tayyebi commented on SPARK-23443: --- Great, thank you so much. I've been stuck with bunch of

[jira] [Created] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-01 Thread Dong Jiang (JIRA)
Dong Jiang created SPARK-23549: -- Summary: Spark SQL unexpected behavior when comparing timestamp to date Key: SPARK-23549 URL: https://issues.apache.org/jira/browse/SPARK-23549 Project: Spark

[jira] [Updated] (SPARK-23549) Spark SQL unexpected behavior when comparing timestamp to date

2018-03-01 Thread Dong Jiang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Jiang updated SPARK-23549: --- Description: {code:java} scala> spark.version res1: String = 2.2.1 scala> spark.sql("select

[jira] [Commented] (SPARK-23443) Spark with Glue as external catalog

2018-03-01 Thread Devin Boyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382085#comment-16382085 ] Devin Boyer commented on SPARK-23443: - I would also be interested in helping if needed, or certainly

[jira] [Resolved] (SPARK-23548) Redirect loop from Resourcemanager to Spark Webui

2018-03-01 Thread Dieter (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dieter resolved SPARK-23548. Resolution: Not A Problem resolved. was name resolution issue > Redirect loop from Resourcemanager to

[jira] [Commented] (SPARK-19185) ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing

2018-03-01 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382034#comment-16382034 ] Gabor Somogyi commented on SPARK-19185: --- Same problem exists in structured streaming. Creating a

[jira] [Created] (SPARK-23548) Redirect loop from Resourcemanager to Spark Webui

2018-03-01 Thread Dieter (JIRA)
Dieter created SPARK-23548: -- Summary: Redirect loop from Resourcemanager to Spark Webui Key: SPARK-23548 URL: https://issues.apache.org/jira/browse/SPARK-23548 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-23547) Cleanup the .pipeout file when the Hive Session closed

2018-03-01 Thread zuotingbing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-23547: Description: !2018-03-01_202415.png!   when the hive session closed, we should also cleanup the

[jira] [Updated] (SPARK-23547) Cleanup the .pipeout file when the Hive Session closed

2018-03-01 Thread zuotingbing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-23547: Attachment: 2018-03-01_202415.png > Cleanup the .pipeout file when the Hive Session closed >

[jira] [Assigned] (SPARK-23547) Cleanup the .pipeout file when the Hive Session closed

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23547: Assignee: (was: Apache Spark) > Cleanup the .pipeout file when the Hive Session

[jira] [Commented] (SPARK-23547) Cleanup the .pipeout file when the Hive Session closed

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381910#comment-16381910 ] Apache Spark commented on SPARK-23547: -- User 'zuotingbing' has created a pull request for this

[jira] [Assigned] (SPARK-23547) Cleanup the .pipeout file when the Hive Session closed

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23547: Assignee: Apache Spark > Cleanup the .pipeout file when the Hive Session closed >

[jira] [Created] (SPARK-23547) Cleanup the .pipeout file when the Hive Session closed

2018-03-01 Thread zuotingbing (JIRA)
zuotingbing created SPARK-23547: --- Summary: Cleanup the .pipeout file when the Hive Session closed Key: SPARK-23547 URL: https://issues.apache.org/jira/browse/SPARK-23547 Project: Spark Issue

[jira] [Commented] (SPARK-23520) Add support for MapType fields in JSON schema inference

2018-03-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381883#comment-16381883 ] Hyukjin Kwon commented on SPARK-23520: -- Is it roughly a duplicate of SPARK-21651? > Add support for

[jira] [Assigned] (SPARK-23528) Expose vital statistics of GaussianMixtureModel

2018-03-01 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23528: Assignee: (was: Apache Spark) > Expose vital statistics of GaussianMixtureModel >

  1   2   >