[jira] [Updated] (SPARK-23295) Exclude Waring message when generating versions in make-distribution.sh

2018-01-31 Thread Kent Yao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-23295: - Description: When we specified a wrong profile to make a spark distribution, such as -Phadoop1000, we wi

[jira] [Assigned] (SPARK-23295) Exclude Waring message when generating versions in make-distribution.sh

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23295: Assignee: Apache Spark > Exclude Waring message when generating versions in make-distribu

[jira] [Commented] (SPARK-23295) Exclude Waring message when generating versions in make-distribution.sh

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348146#comment-16348146 ] Apache Spark commented on SPARK-23295: -- User 'yaooqinn' has created a pull request f

[jira] [Assigned] (SPARK-23295) Exclude Waring message when generating versions in make-distribution.sh

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23295: Assignee: (was: Apache Spark) > Exclude Waring message when generating versions in ma

[jira] [Commented] (SPARK-23284) Document several get API of ColumnVector's behavior when accessing null slot

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348145#comment-16348145 ] Wenchen Fan commented on SPARK-23284: - Since this defines the "return null" behavior

[jira] [Created] (SPARK-23295) Exclude Waring message when generating versions in make-distribution.sh

2018-01-31 Thread Kent Yao (JIRA)
Kent Yao created SPARK-23295: Summary: Exclude Waring message when generating versions in make-distribution.sh Key: SPARK-23295 URL: https://issues.apache.org/jira/browse/SPARK-23295 Project: Spark

[jira] [Updated] (SPARK-23284) Document several get API of ColumnVector's behavior when accessing null slot

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23284: Target Version/s: 2.3.0 > Document several get API of ColumnVector's behavior when accessing null s

[jira] [Updated] (SPARK-23284) Document several get API of ColumnVector's behavior when accessing null slot

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23284: Affects Version/s: (was: 2.4.0) 2.3.0 > Document several get API of Colu

[jira] [Updated] (SPARK-23202) Add new API in DataSourceWriter: onDataWriterCommit

2018-01-31 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-23202: --- Summary: Add new API in DataSourceWriter: onDataWriterCommit (was: Break down DataSourceV2Wr

[jira] [Updated] (SPARK-23202) Add new API in DataSourceWriter: onDataWriterCommit

2018-01-31 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-23202: --- Description: The current DataSourceWriter API makes it hard to implement  {{onTaskCommit(task

[jira] [Updated] (SPARK-23202) Add new API in DataSourceWriter: onDataWriterCommit

2018-01-31 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-23202: --- Description: The current DataSourceWriter API makes it hard to implement  {{onTaskCommit(task

[jira] [Updated] (SPARK-23202) Break down DataSourceV2Writer.commit into two phase

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23202: Priority: Major (was: Blocker) > Break down DataSourceV2Writer.commit into two phase > ---

[jira] [Updated] (SPARK-23202) Break down DataSourceV2Writer.commit into two phase

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23202: Target Version/s: 2.4.0 (was: 2.3.0) > Break down DataSourceV2Writer.commit into two phase > -

[jira] [Commented] (SPARK-23280) add map type support to ColumnVector

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348109#comment-16348109 ] Apache Spark commented on SPARK-23280: -- User 'ueshin' has created a pull request for

[jira] [Commented] (SPARK-10884) Support prediction on single instance for regression and classification related models

2018-01-31 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348108#comment-16348108 ] Weichen Xu commented on SPARK-10884: [~mingma] I hope so but now the code freeze and

[jira] [Commented] (SPARK-10884) Support prediction on single instance for regression and classification related models

2018-01-31 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348094#comment-16348094 ] Ming Ma commented on SPARK-10884: - Cool. Could we get it in for 2.3? This will bring Spar

[jira] [Updated] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

2018-01-31 Thread Ravinder Matte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravinder Matte updated SPARK-23294: --- Labels: ConsoleSink RateSource backpressure maxRate (was: ConsoleSink RateSource) > Spark S

[jira] [Updated] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

2018-01-31 Thread Ravinder Matte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravinder Matte updated SPARK-23294: --- Description: Using following configs while building SparkSession ("spark.streaming.backpress

[jira] [Updated] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

2018-01-31 Thread Ravinder Matte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravinder Matte updated SPARK-23294: --- Labels: ConsoleSink RateSource (was: kinesis) > Spark Streaming + Rate source + Console Sink

[jira] [Updated] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-01-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-23200: Fix Version/s: (was: 2.4.0) > Reset configuration when restarting from checkpoints > --

[jira] [Updated] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

2018-01-31 Thread Ravinder Matte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravinder Matte updated SPARK-23294: --- Fix Version/s: (was: 2.2.0) > Spark Streaming + Rate source + Console Sink : Receiver Max

[jira] [Reopened] (SPARK-23200) Reset configuration when restarting from checkpoints

2018-01-31 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao reopened SPARK-23200: - Assignee: (was: Santiago Saavedra) > Reset configuration when restarting from checkpoints >

[jira] [Updated] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

2018-01-31 Thread Ravinder Matte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravinder Matte updated SPARK-23294: --- Component/s: (was: DStreams) Structured Streaming > Spark Streaming + Ra

[jira] [Updated] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

2018-01-31 Thread Ravinder Matte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravinder Matte updated SPARK-23294: --- Affects Version/s: (was: 2.0.2) 2.2.1 > Spark Streaming + Rate sou

[jira] [Created] (SPARK-23294) Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated

2018-01-31 Thread Ravinder Matte (JIRA)
Ravinder Matte created SPARK-23294: -- Summary: Spark Streaming + Rate source + Console Sink : Receiver MaxRate is violated Key: SPARK-23294 URL: https://issues.apache.org/jira/browse/SPARK-23294 Proje

[jira] [Updated] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-01-31 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-23291: - Shepherd: Felix Cheung (was: Hossein Falaki) > SparkR : substr : In SparkR dataframe , starting

[jira] [Commented] (SPARK-22274) User-defined aggregation functions with pandas udf

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348033#comment-16348033 ] Apache Spark commented on SPARK-22274: -- User 'ueshin' has created a pull request for

[jira] [Commented] (SPARK-23293) data source v2 self join fails

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348020#comment-16348020 ] Apache Spark commented on SPARK-23293: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-23293) data source v2 self join fails

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23293: Assignee: Wenchen Fan (was: Apache Spark) > data source v2 self join fails >

[jira] [Assigned] (SPARK-23293) data source v2 self join fails

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23293: Assignee: Apache Spark (was: Wenchen Fan) > data source v2 self join fails >

[jira] [Created] (SPARK-23293) data source v2 self join fails

2018-01-31 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-23293: --- Summary: data source v2 self join fails Key: SPARK-23293 URL: https://issues.apache.org/jira/browse/SPARK-23293 Project: Spark Issue Type: Bug Compon

[jira] [Resolved] (SPARK-23188) Make vectorized columar reader batch size configurable

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23188. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20361 [https://githu

[jira] [Assigned] (SPARK-23188) Make vectorized columar reader batch size configurable

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23188: --- Assignee: Jiang Xingbo > Make vectorized columar reader batch size configurable > --

[jira] [Commented] (SPARK-23292) python tests related to pandas are skipped

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348008#comment-16348008 ] Apache Spark commented on SPARK-23292: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-23292) python tests related to pandas are skipped

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23292: Assignee: (was: Apache Spark) > python tests related to pandas are skipped > -

[jira] [Commented] (SPARK-10884) Support prediction on single instance for regression and classification related models

2018-01-31 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16348009#comment-16348009 ] Weichen Xu commented on SPARK-10884: [~mingma] sure. But need to wait until spark 2.3

[jira] [Assigned] (SPARK-23292) python tests related to pandas are skipped

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23292: Assignee: Apache Spark > python tests related to pandas are skipped >

[jira] [Assigned] (SPARK-23281) Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-23281: --- Assignee: Dilip Biswal > Query produces results in incorrect order when a composite order by clause

[jira] [Resolved] (SPARK-23281) Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23281. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.2 > Query produces results in incorr

[jira] [Resolved] (SPARK-21396) Spark Hive Thriftserver doesn't return UDT field

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21396. - Resolution: Fixed Fix Version/s: 2.3.0 > Spark Hive Thriftserver doesn't return UDT field > --

[jira] [Assigned] (SPARK-21396) Spark Hive Thriftserver doesn't return UDT field

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-21396: --- Assignee: Ken Tore Tallakstad > Spark Hive Thriftserver doesn't return UDT field > -

[jira] [Resolved] (SPARK-23268) Reorganize packages in data source V2

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23268. - Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 2.3.0 > Reorganize packages in

[jira] [Created] (SPARK-23292) python tests related to pandas are skipped

2018-01-31 Thread Yin Huai (JIRA)
Yin Huai created SPARK-23292: Summary: python tests related to pandas are skipped Key: SPARK-23292 URL: https://issues.apache.org/jira/browse/SPARK-23292 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-23247) combines Unsafe operations and statistics operations in Scan Data Source

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23247. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20415 [https://githu

[jira] [Assigned] (SPARK-23247) combines Unsafe operations and statistics operations in Scan Data Source

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-23247: --- Assignee: caoxuewen > combines Unsafe operations and statistics operations in Scan Data Sour

[jira] [Commented] (SPARK-18016) Code Generation: Constant Pool Past Limit for Wide/Nested Dataset

2018-01-31 Thread Gaurav Garg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347982#comment-16347982 ] Gaurav Garg commented on SPARK-18016: - [~kiszk], I have Spark 2.2.0 environment, will

[jira] [Resolved] (SPARK-23280) add map type support to ColumnVector

2018-01-31 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-23280. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20450 [https://githu

[jira] [Commented] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347967#comment-16347967 ] Apache Spark commented on SPARK-23291: -- User 'viirya' has created a pull request for

[jira] [Assigned] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23291: Assignee: (was: Apache Spark) > SparkR : substr : In SparkR dataframe , starting and e

[jira] [Assigned] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23291: Assignee: Apache Spark > SparkR : substr : In SparkR dataframe , starting and ending posit

[jira] [Commented] (SPARK-14492) Spark SQL 1.6.0 does not work with external Hive metastore version lower than 1.2.0; its not backwards compatible with earlier version

2018-01-31 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347922#comment-16347922 ] Dongjoon Hyun commented on SPARK-14492: --- Hi, [~smilegator]. According to the doc a

[jira] [Reopened] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-23157: -- > withColumn fails for a column that is a result of mapped DataSet > --

[jira] [Assigned] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23157: Assignee: Henry Robinson > withColumn fails for a column that is a result of mapped DataSe

[jira] [Resolved] (SPARK-23157) withColumn fails for a column that is a result of mapped DataSet

2018-01-31 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23157. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20443 [https://git

[jira] [Commented] (SPARK-10884) Support prediction on single instance for regression and classification related models

2018-01-31 Thread Ming Ma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347881#comment-16347881 ] Ming Ma commented on SPARK-10884: - Thanks [~WeichenXu123] and [~yanboliang]. While the lo

[jira] [Created] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1

2018-01-31 Thread Narendra (JIRA)
Narendra created SPARK-23291: Summary: SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1 Key: SPARK-23291 URL: http

[jira] [Commented] (SPARK-23251) ClassNotFoundException: scala.Any when there's a missing implicit Map encoder

2018-01-31 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-23251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347808#comment-16347808 ] Michal Šenkýř commented on SPARK-23251: --- Yes, this does seem like the Map encoder i

[jira] [Created] (SPARK-23290) inadvertent change in handling of DateType when converting to pandas dataframe

2018-01-31 Thread Andre Menck (JIRA)
Andre Menck created SPARK-23290: --- Summary: inadvertent change in handling of DateType when converting to pandas dataframe Key: SPARK-23290 URL: https://issues.apache.org/jira/browse/SPARK-23290 Project:

[jira] [Comment Edited] (SPARK-19209) "No suitable driver" on first try

2018-01-31 Thread Tony Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347700#comment-16347700 ] Tony Xu edited comment on SPARK-19209 at 1/31/18 10:26 PM: --- Thi

[jira] [Commented] (SPARK-19209) "No suitable driver" on first try

2018-01-31 Thread Tony Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347700#comment-16347700 ] Tony Xu commented on SPARK-19209: - This seems like a forgotten issue but I'm still experi

[jira] [Commented] (SPARK-23020) Re-enable Flaky Test: org.apache.spark.launcher.SparkLauncherSuite.testInProcessLauncher

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347684#comment-16347684 ] Apache Spark commented on SPARK-23020: -- User 'vanzin' has created a pull request for

[jira] [Assigned] (SPARK-23289) OneForOneBlockFetcher.DownloadCallback.onData may write just a part of data

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23289: Assignee: Apache Spark (was: Shixiong Zhu) > OneForOneBlockFetcher.DownloadCallback.onDat

[jira] [Commented] (SPARK-23289) OneForOneBlockFetcher.DownloadCallback.onData may write just a part of data

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347679#comment-16347679 ] Apache Spark commented on SPARK-23289: -- User 'zsxwing' has created a pull request fo

[jira] [Assigned] (SPARK-23289) OneForOneBlockFetcher.DownloadCallback.onData may write just a part of data

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23289: Assignee: Shixiong Zhu (was: Apache Spark) > OneForOneBlockFetcher.DownloadCallback.onDat

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-31 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: StageTab.png > Additional Memory Tuning Metrics > > >

[jira] [Updated] (SPARK-23206) Additional Memory Tuning Metrics

2018-01-31 Thread Edwina Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edwina Lu updated SPARK-23206: -- Attachment: (was: StageTab.png) > Additional Memory Tuning Metrics > --

[jira] [Created] (SPARK-23289) OneForOneBlockFetcher.DownloadCallback.onData may write just a part of data

2018-01-31 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-23289: Summary: OneForOneBlockFetcher.DownloadCallback.onData may write just a part of data Key: SPARK-23289 URL: https://issues.apache.org/jira/browse/SPARK-23289 Project:

[jira] [Commented] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-31 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347658#comment-16347658 ] Yanbo Liang commented on SPARK-23107: - [~sameerag] Yes, the fix should be API scope c

[jira] [Assigned] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23285: Assignee: (was: Apache Spark) > Allow spark.executor.cores to be fractional >

[jira] [Assigned] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23285: Assignee: Apache Spark > Allow spark.executor.cores to be fractional > ---

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347651#comment-16347651 ] Apache Spark commented on SPARK-23285: -- User 'liyinan926' has created a pull request

[jira] [Assigned] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23107: Assignee: Apache Spark (was: Yanbo Liang) > ML, Graph 2.3 QA: API: New Scala APIs, docs >

[jira] [Assigned] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23107: Assignee: Yanbo Liang (was: Apache Spark) > ML, Graph 2.3 QA: API: New Scala APIs, docs >

[jira] [Commented] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347592#comment-16347592 ] Apache Spark commented on SPARK-23107: -- User 'yanboliang' has created a pull request

[jira] [Commented] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-31 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347576#comment-16347576 ] Sameer Agarwal commented on SPARK-23107: [~yanboliang] other than adding docs, ar

[jira] [Resolved] (SPARK-21525) ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

2018-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-21525. Resolution: Fixed Fix Version/s: 2.4.0 > ReceiverSupervisorImpl seems to ignore the

[jira] [Assigned] (SPARK-21525) ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

2018-01-31 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-21525: -- Assignee: Marcelo Vanzin > ReceiverSupervisorImpl seems to ignore the error code when

[jira] [Commented] (SPARK-23285) Allow spark.executor.cores to be fractional

2018-01-31 Thread Yinan Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347484#comment-16347484 ] Yinan Li commented on SPARK-23285: -- Another option is to bypass that check for Kubernete

[jira] [Commented] (SPARK-23274) ReplaceExceptWithFilter fails on dataframes filtered on same column

2018-01-31 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347480#comment-16347480 ] Andrew Ash commented on SPARK-23274: Many thanks for the fast fix [~smilegator]! > R

[jira] [Resolved] (SPARK-20826) Support compression/decompression of ColumnVector in generated code

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20826. -- Resolution: Duplicate > Support compression/decompression of ColumnVector in generated

[jira] [Commented] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-31 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347405#comment-16347405 ] Weichen Xu commented on SPARK-23110: [~yanboliang] OK. > ML 2.3 QA: API: Java compat

[jira] [Resolved] (SPARK-20823) Generate code to build table cache using ColumnarBatch and to get value from ColumnVector for other types

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20823. -- Resolution: Won't Fix > Generate code to build table cache using ColumnarBatch and to g

[jira] [Resolved] (SPARK-20821) Add compression/decompression of data to ColumnVector for other data types

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20821. -- Resolution: Won't Fix > Add compression/decompression of data to ColumnVector for other

[jira] [Resolved] (SPARK-20820) Add compression/decompression of data to ColumnVector for other compression schemes

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20820. -- Resolution: Duplicate > Add compression/decompression of data to ColumnVector for other

[jira] [Resolved] (SPARK-20824) Generate code to get value from table cache with wider column in ColumnarBatch

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20824. -- Resolution: Won't Fix > Generate code to get value from table cache with wider column i

[jira] [Resolved] (SPARK-20825) Generate code to get value from table cache with wider column in ColumnarBatch for other data types

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-20825. -- Resolution: Not A Problem > Generate code to get value from table cache with wider colu

[jira] [Resolved] (SPARK-22747) Shorten lifetime of global variables used in HashAggregateExec

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-22747. -- Resolution: Won't Fix > Shorten lifetime of global variables used in HashAggregateExec

[jira] [Updated] (SPARK-21657) Spark has exponential time complexity to explode(array of structs)

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21657: Component/s: (was: Spark Core) > Spark has exponential time complexity to explode(array of structs) > -

[jira] [Commented] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347385#comment-16347385 ] Apache Spark commented on SPARK-23110: -- User 'WeichenXu123' has created a pull reque

[jira] [Updated] (SPARK-22036) BigDecimal multiplication sometimes returns null

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22036: Component/s: (was: Spark Core) SQL > BigDecimal multiplication sometimes returns null

[jira] [Commented] (SPARK-23107) ML, Graph 2.3 QA: API: New Scala APIs, docs

2018-01-31 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347386#comment-16347386 ] Yanbo Liang commented on SPARK-23107: - [~mlnick] Sorry for late response, really busy

[jira] [Assigned] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23110: Assignee: Apache Spark (was: Weichen Xu) > ML 2.3 QA: API: Java compatibility, docs > ---

[jira] [Assigned] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-31 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23110: Assignee: Weichen Xu (was: Apache Spark) > ML 2.3 QA: API: Java compatibility, docs > ---

[jira] [Commented] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-31 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347381#comment-16347381 ] Yanbo Liang commented on SPARK-23110: - [~mlnick] Yes, we should make it \{{private[ml

[jira] [Updated] (SPARK-22130) UTF8String.trim() inefficiently scans all white-space string twice.

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22130: Component/s: (was: Spark Core) SQL > UTF8String.trim() inefficiently scans all white-s

[jira] [Updated] (SPARK-22247) Hive partition filter very slow

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22247: Component/s: (was: Spark Core) > Hive partition filter very slow > --- > >

[jira] [Updated] (SPARK-22291) Postgresql UUID[] to Cassandra: Conversion Error

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22291: Component/s: (was: Spark Core) > Postgresql UUID[] to Cassandra: Conversion Error > ---

[jira] [Resolved] (SPARK-22935) Dataset with Java Beans for java.sql.Date throws CompileException

2018-01-31 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki resolved SPARK-22935. -- Resolution: Invalid > Dataset with Java Beans for java.sql.Date throws CompileException

[jira] [Commented] (SPARK-23110) ML 2.3 QA: API: Java compatibility, docs

2018-01-31 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347369#comment-16347369 ] Weichen Xu commented on SPARK-23110: +1. Should make it to be `private[ml]` > ML 2.3

[jira] [Updated] (SPARK-19236) Add createOrReplaceGlobalTempView

2018-01-31 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19236: Component/s: (was: Spark Core) SQL > Add createOrReplaceGlobalTempView > -

  1   2   >