[jira] [Comment Edited] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

2018-01-04 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312579#comment-16312579 ] wuyi edited comment on SPARK-22967 at 1/5/18 6:51 AM: -- @[~srowen] was (Author:

[jira] [Commented] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

2018-01-04 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312579#comment-16312579 ] wuyi commented on SPARK-22967: -- [~srowen] > VersionSuite failed on Windows caused by unescapeSQLString() >

[jira] [Comment Edited] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

2018-01-04 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312579#comment-16312579 ] wuyi edited comment on SPARK-22967 at 1/5/18 6:51 AM: -- @[~srowen] was (Author:

[jira] [Updated] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

2018-01-04 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-22967: - Shepherd: Sean Owen > VersionSuite failed on Windows caused by unescapeSQLString() >

[jira] [Resolved] (SPARK-22949) Reduce memory requirement for TrainValidationSplit

2018-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-22949. --- Resolution: Fixed Fix Version/s: 2.4.0 2.3.0

[jira] [Assigned] (SPARK-22949) Reduce memory requirement for TrainValidationSplit

2018-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley reassigned SPARK-22949: - Assignee: Bago Amirbekian > Reduce memory requirement for TrainValidationSplit

[jira] [Resolved] (SPARK-22825) Incorrect results of Casting Array to String

2018-01-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22825. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20024

[jira] [Assigned] (SPARK-22825) Incorrect results of Casting Array to String

2018-01-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22825: --- Assignee: Takeshi Yamamuro > Incorrect results of Casting Array to String >

[jira] [Commented] (SPARK-19547) KafkaUtil throw 'No current assignment for partition' Exception

2018-01-04 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312505#comment-16312505 ] Jepson commented on SPARK-19547: I have encounter this problem too.

[jira] [Commented] (SPARK-22805) Use aliases for StorageLevel in event logs

2018-01-04 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312499#comment-16312499 ] Imran Rashid commented on SPARK-22805: -- I'm leaning slightly against this, though could go either

[jira] [Updated] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

2018-01-04 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-22968: --- Description: *Kafka Broker:* {code:java} message.max.bytes : 2621440 {code} *Spark Streaming+Kafka

[jira] [Updated] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

2018-01-04 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-22968: --- Description: *Kafka Broker:* message.max.bytes : 2621440 *Spark Streaming+Kafka Code:* *,

[jira] [Updated] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

2018-01-04 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-22968: --- Description: CDH-KAFKA: message.max.bytes : 2621440 , "max.partition.fetch.bytes" -> (5242880:

[jira] [Closed] (SPARK-17282) Implement ALTER TABLE UPDATE STATISTICS SET

2018-01-04 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang closed SPARK-17282. > Implement ALTER TABLE UPDATE STATISTICS SET > --- > >

[jira] [Resolved] (SPARK-17282) Implement ALTER TABLE UPDATE STATISTICS SET

2018-01-04 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang resolved SPARK-17282. -- Resolution: Won't Fix > Implement ALTER TABLE UPDATE STATISTICS SET >

[jira] [Created] (SPARK-22968) java.lang.IllegalStateException: No current assignment for partition kssh-2

2018-01-04 Thread Jepson (JIRA)
Jepson created SPARK-22968: -- Summary: java.lang.IllegalStateException: No current assignment for partition kssh-2 Key: SPARK-22968 URL: https://issues.apache.org/jira/browse/SPARK-22968 Project: Spark

[jira] [Commented] (SPARK-17282) Implement ALTER TABLE UPDATE STATISTICS SET

2018-01-04 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312487#comment-16312487 ] Zhenhua Wang commented on SPARK-17282: -- I'm closing this for now. [~smilegator], [~hvanhovell],

[jira] [Commented] (SPARK-21209) Implement Incremental PCA algorithm for ML

2018-01-04 Thread Sandeep Kumar Choudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312477#comment-16312477 ] Sandeep Kumar Choudhary commented on SPARK-21209: - I would like to work on it. I have

[jira] [Commented] (SPARK-21727) Operating on an ArrayType in a SparkR DataFrame throws error

2018-01-04 Thread Neil Alexander McQuarrie (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312466#comment-16312466 ] Neil Alexander McQuarrie commented on SPARK-21727: -- I was able to get a version working

[jira] [Updated] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

2018-01-04 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-22967: - Description: On Windows system, two unit test case would fail while running VersionSuite ("A simple set of

[jira] [Updated] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

2018-01-04 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-22967: - Description: On Windows system, two unit test case would fail while running VersionSuite ("A simple set of

[jira] [Created] (SPARK-22967) VersionSuite failed on Windows caused by unescapeSQLString()

2018-01-04 Thread wuyi (JIRA)
wuyi created SPARK-22967: Summary: VersionSuite failed on Windows caused by unescapeSQLString() Key: SPARK-22967 URL: https://issues.apache.org/jira/browse/SPARK-22967 Project: Spark Issue Type:

[jira] [Commented] (SPARK-7721) Generate test coverage report from Python

2018-01-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312398#comment-16312398 ] Reynold Xin commented on SPARK-7721: I think it's fine even if you don't preserve the history forever

[jira] [Resolved] (SPARK-22957) ApproxQuantile breaks if the number of rows exceeds MaxInt

2018-01-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22957. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20152

[jira] [Assigned] (SPARK-22957) ApproxQuantile breaks if the number of rows exceeds MaxInt

2018-01-04 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22957: --- Assignee: Juliusz Sompolski > ApproxQuantile breaks if the number of rows exceeds MaxInt >

[jira] [Commented] (SPARK-22632) Fix the behavior of timestamp values for R's DataFrame to respect session timezone

2018-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312356#comment-16312356 ] Hyukjin Kwon commented on SPARK-22632: -- To me, nope, I don't think so although it might be important

[jira] [Commented] (SPARK-7721) Generate test coverage report from Python

2018-01-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312331#comment-16312331 ] Hyukjin Kwon commented on SPARK-7721: - I (and possibly few committers given [the comment

[jira] [Commented] (SPARK-22966) Spark SQL should handle Python UDFs that return a datetime.date or datetime.datetime

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312312#comment-16312312 ] Apache Spark commented on SPARK-22966: -- User 'rednaxelafx' has created a pull request for this

[jira] [Assigned] (SPARK-22966) Spark SQL should handle Python UDFs that return a datetime.date or datetime.datetime

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22966: Assignee: Apache Spark > Spark SQL should handle Python UDFs that return a datetime.date

[jira] [Assigned] (SPARK-22966) Spark SQL should handle Python UDFs that return a datetime.date or datetime.datetime

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22966: Assignee: (was: Apache Spark) > Spark SQL should handle Python UDFs that return a

[jira] [Created] (SPARK-22966) Spark SQL should handle Python UDFs that return a datetime.date or datetime.datetime

2018-01-04 Thread Kris Mok (JIRA)
Kris Mok created SPARK-22966: Summary: Spark SQL should handle Python UDFs that return a datetime.date or datetime.datetime Key: SPARK-22966 URL: https://issues.apache.org/jira/browse/SPARK-22966

[jira] [Updated] (SPARK-22966) Spark SQL should handle Python UDFs that return a datetime.date or datetime.datetime

2018-01-04 Thread Kris Mok (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kris Mok updated SPARK-22966: - Description: Currently, in Spark SQL, if a Python UDF returns a {{datetime.date}} (which should

[jira] [Updated] (SPARK-21972) Allow users to control input data persistence in ML Estimators via a handlePersistence ml.Param

2018-01-04 Thread Joseph K. Bradley (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-21972: -- Target Version/s: 2.4.0 (was: 2.3.0) > Allow users to control input data persistence

[jira] [Updated] (SPARK-22054) Allow release managers to inject their keys

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-22054: --- Target Version/s: 2.4.0 (was: 2.3.0) > Allow release managers to inject their keys >

[jira] [Commented] (SPARK-22055) Port release scripts

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312292#comment-16312292 ] Sameer Agarwal commented on SPARK-22055: re-targeting for 2.4.0 > Port release scripts >

[jira] [Updated] (SPARK-22055) Port release scripts

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-22055: --- Target Version/s: 2.4.0 (was: 2.3.0) > Port release scripts > > >

[jira] [Commented] (SPARK-22054) Allow release managers to inject their keys

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312293#comment-16312293 ] Sameer Agarwal commented on SPARK-22054: re-targeting for 2.4.0 > Allow release managers to

[jira] [Commented] (SPARK-22632) Fix the behavior of timestamp values for R's DataFrame to respect session timezone

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312287#comment-16312287 ] Sameer Agarwal commented on SPARK-22632: [~hyukjin.kwon] [~felixcheung] should this be a blocker

[jira] [Commented] (SPARK-22739) Additional Expression Support for Objects

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312281#comment-16312281 ] Sameer Agarwal commented on SPARK-22739: Raising priority for this. This would be great to have

[jira] [Updated] (SPARK-22739) Additional Expression Support for Objects

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-22739: --- Priority: Critical (was: Major) > Additional Expression Support for Objects >

[jira] [Updated] (SPARK-22799) Bucketizer should throw exception if single- and multi-column params are both set

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-22799: --- Target Version/s: 2.4.0 (was: 2.3.0) > Bucketizer should throw exception if single- and

[jira] [Updated] (SPARK-22798) Add multiple column support to PySpark StringIndexer

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sameer Agarwal updated SPARK-22798: --- Target Version/s: 2.4.0 (was: 2.3.0) > Add multiple column support to PySpark StringIndexer

[jira] [Commented] (SPARK-22799) Bucketizer should throw exception if single- and multi-column params are both set

2018-01-04 Thread Sameer Agarwal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312275#comment-16312275 ] Sameer Agarwal commented on SPARK-22799: [~mgaido] [~mlnick] I'm re-targeting this for 2.4.0.

[jira] [Assigned] (SPARK-22960) Make build-push-docker-images.sh more dev-friendly

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22960: -- Assignee: Marcelo Vanzin > Make build-push-docker-images.sh more dev-friendly >

[jira] [Resolved] (SPARK-22960) Make build-push-docker-images.sh more dev-friendly

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22960. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20154

[jira] [Commented] (SPARK-22126) Fix model-specific optimization support for ML tuning

2018-01-04 Thread Bago Amirbekian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312264#comment-16312264 ] Bago Amirbekian commented on SPARK-22126: - [~bryanc] thanks for taking the time to put together

[jira] [Assigned] (SPARK-22965) Add deterministic parameter to registerJavaFunction

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22965: Assignee: Apache Spark (was: Xiao Li) > Add deterministic parameter to

[jira] [Assigned] (SPARK-22965) Add deterministic parameter to registerJavaFunction

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22965: Assignee: Xiao Li (was: Apache Spark) > Add deterministic parameter to

[jira] [Commented] (SPARK-22965) Add deterministic parameter to registerJavaFunction

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312232#comment-16312232 ] Apache Spark commented on SPARK-22965: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Created] (SPARK-22965) Add deterministic parameter to registerJavaFunction

2018-01-04 Thread Xiao Li (JIRA)
Xiao Li created SPARK-22965: --- Summary: Add deterministic parameter to registerJavaFunction Key: SPARK-22965 URL: https://issues.apache.org/jira/browse/SPARK-22965 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-22953) Duplicated secret volumes in Spark pods when init-containers are used

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22953. Resolution: Fixed Issue resolved by pull request 20159

[jira] [Assigned] (SPARK-22953) Duplicated secret volumes in Spark pods when init-containers are used

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22953: -- Assignee: Yinan Li > Duplicated secret volumes in Spark pods when init-containers are

[jira] [Commented] (SPARK-21525) ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312202#comment-16312202 ] Apache Spark commented on SPARK-21525: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21525) ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21525: Assignee: Apache Spark > ReceiverSupervisorImpl seems to ignore the error code when

[jira] [Assigned] (SPARK-21525) ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21525: Assignee: (was: Apache Spark) > ReceiverSupervisorImpl seems to ignore the error code

[jira] [Commented] (SPARK-22757) Init-container in the driver/executor pods for downloading remote dependencies

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312172#comment-16312172 ] Apache Spark commented on SPARK-22757: -- User 'liyinan926' has created a pull request for this issue:

[jira] [Commented] (SPARK-22953) Duplicated secret volumes in Spark pods when init-containers are used

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312160#comment-16312160 ] Apache Spark commented on SPARK-22953: -- User 'liyinan926' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22948) "SparkPodInitContainer" shouldn't be in "rest" package

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22948: -- Assignee: Marcelo Vanzin > "SparkPodInitContainer" shouldn't be in "rest" package >

[jira] [Resolved] (SPARK-22948) "SparkPodInitContainer" shouldn't be in "rest" package

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22948. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20156

[jira] [Assigned] (SPARK-22850) Executor page in SHS does not show driver

2018-01-04 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-22850: Assignee: Marcelo Vanzin > Executor page in SHS does not show driver >

[jira] [Resolved] (SPARK-22850) Executor page in SHS does not show driver

2018-01-04 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-22850. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 20039

[jira] [Updated] (SPARK-22963) Make failure recovery global and automatic for continuous processing.

2018-01-04 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Torres updated SPARK-22963: Summary: Make failure recovery global and automatic for continuous processing. (was: Clean up

[jira] [Updated] (SPARK-22963) Make failure recovery global and automatic for continuous processing.

2018-01-04 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Torres updated SPARK-22963: Description: Spark native task restarts don't work well for continuous processing. They will

[jira] [Updated] (SPARK-22963) Clean up continuous processing failure recovery

2018-01-04 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Torres updated SPARK-22963: Description: Spark native task restarts don't work well for continuous processing. They will

[jira] [Updated] (SPARK-22963) Clean up continuous processing failure recovery

2018-01-04 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Torres updated SPARK-22963: Description: Spark native task restarts don't work well for continuous processing. They will

[jira] [Created] (SPARK-22964) don't allow task restarts for continuous processing

2018-01-04 Thread Jose Torres (JIRA)
Jose Torres created SPARK-22964: --- Summary: don't allow task restarts for continuous processing Key: SPARK-22964 URL: https://issues.apache.org/jira/browse/SPARK-22964 Project: Spark Issue

[jira] [Updated] (SPARK-22963) Clean up continuous processing failure recovery

2018-01-04 Thread Jose Torres (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jose Torres updated SPARK-22963: Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-20928) > Clean up

[jira] [Created] (SPARK-22963) Clean up continuous processing failure recovery

2018-01-04 Thread Jose Torres (JIRA)
Jose Torres created SPARK-22963: --- Summary: Clean up continuous processing failure recovery Key: SPARK-22963 URL: https://issues.apache.org/jira/browse/SPARK-22963 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21760) Structured streaming terminates with Exception

2018-01-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312089#comment-16312089 ] Shixiong Zhu commented on SPARK-21760: -- Could you try 2.2.1? It's probably fixed by this line:

[jira] [Commented] (SPARK-22757) Init-container in the driver/executor pods for downloading remote dependencies

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312064#comment-16312064 ] Apache Spark commented on SPARK-22757: -- User 'liyinan926' has created a pull request for this issue:

[jira] [Resolved] (SPARK-21475) Change the usage of FileInputStream/OutputStream to Files.newInput/OutputStream in the critical path

2018-01-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21475. -- Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Issue resolved by pull

[jira] [Updated] (SPARK-21475) Change to use NIO's Files API for external shuffle service

2018-01-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21475: - Summary: Change to use NIO's Files API for external shuffle service (was: Change the usage of

[jira] [Assigned] (SPARK-12344) Remove env-based configurations

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-12344: -- Assignee: (was: Marcelo Vanzin) > Remove env-based configurations >

[jira] [Assigned] (SPARK-12344) Remove env-based configurations

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-12344: -- Assignee: Marcelo Vanzin > Remove env-based configurations >

[jira] [Updated] (SPARK-20657) Speed up Stage page

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-20657: --- Target Version/s: 2.3.0 I'm targeting 2.3.0 since the stages page is really slow for large

[jira] [Updated] (SPARK-22850) Executor page in SHS does not show driver

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-22850: --- Target Version/s: 2.3.0 > Executor page in SHS does not show driver >

[jira] [Commented] (SPARK-22948) "SparkPodInitContainer" shouldn't be in "rest" package

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311864#comment-16311864 ] Apache Spark commented on SPARK-22948: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22948) "SparkPodInitContainer" shouldn't be in "rest" package

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22948: Assignee: Apache Spark > "SparkPodInitContainer" shouldn't be in "rest" package >

[jira] [Assigned] (SPARK-22948) "SparkPodInitContainer" shouldn't be in "rest" package

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22948: Assignee: (was: Apache Spark) > "SparkPodInitContainer" shouldn't be in "rest"

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-01-04 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311859#comment-16311859 ] Li Jin commented on SPARK-22947: I have implemented a simple version of "as-of join" using existing Spark

[jira] [Updated] (SPARK-22962) Kubernetes app fails if local files are used

2018-01-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-22962: --- Issue Type: Bug (was: Improvement) > Kubernetes app fails if local files are used >

[jira] [Created] (SPARK-22962) Kubernetes app fails if local files are used

2018-01-04 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-22962: -- Summary: Kubernetes app fails if local files are used Key: SPARK-22962 URL: https://issues.apache.org/jira/browse/SPARK-22962 Project: Spark Issue Type:

[jira] [Commented] (SPARK-22961) Constant columns no longer picked as constraints in 2.3

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311771#comment-16311771 ] Apache Spark commented on SPARK-22961: -- User 'adrian-ionescu' has created a pull request for this

[jira] [Assigned] (SPARK-22961) Constant columns no longer picked as constraints in 2.3

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22961: Assignee: (was: Apache Spark) > Constant columns no longer picked as constraints in

[jira] [Assigned] (SPARK-22961) Constant columns no longer picked as constraints in 2.3

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22961: Assignee: Apache Spark > Constant columns no longer picked as constraints in 2.3 >

[jira] [Created] (SPARK-22961) Constant columns no longer picked as constraints in 2.3

2018-01-04 Thread Adrian Ionescu (JIRA)
Adrian Ionescu created SPARK-22961: -- Summary: Constant columns no longer picked as constraints in 2.3 Key: SPARK-22961 URL: https://issues.apache.org/jira/browse/SPARK-22961 Project: Spark

[jira] [Assigned] (SPARK-22960) Make build-push-docker-images.sh more dev-friendly

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22960: Assignee: Apache Spark > Make build-push-docker-images.sh more dev-friendly >

[jira] [Commented] (SPARK-22960) Make build-push-docker-images.sh more dev-friendly

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311746#comment-16311746 ] Apache Spark commented on SPARK-22960: -- User 'vanzin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22960) Make build-push-docker-images.sh more dev-friendly

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22960: Assignee: (was: Apache Spark) > Make build-push-docker-images.sh more dev-friendly >

[jira] [Created] (SPARK-22960) Make build-push-docker-images.sh more dev-friendly

2018-01-04 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-22960: -- Summary: Make build-push-docker-images.sh more dev-friendly Key: SPARK-22960 URL: https://issues.apache.org/jira/browse/SPARK-22960 Project: Spark Issue

[jira] [Commented] (SPARK-7721) Generate test coverage report from Python

2018-01-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311731#comment-16311731 ] Reynold Xin commented on SPARK-7721: We can add it first but in my experience this will only be used

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-01-04 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311675#comment-16311675 ] Li Jin commented on SPARK-22947: There could be multiple matches within the range, and in that case,

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-01-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311673#comment-16311673 ] Reynold Xin commented on SPARK-22947: - Basically we should separate the logical plan from the

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-01-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311667#comment-16311667 ] Reynold Xin commented on SPARK-22947: - So this is just a hint that there is only one matching tuple

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2018-01-04 Thread Li Jin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311566#comment-16311566 ] Li Jin commented on SPARK-22947: Thanks for the feedback. [~rxin], as-of join is different from range

[jira] [Assigned] (SPARK-22392) columnar reader interface

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22392: Assignee: (was: Apache Spark) > columnar reader interface >

[jira] [Commented] (SPARK-22392) columnar reader interface

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311487#comment-16311487 ] Apache Spark commented on SPARK-22392: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22392) columnar reader interface

2018-01-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22392: Assignee: Apache Spark > columnar reader interface > -- > >

[jira] [Commented] (SPARK-20436) NullPointerException when restart from checkpoint file

2018-01-04 Thread Riccardo Vincelli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311402#comment-16311402 ] Riccardo Vincelli commented on SPARK-20436: --- I have tested this myself and it seems that this

[jira] [Commented] (SPARK-22955) Error generating jobs when Stopping JobGenerator gracefully

2018-01-04 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16311399#comment-16311399 ] Sean Owen commented on SPARK-22955: --- [~tdas] should the order of these two stops be reversed? the

  1   2   >