[jira] [Created] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
Jiatao Tao created SPARK-27546: -- Summary: Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone Key: SPARK-27546 URL: https://issues.apache.org/jira/browse/SPARK-27546 Project: Spark

[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823801#comment-16823801 ] Jiatao Tao commented on SPARK-27546: see an example like this:   {code:java} // con

[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823801#comment-16823801 ] Jiatao Tao edited comment on SPARK-27546 at 4/23/19 8:10 AM: -

[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823808#comment-16823808 ] Jiatao Tao commented on SPARK-27546: So in my opinion, we should change DateTimeUtil

[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-04-23 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823801#comment-16823801 ] Jiatao Tao edited comment on SPARK-27546 at 4/23/19 8:17 AM: -

[jira] [Commented] (SPARK-27542) SparkHadoopWriter doesn't set call setWorkOutputPath, causing NPEs when using certain legacy OutputFormats

2019-04-23 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823844#comment-16823844 ] Shivu Sondur commented on SPARK-27542: -- [~joshrosen] Can you provide the exact ste

[jira] [Commented] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-23 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823876#comment-16823876 ] Marco Gaido commented on SPARK-27287: - [~dharmesh.kakadia] the point is: if you set

[jira] [Created] (SPARK-27547) fix DataFrame self-join problems

2019-04-23 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27547: --- Summary: fix DataFrame self-join problems Key: SPARK-27547 URL: https://issues.apache.org/jira/browse/SPARK-27547 Project: Spark Issue Type: Bug Comp

[jira] [Resolved] (SPARK-27544) Fix Python test script to work on Scala-2.12 build

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27544. --- Resolution: Fixed Fix Version/s: 2.4.3 This is resolved via https://github.com/apache

[jira] [Resolved] (SPARK-27481) Upgrade commons-logging to 1.1.3

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27481. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24388 [https://github.c

[jira] [Updated] (SPARK-27481) Upgrade commons-logging to 1.1.3

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27481: -- Priority: Minor (was: Major) > Upgrade commons-logging to 1.1.3 > >

[jira] [Assigned] (SPARK-27481) Upgrade commons-logging to 1.1.3

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27481: - Assignee: Yuming Wang > Upgrade commons-logging to 1.1.3 > > >

[jira] [Assigned] (SPARK-27503) JobGenerator thread exit for some fatal errors but application keeps running

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27503: - Assignee: Genmao Yu > JobGenerator thread exit for some fatal errors but application keeps runn

[jira] [Resolved] (SPARK-27503) JobGenerator thread exit for some fatal errors but application keeps running

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27503. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24400 [https://github.c

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824177#comment-16824177 ] Ismael Juma commented on SPARK-25954: - [~dongjoon] Kafka 2.2.0 has a fix for an issu

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824188#comment-16824188 ] Gabor Somogyi commented on SPARK-25954: --- [~ijuma] On the actual master 2.2.0 kafka

[jira] [Issue Comment Deleted] (SPARK-27421) RuntimeException when querying a view on a partitioned parquet table

2019-04-23 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivu Sondur updated SPARK-27421: - Comment: was deleted (was: i am checking this issue) > RuntimeException when querying a view on

[jira] [Issue Comment Deleted] (SPARK-27409) Micro-batch support for Kafka Source in Spark 2.3

2019-04-23 Thread Shivu Sondur (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivu Sondur updated SPARK-27409: - Comment: was deleted (was: i am checking this) > Micro-batch support for Kafka Source in Spark

[jira] [Assigned] (SPARK-27128) Migrate JSON to File Data Source V2

2019-04-23 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27128: --- Assignee: Gengliang Wang > Migrate JSON to File Data Source V2 > --

[jira] [Resolved] (SPARK-27128) Migrate JSON to File Data Source V2

2019-04-23 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27128. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24058 [https://gith

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824226#comment-16824226 ] Ismael Juma commented on SPARK-25954: - That's awesome, sorry for the noise. > Upgra

[jira] [Created] (SPARK-27548) PySpark toLocalIterator does not raise errors from worker

2019-04-23 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-27548: Summary: PySpark toLocalIterator does not raise errors from worker Key: SPARK-27548 URL: https://issues.apache.org/jira/browse/SPARK-27548 Project: Spark Iss

[jira] [Resolved] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-04-23 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-26970. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24426 [https://gi

[jira] [Commented] (SPARK-25954) Upgrade to Kafka 2.1.0

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824550#comment-16824550 ] Dongjoon Hyun commented on SPARK-25954: --- Yep. [~ijuma]. I updated Spark already vi

[jira] [Commented] (SPARK-27534) Do not load `content` column in binary data source if it is not selected

2019-04-23 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824554#comment-16824554 ] Weichen Xu commented on SPARK-27534: I am working on this. :) > Do not load `conten

[jira] [Created] (SPARK-27549) Commit Kafka Source offsets to facilitate external tooling

2019-04-23 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-27549: --- Summary: Commit Kafka Source offsets to facilitate external tooling Key: SPARK-27549 URL: https://issues.apache.org/jira/browse/SPARK-27549 Project: Spar

[jira] [Assigned] (SPARK-26970) Can't load PipelineModel that was created in Scala with Python due to missing Interaction transformer

2019-04-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-26970: - Assignee: Andrew Crosby > Can't load PipelineModel that was created in Scala with Python due to

[jira] [Updated] (SPARK-27550) Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27550: -- Issue Type: Bug (was: Task) > Fix `test-dependencies.sh` not to use `kafka-0-8` profile for S

[jira] [Created] (SPARK-27550) Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

2019-04-23 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-27550: - Summary: Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12 Key: SPARK-27550 URL: https://issues.apache.org/jira/browse/SPARK-27550 Project: S

[jira] [Updated] (SPARK-27550) Fix `test-dependencies.sh` not to use `kafka-0-8` profile for Scala-2.12

2019-04-23 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27550: -- Description: Since SPARK-27274 deprecated Scala-2.11 at Spark 2.4.1, we need to test Scala-2.

[jira] [Commented] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-23 Thread Dharmesh Kakadia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824604#comment-16824604 ] Dharmesh Kakadia commented on SPARK-27287: -- Understood. I do not know how to "u

[jira] [Comment Edited] (SPARK-27287) PCAModel.load() does not honor spark configs

2019-04-23 Thread Dharmesh Kakadia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824604#comment-16824604 ] Dharmesh Kakadia edited comment on SPARK-27287 at 4/23/19 10:27 PM: --

[jira] [Created] (SPARK-27551) Uniformative error message for mismatched types in when().otherwise()

2019-04-23 Thread Huon Wilson (JIRA)
Huon Wilson created SPARK-27551: --- Summary: Uniformative error message for mismatched types in when().otherwise() Key: SPARK-27551 URL: https://issues.apache.org/jira/browse/SPARK-27551 Project: Spark

[jira] [Updated] (SPARK-27551) Uniformative error message for mismatched types in when().otherwise()

2019-04-23 Thread Huon Wilson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huon Wilson updated SPARK-27551: Description: When a {{when(...).otherwise(...)}} construct has a type error, the error message ca

[jira] [Created] (SPARK-27552) The configuration `hive.exec.stagingdir` is invalid on Windows OS

2019-04-23 Thread liuxian (JIRA)
liuxian created SPARK-27552: --- Summary: The configuration `hive.exec.stagingdir` is invalid on Windows OS Key: SPARK-27552 URL: https://issues.apache.org/jira/browse/SPARK-27552 Project: Spark Issu

[jira] [Created] (SPARK-27553) Operation log is not closed when close session

2019-04-23 Thread pin_zhang (JIRA)
pin_zhang created SPARK-27553: - Summary: Operation log is not closed when close session Key: SPARK-27553 URL: https://issues.apache.org/jira/browse/SPARK-27553 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-04-23 Thread Jepson (JIRA)
Jepson created SPARK-27554: -- Summary: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] Key: SPARK-27554 URL: https://issues.apache.org/jira/browse/SPARK-27554

[jira] [Updated] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-04-23 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jepson updated SPARK-27554: --- Description: {code:java}  {code} #basic env JAVA_HOME=/usr/java/jdk1.8.0_181 HADOOP_CONF_DIR=/etc/hadoop/