[jira] [Commented] (SPARK-29014) DataSourceV2: Clean up current, default, and session catalog uses

2019-09-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926338#comment-16926338 ] Wenchen Fan commented on SPARK-29014: - I have a major refactor to solve this issue, so that we don't

[jira] [Resolved] (SPARK-26598) Fix HiveThriftServer2 set hiveconf and hivevar in every sql

2019-09-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-26598. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25722

[jira] [Assigned] (SPARK-26598) Fix HiveThriftServer2 set hiveconf and hivevar in every sql

2019-09-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-26598: --- Assignee: dzcxzl > Fix HiveThriftServer2 set hiveconf and hivevar in every sql >

[jira] [Commented] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926322#comment-16926322 ] Hyukjin Kwon commented on SPARK-29004: -- The exception message is correct and we should fix the doc

[jira] [Commented] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Pavel Parkhomenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926319#comment-16926319 ] Pavel Parkhomenko commented on SPARK-29004: --- This exception contradicts to documentation and

[jira] [Updated] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29004: - Component/s: (was: Spark Core) SQL > DataFrameWriter.save does not work

[jira] [Commented] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926314#comment-16926314 ] Hyukjin Kwon commented on SPARK-29004: -- It correctly throws an exception at this moment so it's

[jira] [Updated] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29004: - Issue Type: Documentation (was: Bug) > DataFrameWriter.save does not work with bucketBy [and

[jira] [Updated] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29004: - Priority: Trivial (was: Major) > DataFrameWriter.save does not work with bucketBy [and sortBy]

[jira] [Comment Edited] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Pavel Parkhomenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926312#comment-16926312 ] Pavel Parkhomenko edited comment on SPARK-29004 at 9/10/19 4:00 AM:

[jira] [Reopened] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Pavel Parkhomenko (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavel Parkhomenko reopened SPARK-29004: --- It is not mentioned in documentation that I should use only saveAsTable. Moreover,

[jira] [Commented] (SPARK-28970) implement USE CATALOG/NAMESPACE for Data Source V2

2019-09-09 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926309#comment-16926309 ] Terry Kim commented on SPARK-28970: --- [~rdblue] I think you probably missed my earlier question. Are

[jira] [Commented] (SPARK-29014) DataSourceV2: Clean up current, default, and session catalog uses

2019-09-09 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926307#comment-16926307 ] Terry Kim commented on SPARK-29014: --- [~rdblue] / [~cloud_fan] did you start working on it? If not, I

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-09 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926299#comment-16926299 ] Jungtaek Lim commented on SPARK-29027: -- The test passed in my local dev. MacOS 10.14.2, JDK

[jira] [Resolved] (SPARK-29004) DataFrameWriter.save does not work with bucketBy [and sortBy]

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29004. -- Resolution: Invalid > DataFrameWriter.save does not work with bucketBy [and sortBy] >

[jira] [Resolved] (SPARK-29009) Returning pojo from udf not working

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29009. -- Resolution: Invalid Resolving due to no feedback from its author. > Returning pojo from udf

[jira] [Resolved] (SPARK-29023) Remove extra logic of executor pod status in ExecutorPodsSnapshot.scala

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29023. -- Resolution: Duplicate > Remove extra logic of executor pod status in

[jira] [Commented] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926288#comment-16926288 ] Hyukjin Kwon commented on SPARK-29027: -- Seems like this is passing correctly in Jenkins. Would you

[jira] [Updated] (SPARK-23519) Create View Commands Fails with The view output (col1,col1) contains duplicate column name

2019-09-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-23519: Fix Version/s: 2.4.5 > Create View Commands Fails with The view output (col1,col1) contains >

[jira] [Issue Comment Deleted] (SPARK-28516) Data Type Formatting Functions: `to_char`

2019-09-09 Thread YoungGyu Chun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YoungGyu Chun updated SPARK-28516: -- Comment: was deleted (was: I'm working on this!) > Data Type Formatting Functions: `to_char`

[jira] [Resolved] (SPARK-28988) "CliSuite#Commands using SerDe provided in --jars" not strong enough

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28988. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25724

[jira] [Assigned] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2019-09-09 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh reassigned SPARK-23265: --- Assignee: Huaxin Gao > Update multi-column error handling logic in

[jira] [Resolved] (SPARK-23265) Update multi-column error handling logic in QuantileDiscretizer

2019-09-09 Thread Liang-Chi Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-23265. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 20442

[jira] [Updated] (SPARK-29029) PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases

2019-09-09 Thread Nikita Konda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Konda updated SPARK-29029: - Description: We have a specific use case where in we are trying insert a custom logical

[jira] [Updated] (SPARK-29029) PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases

2019-09-09 Thread Nikita Konda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Konda updated SPARK-29029: - Description: We have a specific use case where in we are trying insert a custom logical

[jira] [Updated] (SPARK-29029) PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases

2019-09-09 Thread Nikita Konda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Konda updated SPARK-29029: - Description: We have a specific use case where in we are trying insert a custom logical

[jira] [Commented] (SPARK-28902) Spark ML Pipeline with nested Pipelines fails to load when saved from Python

2019-09-09 Thread Saif Addin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926226#comment-16926226 ] Saif Addin commented on SPARK-28902: Hi [~reconjun] I am not sure how you made it work, it fails on

[jira] [Updated] (SPARK-29029) PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases

2019-09-09 Thread Nikita Konda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Konda updated SPARK-29029: - Description: We have a specific use case where in we are trying insert a custom logical

[jira] [Updated] (SPARK-29030) Simplify lookupV2Relation

2019-09-09 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-29030: --- Description: Simplify the return type for {{lookupV2Relation}} which makes the 3 callers more

[jira] [Created] (SPARK-29030) Simplify lookupV2Relation

2019-09-09 Thread John Zhuge (Jira)
John Zhuge created SPARK-29030: -- Summary: Simplify lookupV2Relation Key: SPARK-29030 URL: https://issues.apache.org/jira/browse/SPARK-29030 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-29029) PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases

2019-09-09 Thread Nikita Konda (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikita Konda updated SPARK-29029: - Description: We have a specific use case where in we are trying insert a custom logical

[jira] [Created] (SPARK-29029) PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases

2019-09-09 Thread Nikita Konda (Jira)
Nikita Konda created SPARK-29029: Summary: PhysicalOperation.collectProjectsAndFilters should use AttributeMap while substituting aliases Key: SPARK-29029 URL: https://issues.apache.org/jira/browse/SPARK-29029

[jira] [Updated] (SPARK-28657) Fix currentContext Instance failed sometimes

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28657: -- Fix Version/s: (was: 2.4.5) > Fix currentContext Instance failed sometimes >

[jira] [Resolved] (SPARK-28412) ANSI SQL: OVERLAY function support byte array(T312)

2019-09-09 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-28412. -- Fix Version/s: 3.0.0 Assignee: jiaan.geng Resolution: Fixed Resolved

[jira] [Resolved] (SPARK-28657) Fix currentContext Instance failed sometimes

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28657. --- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-28657) Fix currentContext Instance failed sometimes

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28657: - Assignee: hong dongdong > Fix currentContext Instance failed sometimes >

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-09-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926165#comment-16926165 ] Dongjoon Hyun commented on SPARK-28921: --- Thank you for confirmation, [~Fokko]. > Spark jobs

[jira] [Created] (SPARK-29028) Add links to IBM Cloud Object Storage connector in cloud-integration.md

2019-09-09 Thread Dilip Biswal (Jira)
Dilip Biswal created SPARK-29028: Summary: Add links to IBM Cloud Object Storage connector in cloud-integration.md Key: SPARK-29028 URL: https://issues.apache.org/jira/browse/SPARK-29028 Project:

[jira] [Assigned] (SPARK-28928) Use Kafka delegation token protocol on sources/sinks

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28928: -- Assignee: Gabor Somogyi > Use Kafka delegation token protocol on sources/sinks >

[jira] [Resolved] (SPARK-28928) Use Kafka delegation token protocol on sources/sinks

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28928. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25631

[jira] [Created] (SPARK-29027) KafkaDelegationTokenSuite fails

2019-09-09 Thread koert kuipers (Jira)
koert kuipers created SPARK-29027: - Summary: KafkaDelegationTokenSuite fails Key: SPARK-29027 URL: https://issues.apache.org/jira/browse/SPARK-29027 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-28214) Flaky test: org.apache.spark.streaming.CheckpointSuite.basic rdd checkpoints + dstream graph checkpoint recovery

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28214. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25731

[jira] [Assigned] (SPARK-28214) Flaky test: org.apache.spark.streaming.CheckpointSuite.basic rdd checkpoints + dstream graph checkpoint recovery

2019-09-09 Thread Marcelo Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28214: -- Assignee: Jungtaek Lim > Flaky test:

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-09-09 Thread Fokko Driesprong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926122#comment-16926122 ] Fokko Driesprong commented on SPARK-28921: -- I can confirm that we're running into the same

[jira] [Resolved] (SPARK-28831) Document CLEAR CACHE in SQL Reference

2019-09-09 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28831. - Fix Version/s: 3.0.0 Assignee: Huaxin Gao Resolution: Fixed > Document CLEAR CACHE in

[jira] [Created] (SPARK-29026) Improve error message when constructor in `ScalaReflection` isn't found

2019-09-09 Thread Mick Jermsurawong (Jira)
Mick Jermsurawong created SPARK-29026: - Summary: Improve error message when constructor in `ScalaReflection` isn't found Key: SPARK-29026 URL: https://issues.apache.org/jira/browse/SPARK-29026

[jira] [Resolved] (SPARK-28773) NULL Handling

2019-09-09 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28773. - Fix Version/s: 3.0.0 Resolution: Fixed > NULL Handling > - > > Key:

[jira] [Assigned] (SPARK-28773) NULL Handling

2019-09-09 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-28773: --- Assignee: Dilip Biswal (was: Xiao Li) > NULL Handling > - > > Key:

[jira] [Updated] (SPARK-29011) Upgrade netty-all to 4.1.39-Final

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-29011: -- Fix Version/s: 2.4.5 > Upgrade netty-all to 4.1.39-Final > - > >

[jira] [Updated] (SPARK-14098) Generate Java code to build CachedColumnarBatch and get values from CachedColumnarBatch when DataFrame.cache() is called

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-14098: -- Labels: release-notes (was: releasenotes) > Generate Java code to build CachedColumnarBatch and get

[jira] [Resolved] (SPARK-14643) Remove overloaded methods which become ambiguous in Scala 2.12

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14643. --- Target Version/s: (was: 3.0.0) Resolution: Won't Fix Reopen if anyone has different

[jira] [Resolved] (SPARK-14098) Generate Java code to build CachedColumnarBatch and get values from CachedColumnarBatch when DataFrame.cache() is called

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-14098. --- Target Version/s: (was: 3.0.0) Resolution: Done > Generate Java code to build

[jira] [Resolved] (SPARK-16275) Implement all the Hive fallback functions

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16275. --- Target Version/s: (was: 3.0.0) Resolution: Done > Implement all the Hive fallback

[jira] [Resolved] (SPARK-16390) Dataset API improvements

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-16390. --- Target Version/s: (was: 3.0.0) Resolution: Done > Dataset API improvements >

[jira] [Resolved] (SPARK-18455) General support for correlated subquery processing

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18455. --- Target Version/s: (was: 3.0.0) Resolution: Done Looks like the subtasks are done? >

[jira] [Updated] (SPARK-22236) CSV I/O: does not respect RFC 4180

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22236: -- Target Version/s: (was: 3.0.0) > CSV I/O: does not respect RFC 4180 >

[jira] [Resolved] (SPARK-25261) Standardize the default units of spark.driver|executor.memory

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25261. --- Fix Version/s: 2.4.0 Target Version/s: (was: 3.0.0) Assignee: huangtengfei

[jira] [Updated] (SPARK-25157) Streaming of image files from directory

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25157: -- Target Version/s: (was: 3.0.0) > Streaming of image files from directory >

[jira] [Resolved] (SPARK-25783) Spark shell fails because of jline incompatibility

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-25783. --- Target Version/s: (was: 3.0.0) Resolution: Duplicate > Spark shell fails because of

[jira] [Updated] (SPARK-26247) SPIP - ML Model Extension for no-Spark MLLib Online Serving

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-26247: -- Target Version/s: (was: 3.0.0) > SPIP - ML Model Extension for no-Spark MLLib Online Serving >

[jira] [Created] (SPARK-29025) Support seconds precision by the timestamp type

2019-09-09 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-29025: -- Summary: Support seconds precision by the timestamp type Key: SPARK-29025 URL: https://issues.apache.org/jira/browse/SPARK-29025 Project: Spark Issue Type:

[jira] [Updated] (SPARK-28227) Spark can’t support TRANSFORM with aggregation

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28227: -- Target Version/s: (was: 3.0.0) > Spark can’t support TRANSFORM with aggregation >

[jira] [Updated] (SPARK-29020) Unifying behaviour between array_sort and sort_array

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-29020: -- Target Version/s: (was: 2.4.5, 3.0.0) Labels: (was: functions sql) > Unifying

[jira] [Created] (SPARK-29024) Support the `zulu` time zone

2019-09-09 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-29024: -- Summary: Support the `zulu` time zone Key: SPARK-29024 URL: https://issues.apache.org/jira/browse/SPARK-29024 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-29022) SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class

2019-09-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29022: Summary: SparkSQLCLI can not use 'ADD JAR' 's jar as Serder class (was: SparkSQLCLI can use 'ADD

[jira] [Updated] (SPARK-28840) --jar argument with spark-sql failed to load the jars to driver classpath

2019-09-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28840: Parent: SPARK-23710 Issue Type: Sub-task (was: Bug) > --jar argument with spark-sql

[jira] [Updated] (SPARK-29022) SparkSQLCLI can use 'ADD JAR' 's jar as Serder class

2019-09-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29022: Parent: SPARK-23710 Issue Type: Sub-task (was: Bug) > SparkSQLCLI can use 'ADD JAR' 's

[jira] [Updated] (SPARK-29022) SparkSQLCLI can use 'ADD JAR' 's jar as Serder class

2019-09-09 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-29022: Issue Type: Bug (was: Improvement) > SparkSQLCLI can use 'ADD JAR' 's jar as Serder class >

[jira] [Updated] (SPARK-29023) Remove extra logic of executor pod status in ExecutorPodsSnapshot.scala

2019-09-09 Thread jinxing (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinxing updated SPARK-29023: Description: After k8s sidecar support is merged into k8s 1.17

[jira] [Created] (SPARK-29023) Remove extra logic of executor pod status in ExecutorPodsSnapshot.scala

2019-09-09 Thread jinxing (Jira)
jinxing created SPARK-29023: --- Summary: Remove extra logic of executor pod status in ExecutorPodsSnapshot.scala Key: SPARK-29023 URL: https://issues.apache.org/jira/browse/SPARK-29023 Project: Spark

[jira] [Commented] (SPARK-29006) Support special date/timestamp values `infinity`/`-infinity`

2019-09-09 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925781#comment-16925781 ] Maxim Gekk commented on SPARK-29006: [~f2005...@gmail.com] Sure, you can but you will need code from

[jira] [Resolved] (SPARK-28980) Remove most remaining deprecated items since <= 2.2.0 for 3.0

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28980. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25684

[jira] [Updated] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

2019-09-09 Thread Andy Grove (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove updated SPARK-28921: --- Affects Version/s: 2.3.0 2.3.1 2.4.0

[jira] [Updated] (SPARK-28980) Remove most remaining deprecated items since <= 2.2.0 for 3.0

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28980: -- Description: Following on https://issues.apache.org/jira/browse/SPARK-25908 I'd like to propose

[jira] [Commented] (SPARK-29006) Support special date/timestamp values `infinity`/`-infinity`

2019-09-09 Thread Anurag Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925718#comment-16925718 ] Anurag Sharma commented on SPARK-29006: --- [~maxgekk] Can I work on this? > Support special

[jira] [Resolved] (SPARK-29017) JobGroup and LocalProperty not respected by PySpark

2019-09-09 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-29017. -- Resolution: Duplicate > JobGroup and LocalProperty not respected by PySpark >

[jira] [Assigned] (SPARK-28939) SQL configuration are not always propagated

2019-09-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28939: --- Assignee: Marco Gaido > SQL configuration are not always propagated >

[jira] [Resolved] (SPARK-28939) SQL configuration are not always propagated

2019-09-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28939. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25643

[jira] [Updated] (SPARK-28341) create a public API for V2SessionCatalog

2019-09-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-28341: Summary: create a public API for V2SessionCatalog (was: remove session catalog config) > create

[jira] [Commented] (SPARK-28557) Support empty select list

2019-09-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925673#comment-16925673 ] Gengliang Wang commented on SPARK-28557: Is this duplicated with

[jira] [Resolved] (SPARK-28341) remove session catalog config

2019-09-09 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28341. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25104

[jira] [Commented] (SPARK-29022) SparkSQLCLI can use 'ADD JAR' 's jar as Serder class

2019-09-09 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925669#comment-16925669 ] angerszhu commented on SPARK-29022: --- PR [https://github.com/apache/spark/pull/25729] > SparkSQLCLI

[jira] [Updated] (SPARK-29022) SparkSQLCLI can use 'ADD JAR' 's jar as Serder class

2019-09-09 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-29022: -- Description: Spark SQL CLI can't use class in jars add by SQL 'ADD JAR' {code:java} spark-sql> add

[jira] [Created] (SPARK-29022) SparkSQLCLI can use 'ADD JAR' 's jar as Serder class

2019-09-09 Thread angerszhu (Jira)
angerszhu created SPARK-29022: - Summary: SparkSQLCLI can use 'ADD JAR' 's jar as Serder class Key: SPARK-29022 URL: https://issues.apache.org/jira/browse/SPARK-29022 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29021) NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path

2019-09-09 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925603#comment-16925603 ] Kent Yao commented on SPARK-29021: -- it seems that need to configure options.path to an actual  host

[jira] [Updated] (SPARK-28969) OneVsRestModel in the py side should not set WeightCol and Classifier

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28969: -- Docs Text: The setClassifier method in Pyspark's OneVsRestModel has been removed in 3.0 for parity

[jira] [Resolved] (SPARK-28340) Noisy exceptions when tasks are killed: "DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file: java.nio.channels.ClosedByInterruptException"

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28340. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25674

[jira] [Assigned] (SPARK-28340) Noisy exceptions when tasks are killed: "DiskBlockObjectWriter: Uncaught exception while reverting partial writes to file: java.nio.channels.ClosedByInterruptException"

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-28340: - Assignee: Colin Ma > Noisy exceptions when tasks are killed: "DiskBlockObjectWriter: Uncaught

[jira] [Resolved] (SPARK-28884) [Spark]Spark driver cores is showing 0 instead of 1 in UI for cluster mode deployment

2019-09-09 Thread Sean Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28884. --- Resolution: Not A Problem > [Spark]Spark driver cores is showing 0 instead of 1 in UI for cluster

[jira] [Commented] (SPARK-28068) `lag` second argument must be a literal

2019-09-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925540#comment-16925540 ] Gengliang Wang commented on SPARK-28068: Is this ANSI SQL? From the specification 4.16 Data

[jira] [Updated] (SPARK-29021) NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path

2019-09-09 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-29021: - Description: Mount hostPath has an issue: {code:java} Exception in thread "main"

[jira] [Created] (SPARK-29021) NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path

2019-09-09 Thread Kent Yao (Jira)
Kent Yao created SPARK-29021: Summary: NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path Key: SPARK-29021 URL: https://issues.apache.org/jira/browse/SPARK-29021 Project:

[jira] [Updated] (SPARK-21869) A cached Kafka producer should not be closed if any task is using it.

2019-09-09 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated SPARK-21869: Affects Version/s: (was: 2.2.0) 3.0.0

[jira] [Updated] (SPARK-28912) MatchError exception in CheckpointWriteHandler

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28912: - Fix Version/s: 2.4.5 > MatchError exception in CheckpointWriteHandler >

[jira] [Resolved] (SPARK-23793) Handle database names in spark.udf.register()

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23793. -- Resolution: Won't Fix > Handle database names in spark.udf.register() >

[jira] [Commented] (SPARK-28984) csv files should support multi-character separators

2019-09-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925493#comment-16925493 ] Hyukjin Kwon commented on SPARK-28984: -- Can you file an issue at Univocity side and let this JIRA

[jira] [Updated] (SPARK-29000) [SQL] Decimal precision overflow when don't allow precision loss

2019-09-09 Thread feiwang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-29000: Description: When we set spark.sql.decimalOperations.allowPrecisionLoss=false. For the sql below, the

[jira] [Updated] (SPARK-28512) New optional mode: throw runtime exceptions on casting failures

2019-09-09 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-28512: --- Description: In popular DBMS like MySQL/PostgreSQL/Oracle, runtime exceptions are thrown on

[jira] [Updated] (SPARK-29020) Unifying behaviour between array_sort and sort_array

2019-09-09 Thread German Schiavon Matteo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] German Schiavon Matteo updated SPARK-29020: --- Description: I've noticed that there are two functions to sort arrays

[jira] [Created] (SPARK-29020) Unifying behaviour between array_sort and sort_array

2019-09-09 Thread German Schiavon Matteo (Jira)
German Schiavon Matteo created SPARK-29020: -- Summary: Unifying behaviour between array_sort and sort_array Key: SPARK-29020 URL: https://issues.apache.org/jira/browse/SPARK-29020 Project:

  1   2   >