[jira] [Resolved] (SPARK-29655) Enable adaptive execution should not add more ShuffleExchange

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29655. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26409

[jira] [Assigned] (SPARK-29655) Enable adaptive execution should not add more ShuffleExchange

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29655: --- Assignee: Yuming Wang > Enable adaptive execution should not add more ShuffleExchange >

[jira] [Updated] (SPARK-29911) Cache table may memory leak when session closed

2019-11-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29911: --- Summary: Cache table may memory leak when session closed (was: Cache table may memory leak when

[jira] [Updated] (SPARK-29911) Cache table may memory leak when session stopped

2019-11-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29911: --- Description: How to reproduce: 1. create a local temporary view v1 2. cache it in memory 3. close

[jira] [Updated] (SPARK-29911) Cache table may memory leak when session stopped

2019-11-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29911: --- Attachment: Screen Shot 2019-11-15 at 2.03.49 PM.png > Cache table may memory leak when session

[jira] [Updated] (SPARK-29911) Cache table may memory leak when session stopped

2019-11-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29911: --- Description: How to reproduce: 1. create a local temporary view v1 2. cache it in memory 3. close

[jira] [Created] (SPARK-29911) Cache table may memory leak when session stopped

2019-11-14 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-29911: -- Summary: Cache table may memory leak when session stopped Key: SPARK-29911 URL: https://issues.apache.org/jira/browse/SPARK-29911 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-29910) Add minimum runtime limit to speculation

2019-11-14 Thread Deegue (Jira)
Deegue created SPARK-29910: -- Summary: Add minimum runtime limit to speculation Key: SPARK-29910 URL: https://issues.apache.org/jira/browse/SPARK-29910 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974836#comment-16974836 ] koert kuipers commented on SPARK-29906: --- i added a bit of debug logging: {code:java} $ git diff

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974824#comment-16974824 ] Terry Kim commented on SPARK-29900: --- Cool. I will compile the list and send it out to dev/user list.

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974822#comment-16974822 ] Wenchen Fan commented on SPARK-29900: - Yea exactly! I don't think it's a big breaking change. We

[jira] [Commented] (SPARK-29748) Remove sorting of fields in PySpark SQL Row creation

2019-11-14 Thread Joachim Hereth (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974821#comment-16974821 ] Joachim Hereth commented on SPARK-29748: [~bryanc] With simply removing sorting we change the

[jira] [Assigned] (SPARK-29888) New interval string parser parse '.111 seconds' to null

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29888: --- Assignee: Kent Yao > New interval string parser parse '.111 seconds' to null >

[jira] [Resolved] (SPARK-29888) New interval string parser parse '.111 seconds' to null

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29888. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26514

[jira] [Resolved] (SPARK-28859) Remove value check of MEMORY_OFFHEAP_SIZE in declaration section

2019-11-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28859. --- Assignee: (was: yifan) Resolution: Invalid According to the the test failures on

[jira] [Commented] (SPARK-29892) Add built-in Array Functions: array_cat

2019-11-14 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974800#comment-16974800 ] Aman Omer commented on SPARK-29892: --- This Jira is a duplicate of 

[jira] [Updated] (SPARK-27884) Deprecate Python 2 and Python 3 prior to 3.6 support in Spark 3.0

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27884: - Description: Officially deprecate Python 2 support and and Python 3 prior to 3.6 in Spark 3.0.

[jira] [Updated] (SPARK-27884) Deprecate Python 2 and Python 3 prior to 3.6 support in Spark 3.0

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27884: - Description: Officially deprecate Python 2 support and and Python 3 prior to 3.6 in Spark 3.0.

[jira] [Resolved] (SPARK-27884) Deprecate Python 2 and Python 3 prior to 3.6 support in Spark 3.0

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27884. -- Fix Version/s: 3.0.0 Resolution: Done > Deprecate Python 2 and Python 3 prior to 3.6

[jira] [Commented] (SPARK-29803) remove all instances of 'from __future__ import print_function'

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974791#comment-16974791 ] Hyukjin Kwon commented on SPARK-29803: -- (I converted into a subtask of a new JIRA) > remove all

[jira] [Updated] (SPARK-29803) remove all instances of 'from __future__ import print_function'

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29803: - Parent: SPARK-29909 Issue Type: Sub-task (was: Bug) > remove all instances of 'from

[jira] [Commented] (SPARK-29897) Implicit cast to timestamp is failing

2019-11-14 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974789#comment-16974789 ] Ankit Raj Boudh commented on SPARK-29897: - i will raise PR for this > Implicit cast to

[jira] [Updated] (SPARK-29803) remove all instances of 'from __future__ import print_function'

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29803: - Parent: (was: SPARK-27884) Issue Type: Bug (was: Sub-task) > remove all instances

[jira] [Commented] (SPARK-29802) update remaining python scripts in repo to python3 shebang

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974788#comment-16974788 ] Hyukjin Kwon commented on SPARK-29802: -- (I converted into a subtask of a new JIRA) > update

[jira] [Updated] (SPARK-29909) Drop Python 2 and Python 3.4 and 3.5.

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29909: - Issue Type: Umbrella (was: Bug) > Drop Python 2 and Python 3.4 and 3.5. >

[jira] [Updated] (SPARK-29802) update remaining python scripts in repo to python3 shebang

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29802: - Parent: (was: SPARK-27884) Issue Type: Bug (was: Sub-task) > update remaining

[jira] [Updated] (SPARK-29802) update remaining python scripts in repo to python3 shebang

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29802: - Parent: SPARK-29909 Issue Type: Sub-task (was: Bug) > update remaining python scripts

[jira] [Created] (SPARK-29909) Drop Python 2 and Python 3.4 and 3.5.

2019-11-14 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-29909: Summary: Drop Python 2 and Python 3.4 and 3.5. Key: SPARK-29909 URL: https://issues.apache.org/jira/browse/SPARK-29909 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-28752) Documentation build script to support Python 3

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28752. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26521

[jira] [Assigned] (SPARK-28752) Documentation build script to support Python 3

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28752: Assignee: Hyukjin Kwon > Documentation build script to support Python 3 >

[jira] [Updated] (SPARK-29376) Upgrade Apache Arrow to 0.15.1

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29376: - Fix Version/s: 3.0.0 > Upgrade Apache Arrow to 0.15.1 > -- > >

[jira] [Resolved] (SPARK-29376) Upgrade Apache Arrow to 0.15.1

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29376. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/26133 > Upgrade Apache Arrow

[jira] [Created] (SPARK-29908) Add a Python, Pandas and PyArrow versions in clue at SQL query tests

2019-11-14 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-29908: Summary: Add a Python, Pandas and PyArrow versions in clue at SQL query tests Key: SPARK-29908 URL: https://issues.apache.org/jira/browse/SPARK-29908 Project: Spark

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Description: we observed an issue where spark seems to confuse a data line (not the first

[jira] [Created] (SPARK-29907) Move DELETE/UPDATE/MERGE relative rules to dmlStatementNoWith to support cte.

2019-11-14 Thread Xianyin Xin (Jira)
Xianyin Xin created SPARK-29907: --- Summary: Move DELETE/UPDATE/MERGE relative rules to dmlStatementNoWith to support cte. Key: SPARK-29907 URL: https://issues.apache.org/jira/browse/SPARK-29907 Project:

[jira] [Updated] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-29906: -- Description: we observed an issue where spark seems to confuse a data line (not the first

[jira] [Created] (SPARK-29906) Reading of csv file fails with adaptive execution turned on

2019-11-14 Thread koert kuipers (Jira)
koert kuipers created SPARK-29906: - Summary: Reading of csv file fails with adaptive execution turned on Key: SPARK-29906 URL: https://issues.apache.org/jira/browse/SPARK-29906 Project: Spark

[jira] [Commented] (SPARK-26499) JdbcUtils.makeGetter does not handle ByteType

2019-11-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974757#comment-16974757 ] Dongjoon Hyun commented on SPARK-26499: --- This is backported to branch-2.4 via

[jira] [Updated] (SPARK-26499) JdbcUtils.makeGetter does not handle ByteType

2019-11-14 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26499: -- Fix Version/s: 2.4.5 > JdbcUtils.makeGetter does not handle ByteType >

[jira] [Resolved] (SPARK-28602) Recognize interval as a numeric type

2019-11-14 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-28602. -- Fix Version/s: 3.0.0 Target Version/s: 3.0.0 Resolution: Duplicate > Recognize

[jira] [Resolved] (SPARK-29889) unify the interval tests

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29889. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26515

[jira] [Resolved] (SPARK-29901) Fix broken links in SQL Reference

2019-11-14 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29901. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26528

[jira] [Created] (SPARK-29905) ExecutorPodsLifecycleManager has sub-optimal behavior with dynamic allocation

2019-11-14 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-29905: -- Summary: ExecutorPodsLifecycleManager has sub-optimal behavior with dynamic allocation Key: SPARK-29905 URL: https://issues.apache.org/jira/browse/SPARK-29905

[jira] [Comment Edited] (SPARK-25185) CBO rowcount statistics doesn't work for partitioned parquet external table

2019-11-14 Thread venkata yerubandi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974724#comment-16974724 ] venkata yerubandi edited comment on SPARK-25185 at 11/15/19 1:21 AM: -

[jira] [Commented] (SPARK-25185) CBO rowcount statistics doesn't work for partitioned parquet external table

2019-11-14 Thread venkata yerubandi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974724#comment-16974724 ] venkata yerubandi commented on SPARK-25185: --- Is there any update on this issue ? we are facing

[jira] [Resolved] (SPARK-29857) [WEB UI] Support defer render the spark history summary page.

2019-11-14 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29857. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26482

[jira] [Assigned] (SPARK-29857) [WEB UI] Support defer render the spark history summary page.

2019-11-14 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29857: Assignee: feiwang > [WEB UI] Support defer render the spark history summary page. >

[jira] [Commented] (SPARK-29748) Remove sorting of fields in PySpark SQL Row creation

2019-11-14 Thread Bryan Cutler (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974678#comment-16974678 ] Bryan Cutler commented on SPARK-29748: -- Thanks for discussing [~zero323] . The goal here is to only

[jira] [Assigned] (SPARK-29865) k8s executor pods all have different prefixes in client mode

2019-11-14 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson reassigned SPARK-29865: -- Assignee: Marcelo Masiero Vanzin > k8s executor pods all have different prefixes in

[jira] [Resolved] (SPARK-29865) k8s executor pods all have different prefixes in client mode

2019-11-14 Thread Erik Erlandson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Erlandson resolved SPARK-29865. Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26488

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974655#comment-16974655 ] Terry Kim commented on SPARK-29900: --- If we make the relation lookup behavior consistent such that 1)

[jira] [Comment Edited] (SPARK-29884) spark-submit to kuberentes can not parse valid ca certificate

2019-11-14 Thread Jeremy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974621#comment-16974621 ] Jeremy edited comment on SPARK-29884 at 11/14/19 9:34 PM: -- After doing some

[jira] [Commented] (SPARK-29884) spark-submit to kuberentes can not parse valid ca certificate

2019-11-14 Thread Jeremy (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974621#comment-16974621 ] Jeremy commented on SPARK-29884: After doing some debugging it seams like this might be in fabric k8s

[jira] [Updated] (SPARK-28833) Document ALTER VIEW Statement in SQL Reference.

2019-11-14 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-28833: - Priority: Minor (was: Major) > Document ALTER VIEW Statement in SQL Reference. >

[jira] [Resolved] (SPARK-28833) Document ALTER VIEW Statement in SQL Reference.

2019-11-14 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-28833. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25573

[jira] [Assigned] (SPARK-28833) Document ALTER VIEW Statement in SQL Reference.

2019-11-14 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-28833: Assignee: kevin yu > Document ALTER VIEW Statement in SQL Reference. >

[jira] [Created] (SPARK-29904) Parse timestamps in microsecond precision by JSON/CSV datasources

2019-11-14 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-29904: -- Summary: Parse timestamps in microsecond precision by JSON/CSV datasources Key: SPARK-29904 URL: https://issues.apache.org/jira/browse/SPARK-29904 Project: Spark

[jira] [Commented] (SPARK-29903) Add documentation for recursiveFileLookup

2019-11-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974577#comment-16974577 ] Nicholas Chammas commented on SPARK-29903: -- cc [~cloud_fan] and [~weichenxu123] > Add

[jira] [Created] (SPARK-29903) Add documentation for recursiveFileLookup

2019-11-14 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-29903: Summary: Add documentation for recursiveFileLookup Key: SPARK-29903 URL: https://issues.apache.org/jira/browse/SPARK-29903 Project: Spark Issue

[jira] [Created] (SPARK-29902) Add listener event queue capacity configuration to documentation

2019-11-14 Thread shahid (Jira)
shahid created SPARK-29902: -- Summary: Add listener event queue capacity configuration to documentation Key: SPARK-29902 URL: https://issues.apache.org/jira/browse/SPARK-29902 Project: Spark Issue

[jira] [Commented] (SPARK-29691) Estimator fit method fails to copy params (in PySpark)

2019-11-14 Thread John Bauer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974527#comment-16974527 ] John Bauer commented on SPARK-29691: [[SPARK-29691] ensure Param objects are valid in fit,

[jira] [Resolved] (SPARK-29672) update spark testing framework to use python3

2019-11-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp resolved SPARK-29672. - Resolution: Fixed > update spark testing framework to use python3 >

[jira] [Updated] (SPARK-29672) update spark testing framework to use python3

2019-11-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29672: Description: python 2.7 is EOL jan 1st 2020: [https://github.com/python/devguide/pull/344] it's

[jira] [Updated] (SPARK-29672) update spark testing framework to use python3

2019-11-14 Thread Shane Knapp (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Knapp updated SPARK-29672: Summary: update spark testing framework to use python3 (was: remove python2 tests and test

[jira] [Created] (SPARK-29901) Fix broken links in SQL Reference

2019-11-14 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-29901: -- Summary: Fix broken links in SQL Reference Key: SPARK-29901 URL: https://issues.apache.org/jira/browse/SPARK-29901 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Ryan Blue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974456#comment-16974456 ] Ryan Blue commented on SPARK-29900: --- To be clear, we think this is going to be a breaking change,

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974436#comment-16974436 ] Burak Yavuz commented on SPARK-29900: - I definitely agree the behavior is very confusing here. (For

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974433#comment-16974433 ] Terry Kim commented on SPARK-29900: --- Yes. Thanks [~cloud_fan] > make relation lookup behavior

[jira] [Updated] (SPARK-29899) Can not recursively lookup files in Hive table via SQL

2019-11-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29899: --- Description: SPARK-27990 provide a way to recursively load data from datasource. In SQL, when query

[jira] [Updated] (SPARK-29899) Can not recursively lookup files in Hive table via SQL

2019-11-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29899: --- Summary: Can not recursively lookup files in Hive table via SQL (was: Can not set

[jira] [Updated] (SPARK-29899) Can not set recursiveFileLookup property in SQL

2019-11-14 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-29899: --- Summary: Can not set recursiveFileLookup property in SQL (was: Can not set recursiveFileLookup

[jira] [Commented] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974352#comment-16974352 ] Wenchen Fan commented on SPARK-29900: - [~imback82] do you want to drive it? also cc [~rdblue]

[jira] [Updated] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-29900: Description: Currently, Spark has 2 different relation resolution behaviors: 1. try to look up

[jira] [Created] (SPARK-29900) make relation lookup behavior consistent within Spark SQL

2019-11-14 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-29900: --- Summary: make relation lookup behavior consistent within Spark SQL Key: SPARK-29900 URL: https://issues.apache.org/jira/browse/SPARK-29900 Project: Spark

[jira] [Created] (SPARK-29899) Can not set recursiveFileLookup property in TBLPROPERTIES if file format is Parquet

2019-11-14 Thread Lantao Jin (Jira)
Lantao Jin created SPARK-29899: -- Summary: Can not set recursiveFileLookup property in TBLPROPERTIES if file format is Parquet Key: SPARK-29899 URL: https://issues.apache.org/jira/browse/SPARK-29899

[jira] [Updated] (SPARK-29898) Support Avro Custom Logical Types

2019-11-14 Thread Carlos del Prado Mota (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos del Prado Mota updated SPARK-29898: -- Description: Extends options for the Spark Avro formatter allowing to use

[jira] [Updated] (SPARK-29898) Support Avro Custom Logical Types

2019-11-14 Thread Carlos del Prado Mota (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos del Prado Mota updated SPARK-29898: -- Description: Extends options for the Spark Avro formatter allowing to use

[jira] [Updated] (SPARK-29898) Support Avro Custom Logical Types

2019-11-14 Thread Carlos del Prado Mota (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos del Prado Mota updated SPARK-29898: -- Description: Extends options for the Spark Avro formatter allowing to use

[jira] [Updated] (SPARK-29898) Support Avro Custom Logical Types

2019-11-14 Thread Carlos del Prado Mota (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos del Prado Mota updated SPARK-29898: -- Description: Extends options for the Spark Avro formatter allowing to use

[jira] [Updated] (SPARK-29898) Support Avro Custom Logical Types

2019-11-14 Thread Carlos del Prado Mota (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos del Prado Mota updated SPARK-29898: -- Description: Extends options for the Spark Avro formatter allowing to use

[jira] [Updated] (SPARK-29898) Support Avro Custom Logical Types

2019-11-14 Thread Carlos del Prado Mota (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlos del Prado Mota updated SPARK-29898: -- Description: Extends options for the Spark Avro formatter allowing to use

[jira] [Created] (SPARK-29898) Support Avro Custom Logical Types

2019-11-14 Thread Carlos del Prado Mota (Jira)
Carlos del Prado Mota created SPARK-29898: - Summary: Support Avro Custom Logical Types Key: SPARK-29898 URL: https://issues.apache.org/jira/browse/SPARK-29898 Project: Spark Issue

[jira] [Created] (SPARK-29897) Implicit cast to timestamp is failing

2019-11-14 Thread ABHISHEK KUMAR GUPTA (Jira)
ABHISHEK KUMAR GUPTA created SPARK-29897: Summary: Implicit cast to timestamp is failing Key: SPARK-29897 URL: https://issues.apache.org/jira/browse/SPARK-29897 Project: Spark Issue

[jira] [Commented] (SPARK-20110) Windowed aggregation do not work when the timestamp is a nested field

2019-11-14 Thread hurelhuyag (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974177#comment-16974177 ] hurelhuyag commented on SPARK-20110: I just faced same problem now. It's spark version 2.4.4. I

[jira] [Comment Edited] (SPARK-20110) Windowed aggregation do not work when the timestamp is a nested field

2019-11-14 Thread hurelhuyag (Jira)
[ https://issues.apache.org/jira/browse/SPARK-20110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974177#comment-16974177 ] hurelhuyag edited comment on SPARK-20110 at 11/14/19 12:09 PM: --- I just

[jira] [Created] (SPARK-29896) Extend typed literals support for all spark native types

2019-11-14 Thread Kent Yao (Jira)
Kent Yao created SPARK-29896: Summary: Extend typed literals support for all spark native types Key: SPARK-29896 URL: https://issues.apache.org/jira/browse/SPARK-29896 Project: Spark Issue Type:

[jira] [Created] (SPARK-29895) Extend typed literals support for all spark native types

2019-11-14 Thread Kent Yao (Jira)
Kent Yao created SPARK-29895: Summary: Extend typed literals support for all spark native types Key: SPARK-29895 URL: https://issues.apache.org/jira/browse/SPARK-29895 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21040) On executor/worker decommission consider speculatively re-launching current tasks

2019-11-14 Thread Prakhar Jain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974162#comment-16974162 ] Prakhar Jain commented on SPARK-21040: -- Hi [~holden], At Microsoft, we are also facing same issues

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-11-14 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974143#comment-16974143 ] huangtianhua commented on SPARK-29106: -- [~shaneknapp], the vm is ready, I have build/test in

[jira] [Updated] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-14 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-29894: Attachment: snippet_plan_graph_before_patch.png > Add Codegen Stage Id to Spark plan graphs in

[jira] [Updated] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-14 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-29894: Attachment: snippet__plan_graph_with_Codegen_Stage_Id_Annotated.png > Add Codegen Stage Id to

[jira] [Created] (SPARK-29894) Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab

2019-11-14 Thread Luca Canali (Jira)
Luca Canali created SPARK-29894: --- Summary: Add Codegen Stage Id to Spark plan graphs in Web UI SQL Tab Key: SPARK-29894 URL: https://issues.apache.org/jira/browse/SPARK-29894 Project: Spark

[jira] [Created] (SPARK-29893) Improve the local reader performance by changing the task number from 1 to multi

2019-11-14 Thread Ke Jia (Jira)
Ke Jia created SPARK-29893: -- Summary: Improve the local reader performance by changing the task number from 1 to multi Key: SPARK-29893 URL: https://issues.apache.org/jira/browse/SPARK-29893 Project: Spark

[jira] [Updated] (SPARK-29892) Add built-in Array Functions: array_cat

2019-11-14 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-29892: --- Description: |{{array_cat}}{{(}}{{anyarray}}{{, }}{{anyarray}}{{)}}|{{anyarray}}|concatenate two

[jira] [Commented] (SPARK-29892) Add built-in Array Functions: array_cat

2019-11-14 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974089#comment-16974089 ] jiaan.geng commented on SPARK-29892: I'm working on. > Add built-in Array Functions: array_cat >

[jira] [Created] (SPARK-29892) Add built-in Array Functions: array_cat

2019-11-14 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-29892: -- Summary: Add built-in Array Functions: array_cat Key: SPARK-29892 URL: https://issues.apache.org/jira/browse/SPARK-29892 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-29890) Unable to fill na with 0 with duplicate columns

2019-11-14 Thread sandeshyapuram (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974087#comment-16974087 ] sandeshyapuram edited comment on SPARK-29890 at 11/14/19 9:36 AM: -- I've

[jira] [Commented] (SPARK-29890) Unable to fill na with 0 with duplicate columns

2019-11-14 Thread sandeshyapuram (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974087#comment-16974087 ] sandeshyapuram commented on SPARK-29890: I've raised it as a bug because I feel fill.na(0) needs

[jira] [Commented] (SPARK-29891) Add built-in Array Functions: array_length

2019-11-14 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16974086#comment-16974086 ] jiaan.geng commented on SPARK-29891: I'm working on > Add built-in Array Functions: array_length >

[jira] [Updated] (SPARK-29891) Add built-in Array Functions: array_length

2019-11-14 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-29891: --- Description: |{{array_length}}{{(}}{{anyarray}}{{, }}{{int}}{{)}}|{{int}}|returns the length of

  1   2   >