[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Docs Text: In Spark 3.0, the type of "error" parameter in the

[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Docs Text: In Spark 3.0, the type of "error" parameter in the

[jira] [Commented] (SPARK-27602) SparkSQL CBO can't get true size of partition table after partition pruning

2019-07-29 Thread angerszhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895737#comment-16895737 ] angerszhu commented on SPARK-27602: --- Seems in hive 1.2.1, we can't get true partition level stats too.

[jira] [Commented] (SPARK-27602) SparkSQL CBO can't get true size of partition table after partition pruning

2019-07-29 Thread angerszhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895736#comment-16895736 ] angerszhu commented on SPARK-27602: --- [~lishuming] It's hard to combine several  partition's stats

[jira] [Updated] (SPARK-28560) Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-07-29 Thread Ke Jia (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ke Jia updated SPARK-28560: --- Description: Implement a rule in the new adaptive execution framework introduced in SPARK-23128. This rule

[jira] [Created] (SPARK-28560) Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution

2019-07-29 Thread Ke Jia (JIRA)
Ke Jia created SPARK-28560: -- Summary: Optimize shuffle reader to local shuffle reader when smj converted to bhj in adaptive execution Key: SPARK-28560 URL: https://issues.apache.org/jira/browse/SPARK-28560

[jira] [Created] (SPARK-28559) when execute "create table" on ThriftServer ,the hive table owner is wrong

2019-07-29 Thread yibo hu (JIRA)
yibo hu created SPARK-28559: --- Summary: when execute "create table" on ThriftServer ,the hive table owner is wrong Key: SPARK-28559 URL: https://issues.apache.org/jira/browse/SPARK-28559 Project: Spark

[jira] [Commented] (SPARK-25185) CBO rowcount statistics doesn't work for partitioned parquet external table

2019-07-29 Thread ShuMing Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895726#comment-16895726 ] ShuMing Li commented on SPARK-25185: Does Spark support `External Table`'s Statistics Analyze?  >

[jira] [Commented] (SPARK-27602) SparkSQL CBO can't get true size of partition table after partition pruning

2019-07-29 Thread ShuMing Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895725#comment-16895725 ] ShuMing Li commented on SPARK-27602: What's the progress of this problem? Is there someone doing

[jira] [Updated] (SPARK-28558) DatasetWriter partitionBy is changing the group file permissions in 2.4 for parquets

2019-07-29 Thread Stephen Pearson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Pearson updated SPARK-28558: Description: When writing a parquet using partitionBy the group file permissions are

[jira] [Created] (SPARK-28558) DatasetWriter partitionBy is changing the group file permissions in 2.4 for parquets

2019-07-29 Thread Stephen Pearson (JIRA)
Stephen Pearson created SPARK-28558: --- Summary: DatasetWriter partitionBy is changing the group file permissions in 2.4 for parquets Key: SPARK-28558 URL: https://issues.apache.org/jira/browse/SPARK-28558

[jira] [Resolved] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28556. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25292

[jira] [Updated] (SPARK-28298) Fully support char and varchar types

2019-07-29 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-28298: Summary: Fully support char and varchar types (was: Strange behavior of CAST string to

[jira] [Created] (SPARK-28557) Support empty select list

2019-07-29 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-28557: --- Summary: Support empty select list Key: SPARK-28557 URL: https://issues.apache.org/jira/browse/SPARK-28557 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-28553) subqueries must be aggregated before hand

2019-07-29 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895703#comment-16895703 ] Yuming Wang commented on SPARK-28553: - Sorry [~DylanGuedes]. Duplicate with SPARK-28379?   >

[jira] [Comment Edited] (SPARK-28481) More expressions should extend NullIntolerant

2019-07-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895617#comment-16895617 ] Josh Rosen edited comment on SPARK-28481 at 7/29/19 9:58 PM: - {quote}Just a

[jira] [Commented] (SPARK-28481) More expressions should extend NullIntolerant

2019-07-29 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895617#comment-16895617 ] Josh Rosen commented on SPARK-28481: {quote}Just a suggestion though, we cannot use a style checker

[jira] [Commented] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-29 Thread Ayan Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895599#comment-16895599 ] Ayan Mukherjee commented on SPARK-28522: Your point is well understood but I have searched a

[jira] [Updated] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-28556: - Labels: release-notes (was: ) > Error should also be sent to QueryExecutionListener.onFailure

[jira] [Created] (SPARK-28556) Error should also be sent to QueryExecutionListener.onFailure

2019-07-29 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-28556: Summary: Error should also be sent to QueryExecutionListener.onFailure Key: SPARK-28556 URL: https://issues.apache.org/jira/browse/SPARK-28556 Project: Spark

[jira] [Commented] (SPARK-28502) Error with struct conversion while using pandas_udf

2019-07-29 Thread Blake C (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895562#comment-16895562 ] Blake C commented on SPARK-28502: -  too am having the same issue and am interested in how I can perform

[jira] [Updated] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

2019-07-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-27495: -- Summary: SPIP: Support Stage level resource configuration and scheduling (was: Support Stage

[jira] [Updated] (SPARK-28036) Support negative length at LEFT/RIGHT SQL functions

2019-07-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28036: -- Summary: Support negative length at LEFT/RIGHT SQL functions (was: Support negative

[jira] [Updated] (SPARK-28036) Support negative parameters at LEFT/RIGHT SQL functions

2019-07-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-28036: -- Summary: Support negative parameters at LEFT/RIGHT SQL functions (was: Built-in udf

[jira] [Resolved] (SPARK-28459) Date/Time Functions: make_timestamp

2019-07-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28459. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25220

[jira] [Assigned] (SPARK-28459) Date/Time Functions: make_timestamp

2019-07-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28459: - Assignee: Maxim Gekk > Date/Time Functions: make_timestamp >

[jira] [Assigned] (SPARK-28550) Unset SPARK_HOME environment variable in K8S integration preparation

2019-07-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-28550: - Assignee: Hyukjin Kwon > Unset SPARK_HOME environment variable in K8S integration

[jira] [Resolved] (SPARK-28550) Unset SPARK_HOME environment variable in K8S integration preparation

2019-07-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-28550. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25283

[jira] [Assigned] (SPARK-28042) Support mapping spark.local.dir to hostPath volume

2019-07-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-28042: -- Assignee: Junjie Chen > Support mapping spark.local.dir to hostPath volume >

[jira] [Resolved] (SPARK-28042) Support mapping spark.local.dir to hostPath volume

2019-07-29 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-28042. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24879

[jira] [Commented] (SPARK-28555) Recover options and properties and pass them back into the v1 API

2019-07-29 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895470#comment-16895470 ] Xin Ren commented on SPARK-28555: - I'm working on it :) > Recover options and properties and pass them

[jira] [Updated] (SPARK-28555) Recover options and properties and pass them back into the v1 API

2019-07-29 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Ren updated SPARK-28555: Issue Type: Sub-task (was: Improvement) Parent: SPARK-22386 > Recover options and properties and

[jira] [Created] (SPARK-28555) Recover options and properties and pass them back into the v1 API

2019-07-29 Thread Xin Ren (JIRA)
Xin Ren created SPARK-28555: --- Summary: Recover options and properties and pass them back into the v1 API Key: SPARK-28555 URL: https://issues.apache.org/jira/browse/SPARK-28555 Project: Spark

[jira] [Closed] (SPARK-28139) DataSourceV2: Add AlterTable v2 implementation

2019-07-29 Thread Xin Ren (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xin Ren closed SPARK-28139. --- > DataSourceV2: Add AlterTable v2 implementation > -- > >

[jira] [Updated] (SPARK-25474) Support `spark.sql.statistics.fallBackToHdfs` in data source tables

2019-07-29 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25474: -- Fix Version/s: 2.4.4 2.3.4 > Support `spark.sql.statistics.fallBackToHdfs`

[jira] [Commented] (SPARK-28155) do not leak SaveMode to file source v2

2019-07-29 Thread Ido Michael (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895354#comment-16895354 ] Ido Michael commented on SPARK-28155: - Hi, Why was it reopened? What's left to do?   > do not

[jira] [Created] (SPARK-28554) implement basic catalog functionalities

2019-07-29 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-28554: --- Summary: implement basic catalog functionalities Key: SPARK-28554 URL: https://issues.apache.org/jira/browse/SPARK-28554 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28519: -- Docs Text: The result of the JVM's Math.log and Math.pow may vary across platforms. In Spark 3.0, the

[jira] [Commented] (SPARK-28519) Tests failed on aarch64 due the value of math.log and power function is different

2019-07-29 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895279#comment-16895279 ] Sean Owen commented on SPARK-28519: --- Interesting thread from a JDK port list; it does seem like the

[jira] [Updated] (SPARK-28553) subqueries must be aggregated before hand

2019-07-29 Thread Dylan Guedes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-28553: - Summary: subqueries must be aggregated before hand (was: subqueries always must be aggregated

[jira] [Updated] (SPARK-28553) subqueries always must be aggregated before hand

2019-07-29 Thread Dylan Guedes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-28553: - Description: Looks like Spark subqueries does not work well with variables and values from

[jira] [Updated] (SPARK-28553) subqueries always must be aggregated before hand

2019-07-29 Thread Dylan Guedes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-28553: - Summary: subqueries always must be aggregated before hand (was: subqueries always must be

[jira] [Created] (SPARK-28553) subqueries always must be aggregated before

2019-07-29 Thread Dylan Guedes (JIRA)
Dylan Guedes created SPARK-28553: Summary: subqueries always must be aggregated before Key: SPARK-28553 URL: https://issues.apache.org/jira/browse/SPARK-28553 Project: Spark Issue Type:

[jira] [Commented] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895153#comment-16895153 ] Hyukjin Kwon commented on SPARK-28546: -- Because it looks a question that should go to the mailing

[jira] [Commented] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895148#comment-16895148 ] Hyukjin Kwon commented on SPARK-28522: -- Please ask questions to mailing list and research it by

[jira] [Assigned] (SPARK-28086) Adds `random()` sql function

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28086: Assignee: Dongjoon Hyun > Adds `random()` sql function > >

[jira] [Updated] (SPARK-28551) CTAS SQL with LOCATION properties won't check location path, cause data under this location lose

2019-07-29 Thread angerszhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-28551: -- Description: When we run SQL like  {code:java} CRETE TABLE TBL LOCATION 'PATH_URI' AS SELECT

[jira] [Resolved] (SPARK-28086) Adds `random()` sql function

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28086. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25282

[jira] [Created] (SPARK-28552) The URL prefix lowercase of MySQL is not necessary, but it is necessary in spark

2019-07-29 Thread teeyog (JIRA)
teeyog created SPARK-28552: -- Summary: The URL prefix lowercase of MySQL is not necessary, but it is necessary in spark Key: SPARK-28552 URL: https://issues.apache.org/jira/browse/SPARK-28552 Project: Spark

[jira] [Updated] (SPARK-28551) CTAS SQL with LOCATION properties won't check location path, cause data under this location lose

2019-07-29 Thread angerszhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu updated SPARK-28551: -- Labels: SQL (was: ) > CTAS SQL with LOCATION properties won't check location path, cause data under

[jira] [Created] (SPARK-28551) CTAS SQL with LOCATION properties won't check location path, cause data under this location lose

2019-07-29 Thread angerszhu (JIRA)
angerszhu created SPARK-28551: - Summary: CTAS SQL with LOCATION properties won't check location path, cause data under this location lose Key: SPARK-28551 URL: https://issues.apache.org/jira/browse/SPARK-28551

[jira] [Commented] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-29 Thread Ayan Mukherjee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895069#comment-16895069 ] Ayan Mukherjee commented on SPARK-28522: Please share examples or links to refer.   > Pass

[jira] [Commented] (SPARK-28546) Why does the File Sink operation of Spark 2.4 Structured Streaming include double-level version validation?

2019-07-29 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895031#comment-16895031 ] tommy duan commented on SPARK-28546: [~hyukjin.kwon] I'm sorry. But why closed this topic? Isn't

[jira] [Commented] (SPARK-28522) Pass dynamic parameters to custom file input format

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16895023#comment-16895023 ] Hyukjin Kwon commented on SPARK-28522: -- You can set and get the custom configurations via Hadoop

[jira] [Resolved] (SPARK-28534) Update node affinity for DockerForDesktop backend in PVTestsSuite

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28534. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 25269

[jira] [Assigned] (SPARK-28534) Update node affinity for DockerForDesktop backend in PVTestsSuite

2019-07-29 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28534: Assignee: Dongjoon Hyun > Update node affinity for DockerForDesktop backend in

[jira] [Created] (SPARK-28550) Unset SPARK_HOME environment variable in K8S integration preparation

2019-07-29 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-28550: Summary: Unset SPARK_HOME environment variable in K8S integration preparation Key: SPARK-28550 URL: https://issues.apache.org/jira/browse/SPARK-28550 Project: Spark