[jira] [Updated] (SPARK-31432) bin/sbin scripts should allow to customize jars dir

[ https://issues.apache.org/jira/browse/SPARK-31432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31432: -- Affects Version/s: (was: 2.4.5) (was: 3.0.0)

[jira] [Assigned] (SPARK-30953) InsertAdaptiveSparkPlan should apply AQE on child plan of write commands

[ https://issues.apache.org/jira/browse/SPARK-30953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30953: --- Assignee: wuyi > InsertAdaptiveSparkPlan should apply AQE on child plan of write commands

[jira] [Resolved] (SPARK-30953) InsertAdaptiveSparkPlan should apply AQE on child plan of write commands

[ https://issues.apache.org/jira/browse/SPARK-30953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30953. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27701

[jira] [Assigned] (SPARK-31441) Support duplicated column names for toPandas with Arrow execution.

[ https://issues.apache.org/jira/browse/SPARK-31441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31441: Assignee: Takuya Ueshin > Support duplicated column names for toPandas with Arrow

[jira] [Resolved] (SPARK-31441) Support duplicated column names for toPandas with Arrow execution.

[ https://issues.apache.org/jira/browse/SPARK-31441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31441. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28210

[jira] [Updated] (SPARK-31392) Support CalendarInterval to be reflect to CalendarIntervalType

[ https://issues.apache.org/jira/browse/SPARK-31392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31392: Fix Version/s: (was: 3.1.0) 3.0.0 > Support CalendarInterval to be reflect

[jira] [Assigned] (SPARK-31426) Regression in loading/saving timestamps from/to ORC files

[ https://issues.apache.org/jira/browse/SPARK-31426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31426: --- Assignee: Maxim Gekk > Regression in loading/saving timestamps from/to ORC files >

[jira] [Resolved] (SPARK-31426) Regression in loading/saving timestamps from/to ORC files

[ https://issues.apache.org/jira/browse/SPARK-31426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31426. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28189

[jira] [Updated] (SPARK-31441) Support duplicated column names for toPandas with Arrow execution.

[ https://issues.apache.org/jira/browse/SPARK-31441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-31441: -- Summary: Support duplicated column names for toPandas with Arrow execution. (was: Support

[jira] [Created] (SPARK-31441) Support duplicated column names for toPandas with arrow execution.

Takuya Ueshin created SPARK-31441: - Summary: Support duplicated column names for toPandas with arrow execution. Key: SPARK-31441 URL: https://issues.apache.org/jira/browse/SPARK-31441 Project: Spark

[jira] [Assigned] (SPARK-31434) Drop builtin function pages from SQL references

[ https://issues.apache.org/jira/browse/SPARK-31434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31434: Assignee: Takeshi Yamamuro > Drop builtin function pages from SQL references >

[jira] [Resolved] (SPARK-31434) Drop builtin function pages from SQL references

[ https://issues.apache.org/jira/browse/SPARK-31434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31434. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28203

[jira] [Resolved] (SPARK-31411) Show submitted time and duration in job details page

[ https://issues.apache.org/jira/browse/SPARK-31411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-31411. Fix Version/s: 3.1.0 Resolution: Fixed The issue is resolved in

[jira] [Commented] (SPARK-31420) Infinite timeline redraw in job details page

[ https://issues.apache.org/jira/browse/SPARK-31420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082743#comment-17082743 ] Dongjoon Hyun commented on SPARK-31420: --- Thank you for confirming, [~sarutak]! > Infinite

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082736#comment-17082736 ] Erik Krogen commented on SPARK-22148: - For future folks: the JIRA created for the issue is

[jira] [Updated] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31418: -- Issue Type: Improvement (was: Bug) > Blacklisting feature aborts Spark job without retrying

[jira] [Commented] (SPARK-31418) Blacklisting feature aborts Spark job without retrying for max num retries in case of Dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-31418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082731#comment-17082731 ] Venkata krishnan Sowrirajan commented on SPARK-31418: - [~tgraves] Currently, I'm

[jira] [Assigned] (SPARK-31306) rand() function documentation suggests an inclusive upper bound of 1.0

[ https://issues.apache.org/jira/browse/SPARK-31306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-31306: Assignee: Ben > rand() function documentation suggests an inclusive upper bound of 1.0 >

[jira] [Updated] (SPARK-31440) Improve SQL Rest API

[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Description: SQL Rest API exposes query execution metrics as Public API. This Jira

[jira] [Updated] (SPARK-31440) Improve SQL Rest API

[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Attachment: current_version.json > Improve SQL Rest API > > >

[jira] [Updated] (SPARK-31440) Improve SQL Rest API

[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Attachment: improved_version.json > Improve SQL Rest API > > >

[jira] [Updated] (SPARK-31440) Improve SQL Rest API

[ https://issues.apache.org/jira/browse/SPARK-31440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eren Avsarogullari updated SPARK-31440: --- Description: SQL Rest API exposes query execution metrics as Public API. This Jira

[jira] [Created] (SPARK-31440) Improve SQL Rest API

Eren Avsarogullari created SPARK-31440: -- Summary: Improve SQL Rest API Key: SPARK-31440 URL: https://issues.apache.org/jira/browse/SPARK-31440 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-18299) Allow more aggregations on KeyValueGroupedDataset

[ https://issues.apache.org/jira/browse/SPARK-18299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-18299: --- Assignee: nooberfsh > Allow more aggregations on KeyValueGroupedDataset >

[jira] [Created] (SPARK-31439) Perf regression of fromJavaDate

Maxim Gekk created SPARK-31439: -- Summary: Perf regression of fromJavaDate Key: SPARK-31439 URL: https://issues.apache.org/jira/browse/SPARK-31439 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-31426) Regression in loading/saving timestamps from/to ORC files

[ https://issues.apache.org/jira/browse/SPARK-31426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31426: --- Parent: SPARK-31404 Issue Type: Sub-task (was: Bug) > Regression in loading/saving

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082498#comment-17082498 ] Bruce Robbins commented on SPARK-31423: --- [~cloud_fan]  {quote}FYI this is the behavior of Spark

[jira] [Updated] (SPARK-31438) Support JobCleaned Status in SparkListener

[ https://issues.apache.org/jira/browse/SPARK-31438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jackey Lee updated SPARK-31438: --- Description: In Spark, we need do some hook after job cleaned, such as cleaning hive external

[jira] [Updated] (SPARK-31438) Support JobCleaned Status in SparkListener

[ https://issues.apache.org/jira/browse/SPARK-31438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jackey Lee updated SPARK-31438: --- Description: In Spark, we need do some hook, such as cleaning hive external temporary paths, after

[jira] [Created] (SPARK-31438) Support JobCleaned Status in SparkListener

Jackey Lee created SPARK-31438: -- Summary: Support JobCleaned Status in SparkListener Key: SPARK-31438 URL: https://issues.apache.org/jira/browse/SPARK-31438 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-31391) Add AdaptiveTestUtils to ease the test of AQE

[ https://issues.apache.org/jira/browse/SPARK-31391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31391. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28162

[jira] [Assigned] (SPARK-31391) Add AdaptiveTestUtils to ease the test of AQE

[ https://issues.apache.org/jira/browse/SPARK-31391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31391: --- Assignee: wuyi > Add AdaptiveTestUtils to ease the test of AQE >

[jira] [Resolved] (SPARK-31409) Fix failed tests due to result order changing when we enable AQE

[ https://issues.apache.org/jira/browse/SPARK-31409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31409. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28178

[jira] [Assigned] (SPARK-31409) Fix failed tests due to result order changing when we enable AQE

[ https://issues.apache.org/jira/browse/SPARK-31409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31409: --- Assignee: wuyi > Fix failed tests due to result order changing when we enable AQE >

[jira] [Resolved] (SPARK-31435) Add SPARK_JARS_DIR enviroment variable (new) to Spark configuration documentation

[ https://issues.apache.org/jira/browse/SPARK-31435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-31435. -- Resolution: Duplicate > Add SPARK_JARS_DIR enviroment variable (new) to Spark configuration

[jira] [Created] (SPARK-31437) Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied

Hongze Zhang created SPARK-31437: Summary: Try assigning tasks to existing executors by which required resources in ResourceProfile are satisfied Key: SPARK-31437 URL:

[jira] [Created] (SPARK-31436) MinHash keyDistance optimization

zhengruifeng created SPARK-31436: Summary: MinHash keyDistance optimization Key: SPARK-31436 URL: https://issues.apache.org/jira/browse/SPARK-31436 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-31435) Add SPARK_JARS_DIR enviroment variable (new) to Spark configuration documentation

Pablo Langa Blanco created SPARK-31435: -- Summary: Add SPARK_JARS_DIR enviroment variable (new) to Spark configuration documentation Key: SPARK-31435 URL: https://issues.apache.org/jira/browse/SPARK-31435

[jira] [Commented] (SPARK-31435) Add SPARK_JARS_DIR enviroment variable (new) to Spark configuration documentation

[ https://issues.apache.org/jira/browse/SPARK-31435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082233#comment-17082233 ] Pablo Langa Blanco commented on SPARK-31435: I'm working on this > Add SPARK_JARS_DIR

[jira] [Comment Edited] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082228#comment-17082228 ] Wenchen Fan edited comment on SPARK-31423 at 4/13/20, 10:54 AM: FYI this

[jira] [Commented] (SPARK-31423) DATES and TIMESTAMPS for a certain range are off by 10 days when stored in ORC

[ https://issues.apache.org/jira/browse/SPARK-31423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082228#comment-17082228 ] Wenchen Fan commented on SPARK-31423: - FYI this is the behavior of Spark 2.4: ``` scala> val df =

[jira] [Resolved] (SPARK-31407) Fix hive/SQLQuerySuite.derived from Hive query file: drop_database_removes_partition_dirs.q

[ https://issues.apache.org/jira/browse/SPARK-31407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31407. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31429: - Parent: (was: SPARK-28588) Issue Type: Bug (was: Sub-task) > Add additional fields

[jira] [Commented] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082196#comment-17082196 ] Hyukjin Kwon commented on SPARK-31429: -- Actually, let me retarget this as Spark 3.1. It should be

[jira] [Updated] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31429: - Target Version/s: (was: 3.0.0) > Add additional fields in ExpressionDescription for more

[jira] [Commented] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082190#comment-17082190 ] Hyukjin Kwon commented on SPARK-31429: -- [~huaxingao], [~nchammas], [~kevinyu98], [~dkbiswal],

[jira] [Updated] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31429: - Parent: SPARK-28588 Issue Type: Sub-task (was: Improvement) > Add additional fields in

[jira] [Updated] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31429: - Affects Version/s: (was: 3.1.0) 3.0.0 > Add additional fields in

[jira] [Updated] (SPARK-31429) Add additional fields in ExpressionDescription for more granular category in documentation

[ https://issues.apache.org/jira/browse/SPARK-31429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31429: - Target Version/s: 3.0.0 > Add additional fields in ExpressionDescription for more granular

[jira] [Created] (SPARK-31434) Drop builtin function pages from SQL references

Takeshi Yamamuro created SPARK-31434: Summary: Drop builtin function pages from SQL references Key: SPARK-31434 URL: https://issues.apache.org/jira/browse/SPARK-31434 Project: Spark

[jira] [Commented] (SPARK-31427) Spark Structure streaming read data twice per every micro-batch.

[ https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082163#comment-17082163 ] Nick Hryhoriev commented on SPARK-31427: [~kabhwan] I will try to do it, but do not expect to

[jira] [Created] (SPARK-31433) Summarizer supports string arguments

zhengruifeng created SPARK-31433: Summary: Summarizer supports string arguments Key: SPARK-31433 URL: https://issues.apache.org/jira/browse/SPARK-31433 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-31408) Build Spark’s own datetime pattern definition

[ https://issues.apache.org/jira/browse/SPARK-31408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31408: Summary: Build Spark’s own datetime pattern definition (was: Build Spark’s own Datetime

[jira] [Updated] (SPARK-31432) bin/sbin scripts should allow to customize jars dir

[ https://issues.apache.org/jira/browse/SPARK-31432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31432: - Target Version/s: (was: 2.4.6, 3.0.1) > bin/sbin scripts should allow to customize jars dir >

[jira] [Commented] (SPARK-31432) bin/sbin scripts should allow to customize jars dir

[ https://issues.apache.org/jira/browse/SPARK-31432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082129#comment-17082129 ] Shingo Furuyama commented on SPARK-31432: - I will soon send a PR to the master branch. If the PR

[jira] [Updated] (SPARK-31432) bin/sbin scripts should allow to customize jars dir

[ https://issues.apache.org/jira/browse/SPARK-31432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shingo Furuyama updated SPARK-31432: Environment: (was: In the script under bin/sbin, it is better that we can specify

[jira] [Updated] (SPARK-31432) bin/sbin scripts should allow to customize jars dir

[ https://issues.apache.org/jira/browse/SPARK-31432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shingo Furuyama updated SPARK-31432: Description: In the script under bin/sbin, it is better that we can specify

[jira] [Created] (SPARK-31432) bin/sbin scripts should allow to customize jars dir

Shingo Furuyama created SPARK-31432: --- Summary: bin/sbin scripts should allow to customize jars dir Key: SPARK-31432 URL: https://issues.apache.org/jira/browse/SPARK-31432 Project: Spark

[jira] [Resolved] (SPARK-31402) Incorrect rebasing of BCE dates

[ https://issues.apache.org/jira/browse/SPARK-31402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31402. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28172

[jira] [Assigned] (SPARK-31402) Incorrect rebasing of BCE dates

[ https://issues.apache.org/jira/browse/SPARK-31402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31402: --- Assignee: Maxim Gekk > Incorrect rebasing of BCE dates > --- >

[jira] [Updated] (SPARK-31407) Fix hive/SQLQuerySuite.derived from Hive query file: drop_database_removes_partition_dirs.q

[ https://issues.apache.org/jira/browse/SPARK-31407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31407: - Description: Test "derived from Hive query file: drop_database_removes_partition_dirs.q" can fail if we run it

[jira] [Updated] (SPARK-31407) Fix hive/SQLQuerySuite.derived from Hive query file: drop_database_removes_partition_dirs.q

[ https://issues.apache.org/jira/browse/SPARK-31407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31407: - Description: Test "derived from Hive query file: drop_database_removes_partition_dirs.q" can fail if we run it

[jira] [Resolved] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18886. - Fix Version/s: 3.1.0 Resolution: Fixed > Delay scheduling should not delay some

[jira] [Assigned] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-18886: --- Assignee: Nicholas Brett Marcott > Delay scheduling should not delay some executors