[jira] [Created] (SPARK-31526) Add a new test suite for ExperssionInfo

2020-04-23 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-31526: Summary: Add a new test suite for ExperssionInfo Key: SPARK-31526 URL: https://issues.apache.org/jira/browse/SPARK-31526 Project: Spark Issue Type: T

[jira] [Commented] (SPARK-27039) toPandas with Arrow swallows maxResultSize errors

2020-04-23 Thread peay (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090352#comment-17090352 ] peay commented on SPARK-27039: -- [~hyukjin.kwon] do you know if this was eventually back por

[jira] [Created] (SPARK-31527) date add/subtract interval only allow those day precision in ansi mode

2020-04-23 Thread Kent Yao (Jira)
Kent Yao created SPARK-31527: Summary: date add/subtract interval only allow those day precision in ansi mode Key: SPARK-31527 URL: https://issues.apache.org/jira/browse/SPARK-31527 Project: Spark

[jira] [Updated] (SPARK-31344) Polish implementation of barrier() and allGather()

2020-04-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-31344: Fix Version/s: (was: 3.1.0) 3.0.0 > Polish implementation of barrier() and

[jira] [Commented] (SPARK-31527) date add/subtract interval only allow those day precision in ansi mode

2020-04-23 Thread charity johnson johnson (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090438#comment-17090438 ] charity johnson johnson commented on SPARK-31527: - Ty > date add/subtra

[jira] [Created] (SPARK-31528) Remove millennium, century, decade from trunc/date_trunc fucntions

2020-04-23 Thread Kent Yao (Jira)
Kent Yao created SPARK-31528: Summary: Remove millennium, century, decade from trunc/date_trunc fucntions Key: SPARK-31528 URL: https://issues.apache.org/jira/browse/SPARK-31528 Project: Spark

[jira] [Created] (SPARK-31529) Remove redundant whitespaces in the formatted explain

2020-04-23 Thread wuyi (Jira)
wuyi created SPARK-31529: Summary: Remove redundant whitespaces in the formatted explain Key: SPARK-31529 URL: https://issues.apache.org/jira/browse/SPARK-31529 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-31529) Remove extra whitespaces in the formatted explain

2020-04-23 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31529: - Summary: Remove extra whitespaces in the formatted explain (was: Remove redundant whitespaces in the formatted

[jira] [Updated] (SPARK-31529) Remove extra whitespaces in the formatted explain

2020-04-23 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-31529: - Description: The formatted explain included extra whitespaces. And even the number of spaces are different betwe

[jira] [Created] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

2020-04-23 Thread Mayank (Jira)
Mayank created SPARK-31530: -- Summary: Spark submit fails if we provide extraJavaOption which contains Xmx as substring Key: SPARK-31530 URL: https://issues.apache.org/jira/browse/SPARK-31530 Project: Spark

[jira] [Updated] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

2020-04-23 Thread Mayank (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank updated SPARK-31530: --- Labels: 2.4.0 Spark Submit (was: ) > Spark submit fails if we provide extraJavaOption which contains Xmx a

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2020-04-23 Thread Guillaume Martres (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090582#comment-17090582 ] Guillaume Martres commented on SPARK-25075: --- > SPARK-30132 which is blocked on

[jira] [Commented] (SPARK-30090) Update REPL for 2.13

2020-04-23 Thread Seth Tisue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090590#comment-17090590 ] Seth Tisue commented on SPARK-30090: some further REPL changes have now landed in Sc

[jira] [Created] (SPARK-31531) sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit

2020-04-23 Thread shayoni Halder (Jira)
shayoni Halder created SPARK-31531: -- Summary: sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit Key: SPARK-31531 URL: https://issues.apache.org/jira/browse/SPARK-31531

[jira] [Updated] (SPARK-31531) sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit

2020-04-23 Thread shayoni Halder (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shayoni Halder updated SPARK-31531: --- Attachment: error.PNG > sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found

[jira] [Resolved] (SPARK-31472) allGather() may return null messages

2020-04-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31472. - Fix Version/s: 3.0.0 Assignee: wuyi Resolution: Fixed > allGather() may return n

[jira] [Updated] (SPARK-31531) sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit

2020-04-23 Thread shayoni Halder (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shayoni Halder updated SPARK-31531: --- Description: I am trying to run the following Spark submit from a VM using Yarn cluster mode

[jira] [Resolved] (SPARK-31522) Hive metastore client initialization related configurations should be static

2020-04-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31522. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28302 [https://gith

[jira] [Assigned] (SPARK-31522) Hive metastore client initialization related configurations should be static

2020-04-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31522: --- Assignee: Kent Yao > Hive metastore client initialization related configurations should be

[jira] [Created] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

2020-04-23 Thread Kent Yao (Jira)
Kent Yao created SPARK-31532: Summary: SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession Key: SPARK-31532 URL: https://issues.apache.org/jira/browse/SPARK-31

[jira] [Created] (SPARK-31533) Enable DB2IntegrationSuite test and upgrade the DB2 docker inside

2020-04-23 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-31533: - Summary: Enable DB2IntegrationSuite test and upgrade the DB2 docker inside Key: SPARK-31533 URL: https://issues.apache.org/jira/browse/SPARK-31533 Project: Spark

[jira] [Commented] (SPARK-31533) Enable DB2IntegrationSuite test and upgrade the DB2 docker inside

2020-04-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090725#comment-17090725 ] Gabor Somogyi commented on SPARK-31533: --- Started to work on this. > Enable DB2Int

[jira] [Updated] (SPARK-31533) Enable DB2IntegrationSuite test and upgrade the DB2 docker inside

2020-04-23 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-31533: -- Component/s: Tests > Enable DB2IntegrationSuite test and upgrade the DB2 docker inside > -

[jira] [Updated] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

2020-04-23 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-31532: - Affects Version/s: 3.1.0 2.0.2 2.1.3

[jira] [Updated] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

2020-04-23 Thread Mayank (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank updated SPARK-31530: --- Description: Spark submit doesn't allow Xmx anywhere in the spark.driver.extraJavaOptions For eg: {code:jav

[jira] [Created] (SPARK-31534) Text for tooltip should be escaped

2020-04-23 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-31534: -- Summary: Text for tooltip should be escaped Key: SPARK-31534 URL: https://issues.apache.org/jira/browse/SPARK-31534 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2020-04-23 Thread Geoffrey Blake (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090773#comment-17090773 ] Geoffrey Blake commented on SPARK-29106: Hi all, Regarding the leveldbjni depen

[jira] [Updated] (SPARK-29458) Document scalar functions usage in APIs in SQL getting started.

2020-04-23 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-29458: --- Affects Version/s: (was: 3.1.0) 3.0.0 > Document scalar functions usage i

[jira] [Created] (SPARK-31535) Fix nested CTE substitution

2020-04-23 Thread Peter Toth (Jira)
Peter Toth created SPARK-31535: -- Summary: Fix nested CTE substitution Key: SPARK-31535 URL: https://issues.apache.org/jira/browse/SPARK-31535 Project: Spark Issue Type: Bug Components:

[jira] [Commented] (SPARK-21529) Improve the error message for unsupported Uniontype

2020-04-23 Thread Sudharshann D. (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090862#comment-17090862 ] Sudharshann D. commented on SPARK-21529: If this issue hasnt been solved yet, ma

[jira] [Updated] (SPARK-31535) Fix nested CTE substitution

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31535: -- Target Version/s: 3.0.0 Priority: Blocker (was: Major) > Fix nested CTE substitut

[jira] [Updated] (SPARK-31535) Fix nested CTE substitution

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31535: -- Labels: correctness (was: ) > Fix nested CTE substitution > --- > >

[jira] [Commented] (SPARK-31535) Fix nested CTE substitution

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090872#comment-17090872 ] Dongjoon Hyun commented on SPARK-31535: --- I added `correctness` label and raised th

[jira] [Created] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31536: Summary: Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled Key: SPARK-31536 URL: https://issues.apache.org/jira

[jira] [Created] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31537: Summary: Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible Key: SPARK-31537 URL: https://issues.apache.org/jira/browse/SPARK-31537 Proj

[jira] [Created] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31538: Summary: Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases Key: SPARK-31538 URL: https://issues.apache.org/jira/browse/SPARK-31538

[jira] [Created] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31540: Summary: Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+ Key: SPARK-31540 URL: https://issues.apache.org/jira/browse

[jira] [Created] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31539: Summary: Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation) Key: SPARK-31539 URL: https://issues.apache.org/jira/browse/SPARK-31539 Project: Spark

[jira] [Created] (SPARK-31541) Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging)

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31541: Summary: Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging) Key: SPARK-31541 URL: https://issues.apache.org/jira/browse/SPARK-31541

[jira] [Created] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31542: Summary: Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite as well Key: SPAR

[jira] [Created] (SPARK-31543) Backport SPARK-26306 More memory to de-flake SorterSuite

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31543: Summary: Backport SPARK-26306 More memory to de-flake SorterSuite Key: SPARK-31543 URL: https://issues.apache.org/jira/browse/SPARK-31543 Project: Spark

[jira] [Created] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31544: Summary: Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint Key: SPARK-31544 URL: https://issues.apache.org/jira/browse/SPARK-31544 Pro

[jira] [Updated] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31545: - Component/s: (was: DStreams) SQL > Backport SPARK-27676 InMemoryFileI

[jira] [Created] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31545: Summary: Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles Key: SPARK-31545 URL: https://issues.apache.org/jira/browse/SPARK-31545

[jira] [Updated] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31542: - Component/s: (was: Build) Spark Core > Backport SPARK-25692 Remove st

[jira] [Created] (SPARK-31546) Backport SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

2020-04-23 Thread Holden Karau (Jira)
Holden Karau created SPARK-31546: Summary: Backport SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled Key: SPARK-31546 URL: https://issues.apache.org/jira/browse/SPARK-31546

[jira] [Commented] (SPARK-31543) Backport SPARK-26306 More memory to de-flake SorterSuite

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090901#comment-17090901 ] Holden Karau commented on SPARK-31543: -- cc [~gsomogyi] & [~srowen] for thoughts >

[jira] [Commented] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090900#comment-17090900 ] Holden Karau commented on SPARK-31537: -- cc [~dbtsai]for thoughts > Backport SPARK-

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090903#comment-17090903 ] Dongjoon Hyun commented on SPARK-31544: --- Got it. I'll make a PR. > Backport SPARK

[jira] [Assigned] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31544: - Assignee: Dongjoon Hyun > Backport SPARK-30199 Recover `spark.(ui|blockManager).p

[jira] [Commented] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuit

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090902#comment-17090902 ] Holden Karau commented on SPARK-31542: -- cc [~sanket991] & [~zsxwing] for thoughts

[jira] [Commented] (SPARK-31541) Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging)

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090904#comment-17090904 ] Holden Karau commented on SPARK-31541: -- cc [~vanzin] for thoughts > Backport SPARK

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090905#comment-17090905 ] Holden Karau commented on SPARK-31540: -- cc [~dongjoon] for thoughts > Backport SPA

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090906#comment-17090906 ] Holden Karau commented on SPARK-31539: -- cc [~DylanGuedes] & [~srowen] for thoughts

[jira] [Commented] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090907#comment-17090907 ] Holden Karau commented on SPARK-31538: -- cc [~kiszk] for thoughts > Backport SPARK-

[jira] [Commented] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090908#comment-17090908 ] Holden Karau commented on SPARK-31536: -- cc [~michael] & [~hyukjin.kwon] for thought

[jira] [Commented] (SPARK-31543) Backport SPARK-26306 More memory to de-flake SorterSuite

2020-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090911#comment-17090911 ] Sean R. Owen commented on SPARK-31543: -- Does it need to be a new JIRA? but if it's

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

2020-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090910#comment-17090910 ] Sean R. Owen commented on SPARK-31539: -- Is this deprecated w.r.t. the version used

[jira] [Updated] (SPARK-31485) Barrier stage can hang if only partial tasks launched

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31485: - Target Version/s: 2.4.6, 3.0.0 > Barrier stage can hang if only partial tasks launched > ---

[jira] [Updated] (SPARK-31485) Barrier stage can hang if only partial tasks launched

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31485: - Shepherd: Holden Karau > Barrier stage can hang if only partial tasks launched > ---

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090914#comment-17090914 ] Holden Karau commented on SPARK-31539: -- The other Jira is marked as resolved and I

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090915#comment-17090915 ] Holden Karau commented on SPARK-31539: -- The main reason I'd see to backport the cha

[jira] [Resolved] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31539. -- Resolution: Won't Fix > Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090917#comment-17090917 ] Dongjoon Hyun commented on SPARK-31544: --- BTW, I kept the original authorship from

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090921#comment-17090921 ] Dongjoon Hyun commented on SPARK-31540: --- [~holden]. Could you link the original JI

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090922#comment-17090922 ] Dongjoon Hyun commented on SPARK-31540: --- cc [~srowen] > Backport SPARK-27981

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090924#comment-17090924 ] Holden Karau commented on SPARK-31540: -- Gotcha, I'll go through the backport JIRAs

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090926#comment-17090926 ] Dongjoon Hyun commented on SPARK-31544: --- I made a PR, [~holden]. - https://github.

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090928#comment-17090928 ] Holden Karau commented on SPARK-31544: -- Thanks! > Backport SPARK-30199 Recov

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

2020-04-23 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090944#comment-17090944 ] Sean R. Owen commented on SPARK-31540: -- The backport is probably harmless, but, why

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

2020-04-23 Thread Holden Karau (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090954#comment-17090954 ] Holden Karau commented on SPARK-31540: -- I was thinking some folks might build 2.4 w

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090996#comment-17090996 ] Dongjoon Hyun commented on SPARK-25075: --- Thank you for updates, [~smarter]. > Bui

[jira] [Created] (SPARK-31547) Upgrade Genjavadoc to 0.16

2020-04-23 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-31547: - Summary: Upgrade Genjavadoc to 0.16 Key: SPARK-31547 URL: https://issues.apache.org/jira/browse/SPARK-31547 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

2020-04-23 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-27891. -- Resolution: Cannot Reproduce SPARK-23361 is in Spark 2.4.0 and the fix is not going to be 2.3.

[jira] [Commented] (SPARK-31464) Upgrade Kafka to 2.5.0

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091051#comment-17091051 ] Dongjoon Hyun commented on SPARK-31464: --- Thank you, [~ijuma]! :) > Upgrade Kafka

[jira] [Resolved] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite

2020-04-23 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-31542. -- Resolution: Not A Problem > Backport SPARK-25692 Remove static initialization of worker

[jira] [Commented] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuit

2020-04-23 Thread Shixiong Zhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091060#comment-17091060 ] Shixiong Zhu commented on SPARK-31542: -- [~holden] The flaky test was caused by a ne

[jira] [Commented] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

2020-04-23 Thread JinxinTang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091067#comment-17091067 ] JinxinTang commented on SPARK-31532: Thanks for your issue, these followings not be

[jira] [Comment Edited] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

2020-04-23 Thread JinxinTang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091067#comment-17091067 ] JinxinTang edited comment on SPARK-31532 at 4/24/20, 1:08 AM:

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

2020-04-23 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091075#comment-17091075 ] Jungtaek Lim commented on SPARK-26385: -- The symptoms are mixed up - please clarify

[jira] [Assigned] (SPARK-31526) Add a new test suite for ExperssionInfo

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31526: Assignee: Takeshi Yamamuro > Add a new test suite for ExperssionInfo > --

[jira] [Resolved] (SPARK-31526) Add a new test suite for ExperssionInfo

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31526. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28308 [https://gi

[jira] [Resolved] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

2020-04-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31488. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28259 [https://gith

[jira] [Assigned] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

2020-04-23 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31488: --- Assignee: Maxim Gekk > Support `java.time.LocalDate` in Parquet filter pushdown > -

[jira] [Created] (SPARK-31548) Refactor pyspark code for common methods in JavaParams and Pipeline/OneVsRest

2020-04-23 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-31548: -- Summary: Refactor pyspark code for common methods in JavaParams and Pipeline/OneVsRest Key: SPARK-31548 URL: https://issues.apache.org/jira/browse/SPARK-31548 Project: Sp

[jira] [Created] (SPARK-31549) Pyspark SparkContext.cancelJobGroup do not work correctly

2020-04-23 Thread Weichen Xu (Jira)
Weichen Xu created SPARK-31549: -- Summary: Pyspark SparkContext.cancelJobGroup do not work correctly Key: SPARK-31549 URL: https://issues.apache.org/jira/browse/SPARK-31549 Project: Spark Issue T

[jira] [Updated] (SPARK-31549) Pyspark SparkContext.cancelJobGroup do not work correctly

2020-04-23 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-31549: --- Issue Type: Bug (was: Improvement) > Pyspark SparkContext.cancelJobGroup do not work correctly > --

[jira] [Resolved] (SPARK-30804) Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30804. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27557 [https://gi

[jira] [Assigned] (SPARK-30804) Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30804: Assignee: Jungtaek Lim > Measure and log elapsed time for "compact" operation in > Compa

[jira] [Commented] (SPARK-31546) Backport SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091146#comment-17091146 ] Hyukjin Kwon commented on SPARK-31546: -- I think it's fine to port back. > Backport

[jira] [Commented] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091148#comment-17091148 ] Hyukjin Kwon commented on SPARK-31536: -- To backport this, we should port SPARK-3111

[jira] [Commented] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091149#comment-17091149 ] Hyukjin Kwon commented on SPARK-31537: -- I wouldn't port this back per the guideline

[jira] [Updated] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31537: - Target Version/s: 2.4.6 > Backport SPARK-25559 Remove the unsupported predicates in Parquet whe

[jira] [Updated] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31537: - Fix Version/s: (was: 2.4.6) > Backport SPARK-25559 Remove the unsupported predicates in Par

[jira] [Commented] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091150#comment-17091150 ] Hyukjin Kwon commented on SPARK-31538: -- Hm, I wonder why we should backport this. T

[jira] [Commented] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091152#comment-17091152 ] Hyukjin Kwon commented on SPARK-31545: -- I think no .. it causes a behaviour change

[jira] [Resolved] (SPARK-31547) Upgrade Genjavadoc to 0.16

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31547. -- Fix Version/s: 3.1.0 Resolution: Fixed Fixed in https://github.com/apache/spark/pull/28

[jira] [Assigned] (SPARK-31547) Upgrade Genjavadoc to 0.16

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31547: Assignee: Dongjoon Hyun > Upgrade Genjavadoc to 0.16 > -- > >

[jira] [Commented] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

2020-04-23 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091158#comment-17091158 ] Hyukjin Kwon commented on SPARK-31532: -- The problem is that the static configuratio

[jira] [Updated] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

2020-04-23 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30199: -- Issue Type: Bug (was: Improvement) > Recover spark.ui.port and spark.blockManager.port from c

  1   2   >