[jira] [Commented] (SPARK-31550) nondeterministic configurations with general meanings in sql configuration doc

[ https://issues.apache.org/jira/browse/SPARK-31550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091220#comment-17091220 ] Kent Yao commented on SPARK-31550: -- Github webhook temporary downtime， PR in progress: 

[jira] [Commented] (SPARK-31550) nondeterministic configurations with general meanings in sql configuration doc

[ https://issues.apache.org/jira/browse/SPARK-31550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091218#comment-17091218 ] JinxinTang commented on SPARK-31550: try to specify conf in spark-defaults.conf

[jira] [Updated] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuqi Wang updated SPARK-31551: -- Description: See current 

[jira] [Updated] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuqi Wang updated SPARK-31551: -- Description: See current 

[jira] [Updated] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuqi Wang updated SPARK-31551: -- Description: See current 

[jira] [Updated] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuqi Wang updated SPARK-31551: -- Description: See current 

[jira] [Updated] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuqi Wang updated SPARK-31551: -- Description: See current 

[jira] [Updated] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuqi Wang updated SPARK-31551: -- Description: See current 

[jira] [Updated] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

[ https://issues.apache.org/jira/browse/SPARK-31551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuqi Wang updated SPARK-31551: -- Description: See current *createSparkUser*:

[jira] [Created] (SPARK-31551) createSparkUser lost user's non-Hadoop credentials and fully qualified user name

Yuqi Wang created SPARK-31551: - Summary: createSparkUser lost user's non-Hadoop credentials and fully qualified user name Key: SPARK-31551 URL: https://issues.apache.org/jira/browse/SPARK-31551 Project:

[jira] [Updated] (SPARK-31550) nondeterministic configurations with general meanings in sql configuration doc

[ https://issues.apache.org/jira/browse/SPARK-31550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-31550: - Component/s: Documentation > nondeterministic configurations with general meanings in sql configuration

[jira] [Created] (SPARK-31550) nondeterministic configurations with general meanings in sql configuration doc

Kent Yao created SPARK-31550: Summary: nondeterministic configurations with general meanings in sql configuration doc Key: SPARK-31550 URL: https://issues.apache.org/jira/browse/SPARK-31550 Project:

[jira] [Commented] (SPARK-31463) Enhance JsonDataSource by replacing jackson with simdjson

[ https://issues.apache.org/jira/browse/SPARK-31463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091205#comment-17091205 ] Steven Moy commented on SPARK-31463: Hi [~hyukjin.kwon],  What's Spark recommended path on

[jira] [Commented] (SPARK-31461) RLIKE and LIKE expression compiles every time when it used

[ https://issues.apache.org/jira/browse/SPARK-31461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091201#comment-17091201 ] Linpx commented on SPARK-31461: --- i did search，didn't found any > RLIKE and LIKE expression compiles every

[jira] [Resolved] (SPARK-31461) RLIKE and LIKE expression compiles every time when it used

[ https://issues.apache.org/jira/browse/SPARK-31461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31461. -- Resolution: Duplicate > RLIKE and LIKE expression compiles every time when it used >

[jira] [Commented] (SPARK-31461) RLIKE and LIKE expression compiles every time when it used

[ https://issues.apache.org/jira/browse/SPARK-31461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091200#comment-17091200 ] Hyukjin Kwon commented on SPARK-31461: -- Please search existing jiras first before filing it. >

[jira] [Commented] (SPARK-31463) Enhance JsonDataSource by replacing jackson with simdjson

[ https://issues.apache.org/jira/browse/SPARK-31463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091199#comment-17091199 ] Hyukjin Kwon commented on SPARK-31463: -- So it's about vectorization, right? I think [~maxgekk]

[jira] [Updated] (SPARK-31463) Enhance JsonDataSource by replacing jackson with simdjson

[ https://issues.apache.org/jira/browse/SPARK-31463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31463: - Component/s: (was: Spark Core) SQL > Enhance JsonDataSource by replacing

[jira] [Commented] (SPARK-31466) String/Int to VarcharType cast not supported in Spark

[ https://issues.apache.org/jira/browse/SPARK-31466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091196#comment-17091196 ] Hyukjin Kwon commented on SPARK-31466: -- Can you read the doc for the class? {quote} * Hive

[jira] [Resolved] (SPARK-31466) String/Int to VarcharType cast not supported in Spark

[ https://issues.apache.org/jira/browse/SPARK-31466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31466. -- Resolution: Invalid > String/Int to VarcharType cast not supported in Spark >

[jira] [Resolved] (SPARK-31479) Numbers with thousands separator or locale specific decimal separator not parsed correctly

[ https://issues.apache.org/jira/browse/SPARK-31479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31479. -- Resolution: Duplicate > Numbers with thousands separator or locale specific decimal separator

[jira] [Commented] (SPARK-31479) Numbers with thousands separator or locale specific decimal separator not parsed correctly

[ https://issues.apache.org/jira/browse/SPARK-31479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091193#comment-17091193 ] Hyukjin Kwon commented on SPARK-31479: -- Use locale option. See SPARK-25945 > Numbers with

[jira] [Commented] (SPARK-31493) Optimize InSet to In according partition size at InSubqueryExec

[ https://issues.apache.org/jira/browse/SPARK-31493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091190#comment-17091190 ] Hyukjin Kwon commented on SPARK-31493: -- PR: https://github.com/apache/spark/pull/28269 > Optimize

[jira] [Updated] (SPARK-31482) spark.kubernetes.driver.podTemplateFile Configuration not used by the job

[ https://issues.apache.org/jira/browse/SPARK-31482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31482: - Priority: Major (was: Blocker) > spark.kubernetes.driver.podTemplateFile Configuration not

[jira] [Resolved] (SPARK-31496) Exception in thread "dispatcher-event-loop-1" java.lang.OutOfMemoryError

[ https://issues.apache.org/jira/browse/SPARK-31496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31496. -- Resolution: Invalid > Exception in thread "dispatcher-event-loop-1"

[jira] [Commented] (SPARK-31496) Exception in thread "dispatcher-event-loop-1" java.lang.OutOfMemoryError

[ https://issues.apache.org/jira/browse/SPARK-31496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091189#comment-17091189 ] Hyukjin Kwon commented on SPARK-31496: -- Is this a regression? Sounds more like a question which

[jira] [Commented] (SPARK-31502) document identifier in SQL Reference

[ https://issues.apache.org/jira/browse/SPARK-31502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091188#comment-17091188 ] Hyukjin Kwon commented on SPARK-31502: -- PR: https://github.com/apache/spark/pull/28277 > document

[jira] [Updated] (SPARK-31510) Set setwd in R documentation build

[ https://issues.apache.org/jira/browse/SPARK-31510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31510: - Fix Version/s: 3.0.0 > Set setwd in R documentation build > --

[jira] [Resolved] (SPARK-31510) Set setwd in R documentation build

[ https://issues.apache.org/jira/browse/SPARK-31510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31510. -- Resolution: Fixed Fixed in https://github.com/apache/spark/pull/28285 > Set setwd in R

[jira] [Reopened] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-31530: -- > Spark submit fails if we provide extraJavaOption which contains Xmx as > substring >

[jira] [Updated] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank updated SPARK-31530: --- Description: Spark submit doesn't allow Xmx anywhere in the spark.driver.extraJavaOptions For eg:

[jira] [Commented] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091184#comment-17091184 ] Hyukjin Kwon commented on SPARK-31530: -- Oh, gotya. > Spark submit fails if we provide

[jira] [Commented] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091182#comment-17091182 ] Mayank commented on SPARK-31530: [~hyukjin.kwon] If you check the description I want to specify some

[jira] [Resolved] (SPARK-31514) Kerberos: Spark UGI credentials are not getting passed down to Hive

[ https://issues.apache.org/jira/browse/SPARK-31514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31514. -- Resolution: Invalid > Kerberos: Spark UGI credentials are not getting passed down to Hive >

[jira] [Commented] (SPARK-31514) Kerberos: Spark UGI credentials are not getting passed down to Hive

[ https://issues.apache.org/jira/browse/SPARK-31514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091181#comment-17091181 ] Hyukjin Kwon commented on SPARK-31514: -- It should be best to ask questions into mailing list before

[jira] [Commented] (SPARK-31519) Cast in having aggregate expressions returns the wrong result

[ https://issues.apache.org/jira/browse/SPARK-31519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091179#comment-17091179 ] Hyukjin Kwon commented on SPARK-31519: -- PR: https://github.com/apache/spark/pull/28294 > Cast in

[jira] [Commented] (SPARK-31521) The fetch size is not correct when merging blocks into a merged block

[ https://issues.apache.org/jira/browse/SPARK-31521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091178#comment-17091178 ] Hyukjin Kwon commented on SPARK-31521: -- PR: https://github.com/apache/spark/pull/28301 > The fetch

[jira] [Commented] (SPARK-31524) Add metric to the split number for skew partition when enable AQE

[ https://issues.apache.org/jira/browse/SPARK-31524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091175#comment-17091175 ] Hyukjin Kwon commented on SPARK-31524: -- PR: https://github.com/apache/spark/pull/28109 > Add

[jira] [Commented] (SPARK-31523) LogicalPlan doCanonicalize should throw exception if not resolved

[ https://issues.apache.org/jira/browse/SPARK-31523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091176#comment-17091176 ] Hyukjin Kwon commented on SPARK-31523: -- PR: https://github.com/apache/spark/pull/28304 >

[jira] [Commented] (SPARK-31527) date add/subtract interval only allow those day precision in ansi mode

[ https://issues.apache.org/jira/browse/SPARK-31527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091174#comment-17091174 ] Hyukjin Kwon commented on SPARK-31527: -- PR https://github.com/apache/spark/pull/28310 > date

[jira] [Commented] (SPARK-31529) Remove extra whitespaces in the formatted explain

[ https://issues.apache.org/jira/browse/SPARK-31529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091172#comment-17091172 ] Hyukjin Kwon commented on SPARK-31529: -- PR: https://github.com/apache/spark/pull/28315 > Remove

[jira] [Commented] (SPARK-31528) Remove millennium, century, decade from trunc/date_trunc fucntions

[ https://issues.apache.org/jira/browse/SPARK-31528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091173#comment-17091173 ] Hyukjin Kwon commented on SPARK-31528: -- PR: https://github.com/apache/spark/pull/28313 > Remove

[jira] [Resolved] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31530. -- Resolution: Won't Fix > Spark submit fails if we provide extraJavaOption which contains Xmx

[jira] [Commented] (SPARK-31530) Spark submit fails if we provide extraJavaOption which contains Xmx as substring

[ https://issues.apache.org/jira/browse/SPARK-31530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091171#comment-17091171 ] Hyukjin Kwon commented on SPARK-31530: -- Why don't you follow the guide and use

[jira] [Resolved] (SPARK-31531) sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit

[ https://issues.apache.org/jira/browse/SPARK-31531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31531. -- Resolution: Duplicate > sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found

[jira] [Commented] (SPARK-31531) sun.misc.Cleaner sun.nio.ch.DirectBuffer.cleaner() method not found during spark-submit

[ https://issues.apache.org/jira/browse/SPARK-31531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091170#comment-17091170 ] Hyukjin Kwon commented on SPARK-31531: -- Spark does not support Java 11 in Spark 2.x. It will be

[jira] [Commented] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091169#comment-17091169 ] Hyukjin Kwon commented on SPARK-31532: -- PR is in progress at

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091163#comment-17091163 ] Dongjoon Hyun commented on SPARK-31544: --- This is resolved via

[jira] [Updated] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30199: -- Issue Type: Bug (was: Improvement) > Recover spark.ui.port and spark.blockManager.port from

[jira] [Updated] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30199: -- Fix Version/s: 2.4.6 > Recover spark.ui.port and spark.blockManager.port from checkpoint >

[jira] [Resolved] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31544. --- Fix Version/s: 2.4.6 Resolution: Fixed > Backport SPARK-30199 Recover

[jira] [Commented] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091158#comment-17091158 ] Hyukjin Kwon commented on SPARK-31532: -- The problem is that the static configuration was changed

[jira] [Assigned] (SPARK-31547) Upgrade Genjavadoc to 0.16

[ https://issues.apache.org/jira/browse/SPARK-31547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31547: Assignee: Dongjoon Hyun > Upgrade Genjavadoc to 0.16 > -- > >

[jira] [Resolved] (SPARK-31547) Upgrade Genjavadoc to 0.16

[ https://issues.apache.org/jira/browse/SPARK-31547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31547. -- Fix Version/s: 3.1.0 Resolution: Fixed Fixed in

[jira] [Commented] (SPARK-31545) Backport SPARK-27676 InMemoryFileIndex should respect spark.sql.files.ignoreMissingFiles

[ https://issues.apache.org/jira/browse/SPARK-31545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091152#comment-17091152 ] Hyukjin Kwon commented on SPARK-31545: -- I think no .. it causes a behaviour change which can be

[jira] [Commented] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091150#comment-17091150 ] Hyukjin Kwon commented on SPARK-31538: -- Hm, I wonder why we should backport this. This was just a

[jira] [Commented] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091149#comment-17091149 ] Hyukjin Kwon commented on SPARK-31537: -- I wouldn't port this back per the guidelines in our

[jira] [Updated] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31537: - Target Version/s: 2.4.6 > Backport SPARK-25559 Remove the unsupported predicates in Parquet

[jira] [Updated] (SPARK-31537) Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible

[ https://issues.apache.org/jira/browse/SPARK-31537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31537: - Fix Version/s: (was: 2.4.6) > Backport SPARK-25559 Remove the unsupported predicates in

[jira] [Commented] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

[ https://issues.apache.org/jira/browse/SPARK-31536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091148#comment-17091148 ] Hyukjin Kwon commented on SPARK-31536: -- To backport this, we should port SPARK-31116 together. I

[jira] [Commented] (SPARK-31546) Backport SPARK-25595 Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

[ https://issues.apache.org/jira/browse/SPARK-31546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091146#comment-17091146 ] Hyukjin Kwon commented on SPARK-31546: -- I think it's fine to port back. > Backport SPARK-25595

[jira] [Assigned] (SPARK-30804) Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog

[ https://issues.apache.org/jira/browse/SPARK-30804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30804: Assignee: Jungtaek Lim > Measure and log elapsed time for "compact" operation in >

[jira] [Resolved] (SPARK-30804) Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog

[ https://issues.apache.org/jira/browse/SPARK-30804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30804. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27557

[jira] [Updated] (SPARK-31549) Pyspark SparkContext.cancelJobGroup do not work correctly

[ https://issues.apache.org/jira/browse/SPARK-31549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-31549: --- Issue Type: Bug (was: Improvement) > Pyspark SparkContext.cancelJobGroup do not work correctly >

[jira] [Created] (SPARK-31549) Pyspark SparkContext.cancelJobGroup do not work correctly

Weichen Xu created SPARK-31549: -- Summary: Pyspark SparkContext.cancelJobGroup do not work correctly Key: SPARK-31549 URL: https://issues.apache.org/jira/browse/SPARK-31549 Project: Spark Issue

[jira] [Created] (SPARK-31548) Refactor pyspark code for common methods in JavaParams and Pipeline/OneVsRest

Weichen Xu created SPARK-31548: -- Summary: Refactor pyspark code for common methods in JavaParams and Pipeline/OneVsRest Key: SPARK-31548 URL: https://issues.apache.org/jira/browse/SPARK-31548 Project:

[jira] [Resolved] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

[ https://issues.apache.org/jira/browse/SPARK-31488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31488. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28259

[jira] [Assigned] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

[ https://issues.apache.org/jira/browse/SPARK-31488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31488: --- Assignee: Maxim Gekk > Support `java.time.LocalDate` in Parquet filter pushdown >

[jira] [Assigned] (SPARK-31526) Add a new test suite for ExperssionInfo

[ https://issues.apache.org/jira/browse/SPARK-31526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-31526: Assignee: Takeshi Yamamuro > Add a new test suite for ExperssionInfo >

[jira] [Resolved] (SPARK-31526) Add a new test suite for ExperssionInfo

[ https://issues.apache.org/jira/browse/SPARK-31526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31526. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28308

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091075#comment-17091075 ] Jungtaek Lim commented on SPARK-26385: -- The symptoms are mixed up - please clarify whether the

[jira] [Comment Edited] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091067#comment-17091067 ] JinxinTang edited comment on SPARK-31532 at 4/24/20, 1:08 AM: -- Thanks for

[jira] [Commented] (SPARK-31532) SparkSessionBuilder shoud not propagate static sql configurations to the existing active/default SparkSession

[ https://issues.apache.org/jira/browse/SPARK-31532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091067#comment-17091067 ] JinxinTang commented on SPARK-31532: Thanks for your issue, these followings not be modified after

[jira] [Resolved] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuite

[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-31542. -- Resolution: Not A Problem > Backport SPARK-25692 Remove static initialization of worker

[jira] [Commented] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuit

[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091060#comment-17091060 ] Shixiong Zhu commented on SPARK-31542: -- [~holden] The flaky test was caused by a new improvement in

[jira] [Commented] (SPARK-31464) Upgrade Kafka to 2.5.0

[ https://issues.apache.org/jira/browse/SPARK-31464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091051#comment-17091051 ] Dongjoon Hyun commented on SPARK-31464: --- Thank you, [~ijuma]! :) > Upgrade Kafka to 2.5.0 >

[jira] [Resolved] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-27891. -- Resolution: Cannot Reproduce SPARK-23361 is in Spark 2.4.0 and the fix is not going to be

[jira] [Created] (SPARK-31547) Upgrade Genjavadoc to 0.16

Dongjoon Hyun created SPARK-31547: - Summary: Upgrade Genjavadoc to 0.16 Key: SPARK-31547 URL: https://issues.apache.org/jira/browse/SPARK-31547 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090996#comment-17090996 ] Dongjoon Hyun commented on SPARK-25075: --- Thank you for updates, [~smarter]. > Build and test

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090954#comment-17090954 ] Holden Karau commented on SPARK-31540: -- I was thinking some folks might build 2.4 with newer JDKs.

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090944#comment-17090944 ] Sean R. Owen commented on SPARK-31540: -- The backport is probably harmless, but, why is it needed

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090928#comment-17090928 ] Holden Karau commented on SPARK-31544: -- Thanks! > Backport SPARK-30199 Recover

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090926#comment-17090926 ] Dongjoon Hyun commented on SPARK-31544: --- I made a PR, [~holden]. -

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090924#comment-17090924 ] Holden Karau commented on SPARK-31540: -- Gotcha, I'll go through the backport JIRAs and link. >

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090922#comment-17090922 ] Dongjoon Hyun commented on SPARK-31540: --- cc [~srowen] > Backport SPARK-27981 Remove

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090921#comment-17090921 ] Dongjoon Hyun commented on SPARK-31540: --- [~holden]. Could you link the original JIRA additionally?

[jira] [Commented] (SPARK-31544) Backport SPARK-30199 Recover `spark.(ui|blockManager).port` from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-31544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090917#comment-17090917 ] Dongjoon Hyun commented on SPARK-31544: --- BTW, I kept the original authorship from the beginning.

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090915#comment-17090915 ] Holden Karau commented on SPARK-31539: -- The main reason I'd see to backport the change is it's only

[jira] [Resolved] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-31539. -- Resolution: Won't Fix > Backport SPARK-27138 Remove AdminUtils calls (fixes

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090914#comment-17090914 ] Holden Karau commented on SPARK-31539: -- The other Jira is marked as resolved and I want to track

[jira] [Updated] (SPARK-31485) Barrier stage can hang if only partial tasks launched

[ https://issues.apache.org/jira/browse/SPARK-31485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31485: - Shepherd: Holden Karau > Barrier stage can hang if only partial tasks launched >

[jira] [Updated] (SPARK-31485) Barrier stage can hang if only partial tasks launched

[ https://issues.apache.org/jira/browse/SPARK-31485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau updated SPARK-31485: - Target Version/s: 2.4.6, 3.0.0 > Barrier stage can hang if only partial tasks launched >

[jira] [Commented] (SPARK-31543) Backport SPARK-26306 More memory to de-flake SorterSuite

[ https://issues.apache.org/jira/browse/SPARK-31543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090911#comment-17090911 ] Sean R. Owen commented on SPARK-31543: -- Does it need to be a new JIRA? but if it's a simple test

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090910#comment-17090910 ] Sean R. Owen commented on SPARK-31539: -- Is this deprecated w.r.t. the version used in 2.4.x? Is it

[jira] [Commented] (SPARK-31536) Backport SPARK-25407 Allow nested access for non-existent field for Parquet file when nested pruning is enabled

[ https://issues.apache.org/jira/browse/SPARK-31536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090908#comment-17090908 ] Holden Karau commented on SPARK-31536: -- cc [~michael] & [~hyukjin.kwon] for thoughts > Backport

[jira] [Commented] (SPARK-31538) Backport SPARK-25338 Ensure to call super.beforeAll() and super.afterAll() in test cases

[ https://issues.apache.org/jira/browse/SPARK-31538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090907#comment-17090907 ] Holden Karau commented on SPARK-31538: -- cc [~kiszk] for thoughts > Backport SPARK-25338

[jira] [Commented] (SPARK-31539) Backport SPARK-27138 Remove AdminUtils calls (fixes deprecation)

[ https://issues.apache.org/jira/browse/SPARK-31539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090906#comment-17090906 ] Holden Karau commented on SPARK-31539: -- cc [~DylanGuedes] & [~srowen] for thoughts > Backport

[jira] [Commented] (SPARK-31540) Backport SPARK-27981 Remove `Illegal reflective access` warning for `java.nio.Bits.unaligned()` in JDK9+

[ https://issues.apache.org/jira/browse/SPARK-31540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090905#comment-17090905 ] Holden Karau commented on SPARK-31540: -- cc [~dongjoon] for thoughts > Backport SPARK-27981

[jira] [Commented] (SPARK-31541) Backport SPARK-26095 Disable parallelization in make-distibution.sh. (Avoid build hanging)

[ https://issues.apache.org/jira/browse/SPARK-31541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090904#comment-17090904 ] Holden Karau commented on SPARK-31541: -- cc [~vanzin] for thoughts > Backport SPARK-26095

[jira] [Commented] (SPARK-31542) Backport SPARK-25692 Remove static initialization of worker eventLoop handling chunk fetch requests within TransportContext. This fixes ChunkFetchIntegrationSuit

[ https://issues.apache.org/jira/browse/SPARK-31542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090902#comment-17090902 ] Holden Karau commented on SPARK-31542: -- cc [~sanket991] & [~zsxwing] for thoughts > Backport

  1   2   >