[jira] [Resolved] (SPARK-47398) AQE doesn't allow for extension of InMemoryTableScanExec

[ https://issues.apache.org/jira/browse/SPARK-47398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-47398. --- Fix Version/s: 4.0.0 3.5.2 Assignee: Raza Jafri

[jira] [Resolved] (SPARK-47458) Incorrect to calculate the concurrent task number

[ https://issues.apache.org/jira/browse/SPARK-47458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-47458. --- Fix Version/s: 4.0.0 Assignee: Bobby Wang Resolution: Fixed > Incorrect to

[jira] [Assigned] (SPARK-47208) Allow overriding base overhead memory

[ https://issues.apache.org/jira/browse/SPARK-47208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-47208: - Assignee: Joao Correia > Allow overriding base overhead memory >

[jira] [Resolved] (SPARK-47208) Allow overriding base overhead memory

[ https://issues.apache.org/jira/browse/SPARK-47208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-47208. --- Fix Version/s: 4.0.0 Resolution: Fixed > Allow overriding base overhead memory >

[jira] [Commented] (SPARK-45527) Task fraction resource request is not expected

[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821279#comment-17821279 ] Thomas Graves commented on SPARK-45527: --- Note that this is related to SPARK-39853 which was

[jira] [Resolved] (SPARK-45527) Task fraction resource request is not expected

[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-45527. --- Resolution: Fixed > Task fraction resource request is not expected >

[jira] [Updated] (SPARK-45527) Task fraction resource request is not expected

[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-45527: -- Fix Version/s: 4.0.0 > Task fraction resource request is not expected >

[jira] [Assigned] (SPARK-45527) Task fraction resource request is not expected

[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-45527: - Assignee: Bobby Wang > Task fraction resource request is not expected >

[jira] [Commented] (SPARK-40129) Decimal multiply can produce the wrong answer because it rounds twice

[ https://issues.apache.org/jira/browse/SPARK-40129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790107#comment-17790107 ] Thomas Graves commented on SPARK-40129: --- this looks like a dup of

[jira] [Resolved] (SPARK-45937) Fix documentation of spark.executor.maxNumFailures

[ https://issues.apache.org/jira/browse/SPARK-45937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-45937. --- Resolution: Duplicate > Fix documentation of spark.executor.maxNumFailures >

[jira] [Commented] (SPARK-45937) Fix documentation of spark.executor.maxNumFailures

[ https://issues.apache.org/jira/browse/SPARK-45937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786395#comment-17786395 ] Thomas Graves commented on SPARK-45937: ---   @Cheng Pan  Could you fix this as followup? > Fix

[jira] [Created] (SPARK-45937) Fix documentation of spark.executor.maxNumFailures

Thomas Graves created SPARK-45937: - Summary: Fix documentation of spark.executor.maxNumFailures Key: SPARK-45937 URL: https://issues.apache.org/jira/browse/SPARK-45937 Project: Spark Issue

[jira] [Resolved] (SPARK-45495) Support stage level task resource profile for k8s cluster when dynamic allocation disabled

[ https://issues.apache.org/jira/browse/SPARK-45495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-45495. --- Resolution: Fixed > Support stage level task resource profile for k8s cluster when dynamic

[jira] [Commented] (SPARK-45527) Task fraction resource request is not expected

[ https://issues.apache.org/jira/browse/SPARK-45527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17774957#comment-17774957 ] Thomas Graves commented on SPARK-45527: --- thanks for filing and digging into this. I assume this is

[jira] [Updated] (SPARK-45250) Support stage level task resource profile for yarn cluster when dynamic allocation disabled

[ https://issues.apache.org/jira/browse/SPARK-45250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-45250: -- Fix Version/s: 3.5.1 > Support stage level task resource profile for yarn cluster when

[jira] [Updated] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-44940: -- Fix Version/s: 3.5.0 (was: 3.5.1) > Improve performance of JSON

[jira] [Commented] (SPARK-44940) Improve performance of JSON parsing when "spark.sql.json.enablePartialResults" is enabled

[ https://issues.apache.org/jira/browse/SPARK-44940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17769338#comment-17769338 ] Thomas Graves commented on SPARK-44940: ---  I noticed this went into 3.5.0 

[jira] [Commented] (SPARK-43919) Extract JSON functionality out of Row

[ https://issues.apache.org/jira/browse/SPARK-43919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17766400#comment-17766400 ] Thomas Graves commented on SPARK-43919: ---  This is missing description, comments, and link to the

[jira] [Commented] (SPARK-44284) Introduce simpe conf system for sql/api

[ https://issues.apache.org/jira/browse/SPARK-44284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762102#comment-17762102 ] Thomas Graves commented on SPARK-44284: --- Can we get a description on this? This seems like a

[jira] [Commented] (SPARK-44144) Enable `spark.authenticate` by default in K8s environment

[ https://issues.apache.org/jira/browse/SPARK-44144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759061#comment-17759061 ] Thomas Graves commented on SPARK-44144: --- I'm not necessarily against this but it seems odd to do

[jira] [Commented] (SPARK-44144) Enable `spark.authenticate` by default in K8s environment

[ https://issues.apache.org/jira/browse/SPARK-44144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17758602#comment-17758602 ] Thomas Graves commented on SPARK-44144: --- Can you add a description on this?  Why do we want this

[jira] [Commented] (SPARK-44871) Fix PERCENTILE_DISC behaviour

[ https://issues.apache.org/jira/browse/SPARK-44871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756077#comment-17756077 ] Thomas Graves commented on SPARK-44871: --- Can you add a description to this please > Fix

[jira] [Updated] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-44134: -- Fix Version/s: 3.4.2 (was: 3.4.1) > Can't set resources (GPU/FPGA) to

[jira] [Updated] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-44134: -- Description: With resource aware scheduling, if you specify a default value in the

[jira] [Created] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

Thomas Graves created SPARK-44134: - Summary: Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf Key: SPARK-44134 URL: https://issues.apache.org/jira/browse/SPARK-44134

[jira] [Commented] (SPARK-44134) Can't set resources (GPU/FPGA) to 0 when they are set to positive value in spark-defaults.conf

[ https://issues.apache.org/jira/browse/SPARK-44134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17735746#comment-17735746 ] Thomas Graves commented on SPARK-44134: --- I'm working on a fix for this > Can't set resources

[jira] [Resolved] (SPARK-43510) Spark application hangs when YarnAllocator adds running executors after processing completed containers

[ https://issues.apache.org/jira/browse/SPARK-43510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-43510. --- Fix Version/s: 3.4.1 3.5.0 Assignee: Manu Zhang

[jira] [Commented] (SPARK-41660) only propagate metadata columns if they are used

[ https://issues.apache.org/jira/browse/SPARK-41660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17726740#comment-17726740 ] Thomas Graves commented on SPARK-41660: --- it looks like this was backported to 3.3. with

[jira] [Updated] (SPARK-41660) only propagate metadata columns if they are used

[ https://issues.apache.org/jira/browse/SPARK-41660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-41660: -- Fix Version/s: 3.3.3 > only propagate metadata columns if they are used >

[jira] [Commented] (SPARK-43340) JsonProtocol is not backward compatible

[ https://issues.apache.org/jira/browse/SPARK-43340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17718601#comment-17718601 ] Thomas Graves commented on SPARK-43340: --- Likely related to SPARK-39489 > JsonProtocol is not

[jira] [Resolved] (SPARK-41585) The Spark exclude node functionality for YARN should work independently of dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-41585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-41585. --- Fix Version/s: 3.5.0 Target Version/s: 3.5.0 Assignee: Luca Canali

[jira] [Commented] (SPARK-41793) Incorrect result for window frames defined by a range clause on large decimals

[ https://issues.apache.org/jira/browse/SPARK-41793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692408#comment-17692408 ] Thomas Graves commented on SPARK-41793: --- [~ulysses] [~cloud_fan] [~xinrong]  We need to decide

[jira] [Commented] (SPARK-39375) SPIP: Spark Connect - A client and server interface for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-39375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17688016#comment-17688016 ] Thomas Graves commented on SPARK-39375: --- So regarding UDFs, its not clear to me how that is

[jira] [Commented] (SPARK-42374) User-facing documentaiton

[ https://issues.apache.org/jira/browse/SPARK-42374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687978#comment-17687978 ] Thomas Graves commented on SPARK-42374: --- Just a note that we should make sure to document that

[jira] [Updated] (SPARK-41793) Incorrect result for window frames defined by a range clause on large decimals

[ https://issues.apache.org/jira/browse/SPARK-41793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-41793: -- Labels: correctness (was: ) > Incorrect result for window frames defined by a range clause

[jira] [Updated] (SPARK-41793) Incorrect result for window frames defined by a range clause on large decimals

[ https://issues.apache.org/jira/browse/SPARK-41793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-41793: -- Priority: Blocker (was: Major) > Incorrect result for window frames defined by a range

[jira] [Commented] (SPARK-41793) Incorrect result for window frames defined by a range clause on large decimals

[ https://issues.apache.org/jira/browse/SPARK-41793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17678744#comment-17678744 ] Thomas Graves commented on SPARK-41793: --- this sounds like a correctness issue - [~cloud_fan]

[jira] [Resolved] (SPARK-39601) AllocationFailure should not be treated as exitCausedByApp when driver is shutting down

[ https://issues.apache.org/jira/browse/SPARK-39601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-39601. --- Fix Version/s: 3.4.0 Assignee: Cheng Pan Resolution: Fixed >

[jira] [Updated] (SPARK-40524) local mode with resource scheduling can hang

[ https://issues.apache.org/jira/browse/SPARK-40524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-40524: -- Summary: local mode with resource scheduling can hang (was: local mode with resource

[jira] [Created] (SPARK-40524) local mode with resource scheduling should just fail

Thomas Graves created SPARK-40524: - Summary: local mode with resource scheduling should just fail Key: SPARK-40524 URL: https://issues.apache.org/jira/browse/SPARK-40524 Project: Spark Issue

[jira] [Resolved] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321

[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-40490. --- Fix Version/s: 3.4.0 Assignee: Yang Jie Resolution: Fixed >

[jira] [Resolved] (SPARK-40280) Failure to create parquet predicate push down for ints and longs on some valid files

[ https://issues.apache.org/jira/browse/SPARK-40280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-40280. --- Fix Version/s: 3.4.0 3.3.1 3.2.3 Assignee:

[jira] [Commented] (SPARK-38888) Add `RocksDBProvider` similar to `LevelDBProvider`

[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579738#comment-17579738 ] Thomas Graves commented on SPARK-3: --- Just curious does rocksdb give us some particular benefit

[jira] [Updated] (SPARK-38910) Clean sparkStaging dir should before unregister()

[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38910: -- Fix Version/s: 3.4.0 > Clean sparkStaging dir should before unregister() >

[jira] [Resolved] (SPARK-38910) Clean sparkStaging dir should before unregister()

[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-38910. --- Resolution: Fixed > Clean sparkStaging dir should before unregister() >

[jira] [Assigned] (SPARK-38910) Clean sparkStaging dir should before unregister()

[ https://issues.apache.org/jira/browse/SPARK-38910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-38910: - Assignee: angerszhu > Clean sparkStaging dir should before unregister() >

[jira] [Updated] (SPARK-39976) NULL check in ArrayIntersect adds extraneous null from first param

[ https://issues.apache.org/jira/browse/SPARK-39976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39976: -- Labels: correctness (was: ) > NULL check in ArrayIntersect adds extraneous null from first

[jira] [Updated] (SPARK-39976) NULL check in ArrayIntersect adds extraneous null from first param

[ https://issues.apache.org/jira/browse/SPARK-39976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39976: -- Labels: (was: corr) > NULL check in ArrayIntersect adds extraneous null from first param >

[jira] [Updated] (SPARK-39976) NULL check in ArrayIntersect adds extraneous null from first param

[ https://issues.apache.org/jira/browse/SPARK-39976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39976: -- Priority: Blocker (was: Major) > NULL check in ArrayIntersect adds extraneous null from

[jira] [Updated] (SPARK-39976) NULL check in ArrayIntersect adds extraneous null from first param

[ https://issues.apache.org/jira/browse/SPARK-39976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39976: -- Labels: corr (was: ) > NULL check in ArrayIntersect adds extraneous null from first param >

[jira] [Commented] (SPARK-39976) NULL check in ArrayIntersect adds extraneous null from first param

[ https://issues.apache.org/jira/browse/SPARK-39976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17575398#comment-17575398 ] Thomas Graves commented on SPARK-39976: --- [~cloud_fan]  [~angerszhuuu]  who worked on original

[jira] [Created] (SPARK-39491) Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING

Thomas Graves created SPARK-39491: - Summary: Hadoop 2.7 build fails due to org.apache.hadoop.yarn.api.records.NodeState.DECOMMISSIONING Key: SPARK-39491 URL: https://issues.apache.org/jira/browse/SPARK-39491

[jira] [Commented] (SPARK-39107) Silent change in regexp_replace's handling of empty strings

[ https://issues.apache.org/jira/browse/SPARK-39107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17555111#comment-17555111 ] Thomas Graves commented on SPARK-39107: --- [~srowen]   I think this actually went into 3.1.4,  not

[jira] [Updated] (SPARK-39434) Provide runtime error query context when array index is out of bound

[ https://issues.apache.org/jira/browse/SPARK-39434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39434: -- Fix Version/s: 3.4.0 > Provide runtime error query context when array index is out of bound >

[jira] [Commented] (SPARK-39363) fix spark.kubernetes.memoryOverheadFactor deprecation warning

[ https://issues.apache.org/jira/browse/SPARK-39363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545479#comment-17545479 ] Thomas Graves commented on SPARK-39363: --- [~Kimahriman]  > fix

[jira] [Updated] (SPARK-39363) fix spark.kubernetes.memoryOverheadFactor deprecation warning

[ https://issues.apache.org/jira/browse/SPARK-39363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-39363: -- Description: see [https://github.com/apache/spark/pull/36744] for details.   It looks like

[jira] [Created] (SPARK-39363) fix spark.kubernetes.memoryOverheadFactor deprecation warning

Thomas Graves created SPARK-39363: - Summary: fix spark.kubernetes.memoryOverheadFactor deprecation warning Key: SPARK-39363 URL: https://issues.apache.org/jira/browse/SPARK-39363 Project: Spark

[jira] [Updated] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38955: -- Labels: (was: corr) > from_csv can corrupt surrounding lines if a lineSep is in the data >

[jira] [Updated] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38955: -- Priority: Blocker (was: Major) > from_csv can corrupt surrounding lines if a lineSep is in

[jira] [Updated] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38955: -- Labels: corr (was: ) > from_csv can corrupt surrounding lines if a lineSep is in the data >

[jira] [Commented] (SPARK-38955) from_csv can corrupt surrounding lines if a lineSep is in the data

[ https://issues.apache.org/jira/browse/SPARK-38955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17524985#comment-17524985 ] Thomas Graves commented on SPARK-38955: --- the from_csv docs point to the data source options which

[jira] [Updated] (SPARK-38677) pyspark hangs in local mode running rdd map operation

[ https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38677: -- Description: In spark 3.2.1 (spark 3.2.0 doesn't show this issue), pyspark will hang when

[jira] [Commented] (SPARK-38677) pyspark hangs in local mode running rdd map operation

[ https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513536#comment-17513536 ] Thomas Graves commented on SPARK-38677: --- Note, if you kill the python.daemon process while its

[jira] [Updated] (SPARK-38677) pyspark hangs in local mode running rdd map operation

[ https://issues.apache.org/jira/browse/SPARK-38677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38677: -- Affects Version/s: 3.3.0 > pyspark hangs in local mode running rdd map operation >

[jira] [Created] (SPARK-38677) pyspark hangs in local mode running rdd map operation

Thomas Graves created SPARK-38677: - Summary: pyspark hangs in local mode running rdd map operation Key: SPARK-38677 URL: https://issues.apache.org/jira/browse/SPARK-38677 Project: Spark

[jira] [Resolved] (SPARK-37618) Support cleaning up shuffle blocks from external shuffle service

[ https://issues.apache.org/jira/browse/SPARK-37618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-37618. --- Fix Version/s: 3.3.0 Assignee: Adam Binford Resolution: Fixed > Support

[jira] [Updated] (SPARK-38194) Make memory overhead factor configurable

[ https://issues.apache.org/jira/browse/SPARK-38194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38194: -- Fix Version/s: 3.3.0 > Make memory overhead factor configurable >

[jira] [Updated] (SPARK-38194) Make Yarn memory overhead factor configurable

[ https://issues.apache.org/jira/browse/SPARK-38194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38194: -- Fix Version/s: 3.3.0 (was: 3.4.0) > Make Yarn memory overhead factor

[jira] [Updated] (SPARK-38194) Make Yarn memory overhead factor configurable

[ https://issues.apache.org/jira/browse/SPARK-38194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-38194: -- Fix Version/s: 3.4.0 (was: 3.3.0) > Make Yarn memory overhead factor

[jira] [Resolved] (SPARK-38194) Make Yarn memory overhead factor configurable

[ https://issues.apache.org/jira/browse/SPARK-38194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-38194. --- Fix Version/s: 3.3.0 Assignee: Adam Binford Resolution: Fixed > Make Yarn

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503229#comment-17503229 ] Thomas Graves commented on SPARK-38379: --- so the issue here is there is a race between when

[jira] [Commented] (SPARK-34960) Aggregate (Min/Max/Count) push down for ORC

[ https://issues.apache.org/jira/browse/SPARK-34960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503221#comment-17503221 ] Thomas Graves commented on SPARK-34960: --- if I'm reading the orc spec right the ColumnStatistics

[jira] [Updated] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-36645: -- Summary: Aggregate (Min/Max/Count) push down for Parquet (was: Aggregate (Count) push down

[jira] [Comment Edited] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503212#comment-17503212 ] Thomas Graves edited comment on SPARK-36645 at 3/8/22, 10:52 PM: - Note

[jira] [Updated] (SPARK-36645) Aggregate (Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-36645: -- Summary: Aggregate (Count) push down for Parquet (was: Aggregate (Min/Max/Count) push down

[jira] [Commented] (SPARK-36645) Aggregate (Min/Max/Count) push down for Parquet

[ https://issues.apache.org/jira/browse/SPARK-36645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503212#comment-17503212 ] Thomas Graves commented on SPARK-36645: --- Note it appears this only really pushes down count

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17502555#comment-17502555 ] Thomas Graves commented on SPARK-38379: --- so I actually created another pod with Spark client in it

[jira] [Commented] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

[ https://issues.apache.org/jira/browse/SPARK-38379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500354#comment-17500354 ] Thomas Graves commented on SPARK-38379: --- just going by the stack trace this looks related to

[jira] [Created] (SPARK-38379) Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes

Thomas Graves created SPARK-38379: - Summary: Kubernetes: NoSuchElementException: spark.app.id when using PersistentVolumes Key: SPARK-38379 URL: https://issues.apache.org/jira/browse/SPARK-38379

[jira] [Commented] (SPARK-37461) yarn-client mode client's appid value is null

[ https://issues.apache.org/jira/browse/SPARK-37461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451172#comment-17451172 ] Thomas Graves commented on SPARK-37461: --- [~angerszhuuu] please add a description to this issue. >

[jira] [Created] (SPARK-37260) PYSPARK Arrow 3.2.0 docs link invalid

Thomas Graves created SPARK-37260: - Summary: PYSPARK Arrow 3.2.0 docs link invalid Key: SPARK-37260 URL: https://issues.apache.org/jira/browse/SPARK-37260 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

[ https://issues.apache.org/jira/browse/SPARK-37208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17438784#comment-17438784 ] Thomas Graves commented on SPARK-37208: --- Note, I'm working on this. > Support mapping Spark

[jira] [Created] (SPARK-37208) Support mapping Spark gpu/fpga resource types to custom YARN resource type

Thomas Graves created SPARK-37208: - Summary: Support mapping Spark gpu/fpga resource types to custom YARN resource type Key: SPARK-37208 URL: https://issues.apache.org/jira/browse/SPARK-37208

[jira] [Resolved] (SPARK-36540) AM should not just finish with Success when dissconnected

[ https://issues.apache.org/jira/browse/SPARK-36540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-36540. --- Fix Version/s: 3.3.0 Assignee: angerszhu Resolution: Fixed > AM should not

[jira] [Resolved] (SPARK-36624) When application killed, sc should not exit with code 0

[ https://issues.apache.org/jira/browse/SPARK-36624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-36624. --- Fix Version/s: 3.3.0 Assignee: angerszhu Resolution: Fixed > When

[jira] [Commented] (SPARK-36817) Does Apache Spark 3 support GPU usage for Spark RDDs?

[ https://issues.apache.org/jira/browse/SPARK-36817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17420737#comment-17420737 ] Thomas Graves commented on SPARK-36817: --- please refer to 

[jira] [Reopened] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-35672: --- > Spark fails to launch executors with very large user classpath lists on YARN >

[jira] [Commented] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419312#comment-17419312 ] Thomas Graves commented on SPARK-35672: --- Ok, sounds like we should revert then so this doesn't

[jira] [Updated] (SPARK-36772) FinalizeShuffleMerge fails with an exception due to attempt id not matching

[ https://issues.apache.org/jira/browse/SPARK-36772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-36772: -- Target Version/s: 3.2.0 > FinalizeShuffleMerge fails with an exception due to attempt id not

[jira] [Updated] (SPARK-36666) [SQL] Regression in AQEShuffleReadExec

[ https://issues.apache.org/jira/browse/SPARK-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-3: -- Priority: Blocker (was: Major) > [SQL] Regression in AQEShuffleReadExec >

[jira] [Commented] (SPARK-36622) spark.history.kerberos.principal doesn't take value _HOST

[ https://issues.apache.org/jira/browse/SPARK-36622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408813#comment-17408813 ] Thomas Graves commented on SPARK-36622: --- Supported _HOST for SHS likely makes sense since its a

[jira] [Commented] (SPARK-32333) Drop references to Master

[ https://issues.apache.org/jira/browse/SPARK-32333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405814#comment-17405814 ] Thomas Graves commented on SPARK-32333: --- I was looking to break this up into subtasks but not sure

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397461#comment-17397461 ] Thomas Graves commented on SPARK-36446: --- [~adamkennedy77] ^ > YARN shuffle server restart crashes

[jira] [Commented] (SPARK-36446) YARN shuffle server restart crashes all dynamic allocation jobs that have deallocated an executor

[ https://issues.apache.org/jira/browse/SPARK-36446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17394931#comment-17394931 ] Thomas Graves commented on SPARK-36446: --- Is this with the yarn nodemangar recovery enabled?  ie

[jira] [Assigned] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-595: --- Assignee: Yuto Akutsu > Document "local-cluster" mode > - > >

[jira] [Resolved] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-595. - Fix Version/s: 3.3.0 3.2.0 Resolution: Fixed > Document "local-cluster"

[jira] [Reopened] (SPARK-595) Document "local-cluster" mode

[ https://issues.apache.org/jira/browse/SPARK-595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reopened SPARK-595: - > Document "local-cluster" mode > - > > Key: SPARK-595 >

[jira] [Updated] (SPARK-35881) [SQL] AQE does not support columnar execution for the final query stage

[ https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-35881: -- Fix Version/s: 3.2.0 > [SQL] AQE does not support columnar execution for the final query

[jira] [Resolved] (SPARK-35881) [SQL] AQE does not support columnar execution for the final query stage

[ https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-35881. --- Fix Version/s: 3.3.0 3.2.0 Resolution: Fixed > [SQL] AQE does not

[jira] [Assigned] (SPARK-35881) [SQL] AQE does not support columnar execution for the final query stage

[ https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-35881: - Assignee: Andy Grove > [SQL] AQE does not support columnar execution for the final

  1   2   3   4   5   6   7   8   9   10   >