[jira] [Updated] (SPARK-39063) Fix potential JVM crash and RocksDBIterator resource leak
[ https://issues.apache.org/jira/browse/SPARK-39063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-39063: - Description: There is lock contention between `RocksDBIterator.finalize()` and `RockDB.close()` method. If `RocksDB.close()` method is executed first, the `RocksDBIterator`s hold by `Finalizer` will not close before the `RockDB` close. If user use a `rocksdbjni` compile with `DEBUG_LEVEL=1` to enables all assertions of `rocksdbjni`(official `rocksdbjni` compile with `DEBUG_LEVEL=0` and diabled all assertions), above issue will lead to JVM crash. was:After SPARK-38896, all `LevelDB/RocksDBIterator` handle open by `LevelDB/RocksDB.view` method already closed by `tryWithResource` > Fix potential JVM crash and RocksDBIterator resource leak > - > > Key: SPARK-39063 > URL: https://issues.apache.org/jira/browse/SPARK-39063 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > > There is lock contention between `RocksDBIterator.finalize()` and > `RockDB.close()` method. If `RocksDB.close()` method is executed first, the > `RocksDBIterator`s hold by `Finalizer` will not close before the `RockDB` > close. > If user use a `rocksdbjni` compile with `DEBUG_LEVEL=1` to enables all > assertions of `rocksdbjni`(official `rocksdbjni` compile with `DEBUG_LEVEL=0` > and diabled all assertions), above issue will lead to JVM crash. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39063) Fix potential JVM crash and RocksDBIterator resource leak
[ https://issues.apache.org/jira/browse/SPARK-39063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-39063: - Summary: Fix potential JVM crash and RocksDBIterator resource leak (was: Remove `finalize()` from `LevelDB/RocksDBIterator`) > Fix potential JVM crash and RocksDBIterator resource leak > - > > Key: SPARK-39063 > URL: https://issues.apache.org/jira/browse/SPARK-39063 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > > After SPARK-38896, all `LevelDB/RocksDBIterator` handle open by > `LevelDB/RocksDB.view` method already closed by `tryWithResource` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39063) Fix potential JVM crash and RocksDBIterator resource leak
[ https://issues.apache.org/jira/browse/SPARK-39063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-39063: - Issue Type: Bug (was: Improvement) > Fix potential JVM crash and RocksDBIterator resource leak > - > > Key: SPARK-39063 > URL: https://issues.apache.org/jira/browse/SPARK-39063 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > > After SPARK-38896, all `LevelDB/RocksDBIterator` handle open by > `LevelDB/RocksDB.view` method already closed by `tryWithResource` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39063) Remove `finalize()` from `LevelDB/RocksDBIterator`
[ https://issues.apache.org/jira/browse/SPARK-39063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533413#comment-17533413 ] Apache Spark commented on SPARK-39063: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/36467 > Remove `finalize()` from `LevelDB/RocksDBIterator` > -- > > Key: SPARK-39063 > URL: https://issues.apache.org/jira/browse/SPARK-39063 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > > After SPARK-38896, all `LevelDB/RocksDBIterator` handle open by > `LevelDB/RocksDB.view` method already closed by `tryWithResource` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39093) Dividing interval by integral can result in codegen compilation error
[
https://issues.apache.org/jira/browse/SPARK-39093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533409#comment-17533409
]
Apache Spark commented on SPARK-39093:
--
User 'bersprockets' has created a pull request for this issue:
https://github.com/apache/spark/pull/36481
> Dividing interval by integral can result in codegen compilation error
> -
>
> Key: SPARK-39093
> URL: https://issues.apache.org/jira/browse/SPARK-39093
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.1, 3.3.0, 3.4.0
>Reporter: Bruce Robbins
>Assignee: Bruce Robbins
>Priority: Minor
> Fix For: 3.3.1
>
>
> Assume this data:
> {noformat}
> create or replace temp view v1 as
> select * FROM VALUES
> (interval '10' months, interval '10' day, 2)
> as v1(period, duration, num);
> cache table v1;
> {noformat}
> These two queries work:
> {noformat}
> spark-sql> select period/num from v1;
> 0-5
> Time taken: 0.143 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/num from v1;
> 5 00:00:00.0
> Time taken: 0.094 seconds, Fetched 1 row(s)
> {noformat}
> However, these two queries get a codegen compilation error:
> {noformat}
> spark-sql> select period/(num + 3) from v1;
> 22/05/03 08:56:37 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 40, Column 44: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:56:37 WARN UnsafeProjection: Expr codegen error and falling back
> to interpreter mode
> ...
> 0-2
> Time taken: 0.149 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/(num + 3) from v1;
> 22/05/03 08:57:29 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 40, Column 54: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:57:29 WARN UnsafeProjection: Expr codegen error and falling back
> to interpreter mode
> ...
> 2 00:00:00.0
> Time taken: 0.089 seconds, Fetched 1 row(s)
> {noformat}
> Even the first two queries will get a compilation error if you turn off
> whole-stage codegen:
> {noformat}
> spark-sql> set spark.sql.codegen.wholeStage=false;
> spark.sql.codegen.wholeStage false
> Time taken: 0.055 seconds, Fetched 1 row(s)
> spark-sql> select period/num from v1;
> 22/05/03 09:16:42 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 37, Column 5: Expression "value_1" is not an rvalue
>
> 0-5
> Time taken: 0.175 seconds, Fetched 1 row(s)
> spark-sql> select duration/num from v1;
> 22/05/03 09:17:41 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 37, Column 5: Expression "value_1" is not an rvalue
> ...
> 5 00:00:00.0
> Time taken: 0.104 seconds, Fetched 1 row(s)
> {noformat}
> Note that in the error cases, the queries still return a result because Spark
> falls back on interpreting the divide expression (so I marked this as
> "minor").
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39093) Dividing interval by integral can result in codegen compilation error
[
https://issues.apache.org/jira/browse/SPARK-39093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533408#comment-17533408
]
Apache Spark commented on SPARK-39093:
--
User 'bersprockets' has created a pull request for this issue:
https://github.com/apache/spark/pull/36481
> Dividing interval by integral can result in codegen compilation error
> -
>
> Key: SPARK-39093
> URL: https://issues.apache.org/jira/browse/SPARK-39093
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.1, 3.3.0, 3.4.0
>Reporter: Bruce Robbins
>Assignee: Bruce Robbins
>Priority: Minor
> Fix For: 3.3.1
>
>
> Assume this data:
> {noformat}
> create or replace temp view v1 as
> select * FROM VALUES
> (interval '10' months, interval '10' day, 2)
> as v1(period, duration, num);
> cache table v1;
> {noformat}
> These two queries work:
> {noformat}
> spark-sql> select period/num from v1;
> 0-5
> Time taken: 0.143 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/num from v1;
> 5 00:00:00.0
> Time taken: 0.094 seconds, Fetched 1 row(s)
> {noformat}
> However, these two queries get a codegen compilation error:
> {noformat}
> spark-sql> select period/(num + 3) from v1;
> 22/05/03 08:56:37 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 40, Column 44: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:56:37 WARN UnsafeProjection: Expr codegen error and falling back
> to interpreter mode
> ...
> 0-2
> Time taken: 0.149 seconds, Fetched 1 row(s)
> {noformat}
> {noformat}
> spark-sql> select duration/(num + 3) from v1;
> 22/05/03 08:57:29 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 40, Column 54: Expression "project_value_2" is not an rvalue
> ...
> 22/05/03 08:57:29 WARN UnsafeProjection: Expr codegen error and falling back
> to interpreter mode
> ...
> 2 00:00:00.0
> Time taken: 0.089 seconds, Fetched 1 row(s)
> {noformat}
> Even the first two queries will get a compilation error if you turn off
> whole-stage codegen:
> {noformat}
> spark-sql> set spark.sql.codegen.wholeStage=false;
> spark.sql.codegen.wholeStage false
> Time taken: 0.055 seconds, Fetched 1 row(s)
> spark-sql> select period/num from v1;
> 22/05/03 09:16:42 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 37, Column 5: Expression "value_1" is not an rvalue
>
> 0-5
> Time taken: 0.175 seconds, Fetched 1 row(s)
> spark-sql> select duration/num from v1;
> 22/05/03 09:17:41 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 37, Column 5: Expression "value_1" is not an rvalue
> ...
> 5 00:00:00.0
> Time taken: 0.104 seconds, Fetched 1 row(s)
> {noformat}
> Note that in the error cases, the queries still return a result because Spark
> falls back on interpreting the divide expression (so I marked this as
> "minor").
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39124) Upgrade rocksdbjni to 7.1.2
[ https://issues.apache.org/jira/browse/SPARK-39124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533377#comment-17533377 ] Apache Spark commented on SPARK-39124: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/36480 > Upgrade rocksdbjni to 7.1.2 > --- > > Key: SPARK-39124 > URL: https://issues.apache.org/jira/browse/SPARK-39124 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39124) Upgrade rocksdbjni to 7.1.2
[ https://issues.apache.org/jira/browse/SPARK-39124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39124: Assignee: (was: Apache Spark) > Upgrade rocksdbjni to 7.1.2 > --- > > Key: SPARK-39124 > URL: https://issues.apache.org/jira/browse/SPARK-39124 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39124) Upgrade rocksdbjni to 7.1.2
[ https://issues.apache.org/jira/browse/SPARK-39124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-39124: Assignee: Apache Spark > Upgrade rocksdbjni to 7.1.2 > --- > > Key: SPARK-39124 > URL: https://issues.apache.org/jira/browse/SPARK-39124 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39124) Upgrade rocksdbjni to 7.1.2
[ https://issues.apache.org/jira/browse/SPARK-39124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533376#comment-17533376 ] Apache Spark commented on SPARK-39124: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/36480 > Upgrade rocksdbjni to 7.1.2 > --- > > Key: SPARK-39124 > URL: https://issues.apache.org/jira/browse/SPARK-39124 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.4.0 >Reporter: Dongjoon Hyun >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39124) Upgrade rocksdbjni to 7.1.2
Dongjoon Hyun created SPARK-39124: - Summary: Upgrade rocksdbjni to 7.1.2 Key: SPARK-39124 URL: https://issues.apache.org/jira/browse/SPARK-39124 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.4.0 Reporter: Dongjoon Hyun -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-38688) Use error classes in the compilation errors of deserializer
[ https://issues.apache.org/jira/browse/SPARK-38688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533375#comment-17533375 ] Apache Spark commented on SPARK-38688: -- User 'panbingkun' has created a pull request for this issue: https://github.com/apache/spark/pull/36479 > Use error classes in the compilation errors of deserializer > --- > > Key: SPARK-38688 > URL: https://issues.apache.org/jira/browse/SPARK-38688 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.4.0 >Reporter: Max Gekk >Priority: Major > > Migrate the following errors in QueryCompilationErrors: > * dataTypeMismatchForDeserializerError > * fieldNumberMismatchForDeserializerError > onto use error classes. Throw an implementation of SparkThrowable. Also write > a test per every error in QueryCompilationErrorsSuite. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-38688) Use error classes in the compilation errors of deserializer
[ https://issues.apache.org/jira/browse/SPARK-38688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38688: Assignee: (was: Apache Spark) > Use error classes in the compilation errors of deserializer > --- > > Key: SPARK-38688 > URL: https://issues.apache.org/jira/browse/SPARK-38688 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.4.0 >Reporter: Max Gekk >Priority: Major > > Migrate the following errors in QueryCompilationErrors: > * dataTypeMismatchForDeserializerError > * fieldNumberMismatchForDeserializerError > onto use error classes. Throw an implementation of SparkThrowable. Also write > a test per every error in QueryCompilationErrorsSuite. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-38688) Use error classes in the compilation errors of deserializer
[ https://issues.apache.org/jira/browse/SPARK-38688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-38688: Assignee: Apache Spark > Use error classes in the compilation errors of deserializer > --- > > Key: SPARK-38688 > URL: https://issues.apache.org/jira/browse/SPARK-38688 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.4.0 >Reporter: Max Gekk >Assignee: Apache Spark >Priority: Major > > Migrate the following errors in QueryCompilationErrors: > * dataTypeMismatchForDeserializerError > * fieldNumberMismatchForDeserializerError > onto use error classes. Throw an implementation of SparkThrowable. Also write > a test per every error in QueryCompilationErrorsSuite. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39026) Add k8s tolerations support for apache spark
[ https://issues.apache.org/jira/browse/SPARK-39026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39026. --- Fix Version/s: (was: 3.4.0) Resolution: Duplicate This issue is closed as `Duplicated` of SPARK-24434 because there is no justification of new configuration. > Add k8s tolerations support for apache spark > > > Key: SPARK-39026 > URL: https://issues.apache.org/jira/browse/SPARK-39026 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.2.1 >Reporter: qian >Priority: Major > > [https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/] > As document shows, this issue aims to support tolerations for apache spark -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39025) Add k8s Scheduling, Preemption and Eviction feature to apache spark
[ https://issues.apache.org/jira/browse/SPARK-39025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-39025. --- Resolution: Duplicate This issue is closed as a duplicate of SPARK-24434 because there is no justification of the claims. > Add k8s Scheduling, Preemption and Eviction feature to apache spark > --- > > Key: SPARK-39025 > URL: https://issues.apache.org/jira/browse/SPARK-39025 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 3.2.1 >Reporter: qian >Priority: Major > > As [https://kubernetes.io/docs/concepts/scheduling-eviction/] show, apache > spark lacks of support for k8s scheduling, preemption and eviction. > This issue wants to support toleration/priorityClass/runtimeClass etc , more > information please refer > [https://kubernetes.io/docs/concepts/scheduling-eviction/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/api-eviction/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/resource-bin-packing/] > [https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/] > https://kubernetes.io/docs/concepts/scheduling-eviction/scheduler-perf-tuning/ -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure
[
https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-39006:
--
Priority: Minor (was: Major)
> Show a directional error message for PVC Dynamic Allocation Failure
> ---
>
> Key: SPARK-39006
> URL: https://issues.apache.org/jira/browse/SPARK-39006
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.4.0
>Reporter: qian
>Assignee: qian
>Priority: Minor
> Fix For: 3.4.0
>
>
> When spark application requires multiple executors and not set pvc claimName
> with onDemand or SPARK_EXECUTOR_ID, it always create executor pods. Because
> pvc has be created by first executor pod.
> {noformat}
> 22/04/22 08:55:47 WARN ExecutorPodsSnapshotsStoreImpl: Exception when
> notifying snapshot subscriber.
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at:
> https://kubernetes.default.svc/api/v1/namespaces/default/persistentvolumeclaims.
> Message: persistentvolumeclaims "test-1" already exists. Received status:
> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null,
> kind=persistentvolumeclaims, name=test-1, retryAfterSeconds=null, uid=null,
> additionalProperties={}), kind=Status, message=persistentvolumeclaims
> "test-1" already exists, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=AlreadyExists, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$3(ExecutorPodsAllocator.scala:415)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.List.foreach(List.scala:431)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:408)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:385)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35(ExecutorPodsAllocator.scala:349)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35$adapted(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:120)
[jira] [Assigned] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure
[
https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-39006:
Assignee: qian (was: Apache Spark)
> Show a directional error message for PVC Dynamic Allocation Failure
> ---
>
> Key: SPARK-39006
> URL: https://issues.apache.org/jira/browse/SPARK-39006
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: qian
>Assignee: qian
>Priority: Major
>
> When spark application requires multiple executors and not set pvc claimName
> with onDemand or SPARK_EXECUTOR_ID, it always create executor pods. Because
> pvc has be created by first executor pod.
> {noformat}
> 22/04/22 08:55:47 WARN ExecutorPodsSnapshotsStoreImpl: Exception when
> notifying snapshot subscriber.
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at:
> https://kubernetes.default.svc/api/v1/namespaces/default/persistentvolumeclaims.
> Message: persistentvolumeclaims "test-1" already exists. Received status:
> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null,
> kind=persistentvolumeclaims, name=test-1, retryAfterSeconds=null, uid=null,
> additionalProperties={}), kind=Status, message=persistentvolumeclaims
> "test-1" already exists, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=AlreadyExists, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$3(ExecutorPodsAllocator.scala:415)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.List.foreach(List.scala:431)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:408)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:385)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35(ExecutorPodsAllocator.scala:349)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35$adapted(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:120)
> ~[spark-kubernete
[jira] [Reopened] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure
[
https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reopened SPARK-39006:
---
> Show a directional error message for PVC Dynamic Allocation Failure
> ---
>
> Key: SPARK-39006
> URL: https://issues.apache.org/jira/browse/SPARK-39006
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: qian
>Assignee: qian
>Priority: Major
>
> When spark application requires multiple executors and not set pvc claimName
> with onDemand or SPARK_EXECUTOR_ID, it always create executor pods. Because
> pvc has be created by first executor pod.
> {noformat}
> 22/04/22 08:55:47 WARN ExecutorPodsSnapshotsStoreImpl: Exception when
> notifying snapshot subscriber.
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at:
> https://kubernetes.default.svc/api/v1/namespaces/default/persistentvolumeclaims.
> Message: persistentvolumeclaims "test-1" already exists. Received status:
> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null,
> kind=persistentvolumeclaims, name=test-1, retryAfterSeconds=null, uid=null,
> additionalProperties={}), kind=Status, message=persistentvolumeclaims
> "test-1" already exists, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=AlreadyExists, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$3(ExecutorPodsAllocator.scala:415)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.List.foreach(List.scala:431)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:408)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:385)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35(ExecutorPodsAllocator.scala:349)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35$adapted(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:120)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
>
[jira] [Updated] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure
[
https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-39006:
--
Affects Version/s: 3.4.0
(was: 3.1.0)
> Show a directional error message for PVC Dynamic Allocation Failure
> ---
>
> Key: SPARK-39006
> URL: https://issues.apache.org/jira/browse/SPARK-39006
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.4.0
>Reporter: qian
>Assignee: qian
>Priority: Major
> Fix For: 3.4.0
>
>
> When spark application requires multiple executors and not set pvc claimName
> with onDemand or SPARK_EXECUTOR_ID, it always create executor pods. Because
> pvc has be created by first executor pod.
> {noformat}
> 22/04/22 08:55:47 WARN ExecutorPodsSnapshotsStoreImpl: Exception when
> notifying snapshot subscriber.
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at:
> https://kubernetes.default.svc/api/v1/namespaces/default/persistentvolumeclaims.
> Message: persistentvolumeclaims "test-1" already exists. Received status:
> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null,
> kind=persistentvolumeclaims, name=test-1, retryAfterSeconds=null, uid=null,
> additionalProperties={}), kind=Status, message=persistentvolumeclaims
> "test-1" already exists, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=AlreadyExists, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$3(ExecutorPodsAllocator.scala:415)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.List.foreach(List.scala:431)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:408)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:385)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35(ExecutorPodsAllocator.scala:349)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35$adapted(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$
[jira] [Assigned] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure
[
https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-39006:
Assignee: Apache Spark (was: qian)
> Show a directional error message for PVC Dynamic Allocation Failure
> ---
>
> Key: SPARK-39006
> URL: https://issues.apache.org/jira/browse/SPARK-39006
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: qian
>Assignee: Apache Spark
>Priority: Major
>
> When spark application requires multiple executors and not set pvc claimName
> with onDemand or SPARK_EXECUTOR_ID, it always create executor pods. Because
> pvc has be created by first executor pod.
> {noformat}
> 22/04/22 08:55:47 WARN ExecutorPodsSnapshotsStoreImpl: Exception when
> notifying snapshot subscriber.
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at:
> https://kubernetes.default.svc/api/v1/namespaces/default/persistentvolumeclaims.
> Message: persistentvolumeclaims "test-1" already exists. Received status:
> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null,
> kind=persistentvolumeclaims, name=test-1, retryAfterSeconds=null, uid=null,
> additionalProperties={}), kind=Status, message=persistentvolumeclaims
> "test-1" already exists, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=AlreadyExists, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$3(ExecutorPodsAllocator.scala:415)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.List.foreach(List.scala:431)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:408)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:385)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35(ExecutorPodsAllocator.scala:349)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35$adapted(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:120)
> ~[spark-k
[jira] [Resolved] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure
[
https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-39006.
---
Fix Version/s: 3.4.0
Resolution: Fixed
This is resolved via https://github.com/apache/spark/pull/36374
> Show a directional error message for PVC Dynamic Allocation Failure
> ---
>
> Key: SPARK-39006
> URL: https://issues.apache.org/jira/browse/SPARK-39006
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: qian
>Assignee: qian
>Priority: Major
> Fix For: 3.4.0
>
>
> When spark application requires multiple executors and not set pvc claimName
> with onDemand or SPARK_EXECUTOR_ID, it always create executor pods. Because
> pvc has be created by first executor pod.
> {noformat}
> 22/04/22 08:55:47 WARN ExecutorPodsSnapshotsStoreImpl: Exception when
> notifying snapshot subscriber.
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at:
> https://kubernetes.default.svc/api/v1/namespaces/default/persistentvolumeclaims.
> Message: persistentvolumeclaims "test-1" already exists. Received status:
> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null,
> kind=persistentvolumeclaims, name=test-1, retryAfterSeconds=null, uid=null,
> additionalProperties={}), kind=Status, message=persistentvolumeclaims
> "test-1" already exists, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=AlreadyExists, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$3(ExecutorPodsAllocator.scala:415)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.List.foreach(List.scala:431)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:408)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:385)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35(ExecutorPodsAllocator.scala:349)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35$adapted(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.c
[jira] [Assigned] (SPARK-39006) Show a directional error message for PVC Dynamic Allocation Failure
[
https://issues.apache.org/jira/browse/SPARK-39006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-39006:
-
Assignee: qian
> Show a directional error message for PVC Dynamic Allocation Failure
> ---
>
> Key: SPARK-39006
> URL: https://issues.apache.org/jira/browse/SPARK-39006
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: qian
>Assignee: qian
>Priority: Major
>
> When spark application requires multiple executors and not set pvc claimName
> with onDemand or SPARK_EXECUTOR_ID, it always create executor pods. Because
> pvc has be created by first executor pod.
> {noformat}
> 22/04/22 08:55:47 WARN ExecutorPodsSnapshotsStoreImpl: Exception when
> notifying snapshot subscriber.
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing:
> POST at:
> https://kubernetes.default.svc/api/v1/namespaces/default/persistentvolumeclaims.
> Message: persistentvolumeclaims "test-1" already exists. Received status:
> Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null,
> kind=persistentvolumeclaims, name=test-1, retryAfterSeconds=null, uid=null,
> additionalProperties={}), kind=Status, message=persistentvolumeclaims
> "test-1" already exists, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=AlreadyExists, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:697)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:676)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:629)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:566)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:527)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:315)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:651)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:91)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61)
> ~[kubernetes-client-5.10.1.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$3(ExecutorPodsAllocator.scala:415)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.List.foreach(List.scala:431)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$requestNewExecutors$1(ExecutorPodsAllocator.scala:408)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.requestNewExecutors(ExecutorPodsAllocator.scala:385)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35(ExecutorPodsAllocator.scala:349)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$onNewSnapshots$35$adapted(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.onNewSnapshots(ExecutorPodsAllocator.scala:342)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
> at
> org.apache.spark.scheduler.cluster.k8s.ExecutorPodsAllocator.$anonfun$start$3(ExecutorPodsAllocator.scala:120)
> ~[spark-kubernetes_2.12-3.3.0-SNAPSH
[jira] [Resolved] (SPARK-39123) Upgrade `org.scalatestplus:mockito` to 3.2.12.0
[
https://issues.apache.org/jira/browse/SPARK-39123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-39123.
---
Fix Version/s: 3.4.0
Assignee: Yang Jie
Resolution: Fixed
This is resolved via https://github.com/apache/spark/pull/36478
> Upgrade `org.scalatestplus:mockito` to 3.2.12.0
> ---
>
> Key: SPARK-39123
> URL: https://issues.apache.org/jira/browse/SPARK-39123
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Assignee: Yang Jie
>Priority: Minor
> Fix For: 3.4.0
>
>
> `mockito-4-2_${scala.binary.version}:3.2.11.0` ->
> `mockito-4-5_${scala.binary.version}:3.2.12.0`
>
> mockito 4.2.0 -> 4.5.1
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-28594) Allow event logs for running streaming apps to be rolled over
[ https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533313#comment-17533313 ] Itay Bittan commented on SPARK-28594: - Hi, Just want to highlight the cost (in terms of money) of the new feature. I'm running tens of thousands of Spark jobs (in Kubernetes) every day. I have noticed that I pay dozens of dollars for `ListBucket` operation in S3. After debugging spark-history I found that every 10s ([default|https://spark.apache.org/docs/latest/monitoring.html#spark-history-server-configuration-options]) we perform O(N) `ListBucket` operations - to get the content each folder. A better solution could be to perform a deep listing as suggested [here|https://stackoverflow.com/a/71195428/1011253]. I tried to do it but it seems like there's abstract file system class and it would require a massive change. > Allow event logs for running streaming apps to be rolled over > - > > Key: SPARK-28594 > URL: https://issues.apache.org/jira/browse/SPARK-28594 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Stephen Levett >Assignee: Jungtaek Lim >Priority: Major > Labels: releasenotes > Fix For: 3.0.0 > > > At all current Spark releases when event logging on spark streaming is > enabled the event logs grow massively. The files continue to grow until the > application is stopped or killed. > The Spark history server then has difficulty processing the files. > https://issues.apache.org/jira/browse/SPARK-8617 > Addresses .inprogress files but not event log files that are still running. > Identify a mechanism to set a "max file" size so that the file is rolled over > when it reaches this size? > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39097) Improve the test coverage for pyspark/taskcontext.py
[ https://issues.apache.org/jira/browse/SPARK-39097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533251#comment-17533251 ] pralabhkumar commented on SPARK-39097: -- [~hyukjin.kwon] While analysis the unit test cases for taskcontext in test_taskcontext.py , i found most of the test cases are there . However its not coming in the coverage , possibly because methods are called inside tasks (rdd.map(lambda x: TaskContext.get().stageId()). So for e.g report says [https://app.codecov.io/gh/apache/spark/blob/master/python/pyspark/taskcontext.py] stageID is not covered in the test case. However test , test_stage_id is testing the stageid method . stage1 = rdd.map(lambda x: TaskContext.get().stageId()).take(1)[0] If I change the code to below and bring taskcontext to driver , then the coverage report says stageid is covered via unit test case. rdd.map(lambda x: TaskContext.get()).take(1)[0].stageId() I can change the code to the above one to have the coverage , please let me know , if this is correct. > Improve the test coverage for pyspark/taskcontext.py > - > > Key: SPARK-39097 > URL: https://issues.apache.org/jira/browse/SPARK-39097 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: pralabhkumar >Assignee: pralabhkumar >Priority: Minor > Fix For: 3.4.0 > > > Improve the test coverage of taskcontext.py -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39097) Improve the test coverage for pyspark/taskcontext.py
[ https://issues.apache.org/jira/browse/SPARK-39097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pralabhkumar updated SPARK-39097: - Description: Improve the test coverage of taskcontext.py (was: Improve the test coverage of rddsampler.py) > Improve the test coverage for pyspark/taskcontext.py > - > > Key: SPARK-39097 > URL: https://issues.apache.org/jira/browse/SPARK-39097 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: pralabhkumar >Assignee: pralabhkumar >Priority: Minor > Fix For: 3.4.0 > > > Improve the test coverage of taskcontext.py -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39123) Upgrade `org.scalatestplus:mockito` to 3.2.12.0
[
https://issues.apache.org/jira/browse/SPARK-39123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-39123:
Assignee: Apache Spark
> Upgrade `org.scalatestplus:mockito` to 3.2.12.0
> ---
>
> Key: SPARK-39123
> URL: https://issues.apache.org/jira/browse/SPARK-39123
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Assignee: Apache Spark
>Priority: Minor
>
> `mockito-4-2_${scala.binary.version}:3.2.11.0` ->
> `mockito-4-5_${scala.binary.version}:3.2.12.0`
>
> mockito 4.2.0 -> 4.5.1
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-39123) Upgrade `org.scalatestplus:mockito` to 3.2.12.0
[
https://issues.apache.org/jira/browse/SPARK-39123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533239#comment-17533239
]
Apache Spark commented on SPARK-39123:
--
User 'LuciferYang' has created a pull request for this issue:
https://github.com/apache/spark/pull/36478
> Upgrade `org.scalatestplus:mockito` to 3.2.12.0
> ---
>
> Key: SPARK-39123
> URL: https://issues.apache.org/jira/browse/SPARK-39123
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Priority: Minor
>
> `mockito-4-2_${scala.binary.version}:3.2.11.0` ->
> `mockito-4-5_${scala.binary.version}:3.2.12.0`
>
> mockito 4.2.0 -> 4.5.1
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39123) Upgrade `org.scalatestplus:mockito` to 3.2.12.0
[
https://issues.apache.org/jira/browse/SPARK-39123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-39123:
Assignee: (was: Apache Spark)
> Upgrade `org.scalatestplus:mockito` to 3.2.12.0
> ---
>
> Key: SPARK-39123
> URL: https://issues.apache.org/jira/browse/SPARK-39123
> Project: Spark
> Issue Type: Improvement
> Components: Build
>Affects Versions: 3.4.0
>Reporter: Yang Jie
>Priority: Minor
>
> `mockito-4-2_${scala.binary.version}:3.2.11.0` ->
> `mockito-4-5_${scala.binary.version}:3.2.12.0`
>
> mockito 4.2.0 -> 4.5.1
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39114) ml.optim.aggregator avoid re-allocating buffers
[ https://issues.apache.org/jira/browse/SPARK-39114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-39114. -- Fix Version/s: 3.4.0 Resolution: Resolved > ml.optim.aggregator avoid re-allocating buffers > --- > > Key: SPARK-39114 > URL: https://issues.apache.org/jira/browse/SPARK-39114 > Project: Spark > Issue Type: Sub-task > Components: ML >Affects Versions: 3.4.0 >Reporter: zhengruifeng >Priority: Minor > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39123) Upgrade `org.scalatestplus:mockito` to 3.2.12.0
Yang Jie created SPARK-39123:
Summary: Upgrade `org.scalatestplus:mockito` to 3.2.12.0
Key: SPARK-39123
URL: https://issues.apache.org/jira/browse/SPARK-39123
Project: Spark
Issue Type: Improvement
Components: Build
Affects Versions: 3.4.0
Reporter: Yang Jie
`mockito-4-2_${scala.binary.version}:3.2.11.0` ->
`mockito-4-5_${scala.binary.version}:3.2.12.0`
mockito 4.2.0 -> 4.5.1
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-39122) Python UDF does not follow the conditional expression evaluation order
XiDuo You created SPARK-39122:
-
Summary: Python UDF does not follow the conditional expression
evaluation order
Key: SPARK-39122
URL: https://issues.apache.org/jira/browse/SPARK-39122
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 3.4.0
Reporter: XiDuo You
We will pull out python udf and run them eagerly whatever can be reached or
not, so the below query will fail:
{code:java}
SELECT if(true, 1, python_udf(1/0)) {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-39121) Fix doc format/syntax error
[ https://issues.apache.org/jira/browse/SPARK-39121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-39121. -- Fix Version/s: 3.3.1 3.4.0 Resolution: Fixed Issue resolved by pull request 36476 [https://github.com/apache/spark/pull/36476] > Fix doc format/syntax error > --- > > Key: SPARK-39121 > URL: https://issues.apache.org/jira/browse/SPARK-39121 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Yikun Jiang >Assignee: Yikun Jiang >Priority: Major > Fix For: 3.3.1, 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-39121) Fix doc format/syntax error
[ https://issues.apache.org/jira/browse/SPARK-39121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk reassigned SPARK-39121: Assignee: Yikun Jiang > Fix doc format/syntax error > --- > > Key: SPARK-39121 > URL: https://issues.apache.org/jira/browse/SPARK-39121 > Project: Spark > Issue Type: Sub-task > Components: Kubernetes >Affects Versions: 3.3.0 >Reporter: Yikun Jiang >Assignee: Yikun Jiang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
