[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608174#comment-17608174 ] Dongjoon Hyun commented on SPARK-40490: --- This landed at branch-3.3 via [https://github.com/apache/spark/pull/37962] > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload > after SPARK-17321 > > > Key: SPARK-40490 > URL: https://issues.apache.org/jira/browse/SPARK-40490 > Project: Spark > Issue Type: Improvement > Components: Tests, YARN >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Fix For: 3.4.0, 3.3.2 > > > After SPARK-17321, YarnShuffleService will persist data to local shuffle > state db and reload data from local shuffle state db only when Yarn > NodeManager start with `YarnConfiguration#NM_RECOVERY_ENABLED = true` , but > `YarnShuffleIntegrationSuite` not set this config and the default value of > the configuration is false, so `YarnShuffleIntegrationSuite` will neither > trigger data persistence to the db nor verify the reload of data > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608042#comment-17608042 ] Apache Spark commented on SPARK-40490: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/37963 > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload > after SPARK-17321 > > > Key: SPARK-40490 > URL: https://issues.apache.org/jira/browse/SPARK-40490 > Project: Spark > Issue Type: Improvement > Components: Tests, YARN >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Fix For: 3.4.0 > > > After SPARK-17321, YarnShuffleService will persist data to local shuffle > state db and reload data from local shuffle state db only when Yarn > NodeManager start with `YarnConfiguration#NM_RECOVERY_ENABLED = true` , but > `YarnShuffleIntegrationSuite` not set this config and the default value of > the configuration is false, so `YarnShuffleIntegrationSuite` will neither > trigger data persistence to the db nor verify the reload of data > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608043#comment-17608043 ] Apache Spark commented on SPARK-40490: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/37962 > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload > after SPARK-17321 > > > Key: SPARK-40490 > URL: https://issues.apache.org/jira/browse/SPARK-40490 > Project: Spark > Issue Type: Improvement > Components: Tests, YARN >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Yang Jie >Priority: Major > Fix For: 3.4.0 > > > After SPARK-17321, YarnShuffleService will persist data to local shuffle > state db and reload data from local shuffle state db only when Yarn > NodeManager start with `YarnConfiguration#NM_RECOVERY_ENABLED = true` , but > `YarnShuffleIntegrationSuite` not set this config and the default value of > the configuration is false, so `YarnShuffleIntegrationSuite` will neither > trigger data persistence to the db nor verify the reload of data > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606851#comment-17606851 ] Apache Spark commented on SPARK-40490: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/37938 > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload > after SPARK-17321 > > > Key: SPARK-40490 > URL: https://issues.apache.org/jira/browse/SPARK-40490 > Project: Spark > Issue Type: Improvement > Components: Tests, YARN >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Major > > After SPARK-17321, YarnShuffleService will persist data to local shuffle > state db and reload data from local shuffle state db only when Yarn > NodeManager start with `YarnConfiguration#NM_RECOVERY_ENABLED = true` , but > `YarnShuffleIntegrationSuite` not set this config and the default value of > the configuration is false, so `YarnShuffleIntegrationSuite` will neither > trigger data persistence to the db nor verify the reload of data > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40490) `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload after SPARK-17321
[ https://issues.apache.org/jira/browse/SPARK-40490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17606849#comment-17606849 ] Apache Spark commented on SPARK-40490: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/37938 > `YarnShuffleIntegrationSuite` no longer verifies `registeredExecFile` reload > after SPARK-17321 > > > Key: SPARK-40490 > URL: https://issues.apache.org/jira/browse/SPARK-40490 > Project: Spark > Issue Type: Improvement > Components: Tests, YARN >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Major > > After SPARK-17321, YarnShuffleService will persist data to local shuffle > state db and reload data from local shuffle state db only when Yarn > NodeManager start with `YarnConfiguration#NM_RECOVERY_ENABLED = true` , but > `YarnShuffleIntegrationSuite` not set this config and the default value of > the configuration is false, so `YarnShuffleIntegrationSuite` will neither > trigger data persistence to the db nor verify the reload of data > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
