[
https://issues.apache.org/jira/browse/SPARK-61788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61788:
-----------------------------------
Labels: pull-request-available (was: )
> Fix StateStoreInstanceMetricSuite depending on the state store maintenance
> task of an earlier test
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-61788
> URL: https://issues.apache.org/jira/browse/SPARK-61788
> Project: Spark
> Issue Type: Test
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Raviston Thanasekar
> Priority: Major
> Labels: pull-request-available
>
> StateStoreInstanceMetricSuiteWithRowChecksum "SPARK-51097: Verify snapshot
> lag metrics are updated correctly with RocksDBStateStoreProvider (with
> changelog checkpointing)" fails when another test in the same JVM has started
> the state store maintenance task before it:
> {code}
> Assert on query failed: Execute: The code passed to eventually never returned
> normally. Attempted 669 times over 10.005817232000002 seconds. Last failure
> message: instanceMetrics.forall(((x$2: (String, Long)) =>
> scala.Predef.Long2long(x$2._2).>=(0))) was false.
> {code}
> The maintenance task is shared by the JVM, and
> StateStore.startMaintenanceIfNeeded only starts one when none is running. So
> it keeps the interval of the test that started it. The test sets
> spark.sql.streaming.stateStore.maintenanceInterval to 100 ms, but when a task
> with the default interval of 60 seconds is already running, no snapshot is
> uploaded during the test, and every SnapshotLastUploaded metric stays -1.
> In "sql - other tests", WriteDistributionAndOrderingSuite ("ordered
> distribution and sort with same exprs: micro-batch complete") started the
> task just before this test, e.g. in
> https://github.com/raviston/spark/actions/runs/43067912461 (attempts 10 to
> 12). StateStoreInstanceMetricSuiteWithRowChecksum runs in that group because
> the ExtendedSQLTest tag of StateStoreInstanceMetricSuite is not inherited.
> To reproduce:
> {code}
> build/sbt 'sql/testOnly
> org.apache.spark.sql.connector.WriteDistributionAndOrderingSuite
> org.apache.spark.sql.execution.streaming.state.StateStoreInstanceMetricSuiteWithRowChecksum
> -- -z "ordered distribution and sort with same exprs: micro-batch complete"
> -z "SPARK-51097: Verify snapshot lag metrics are updated correctly with
> RocksDBStateStoreProvider (with changelog"'
> {code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]