This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c77035238ff7 [SPARK-57603][SQL][TEST] Dump DB container logs on
docker-integration connection failure and extend Oracle connection timeout
c77035238ff7 is described below
commit c77035238ff7178526be71495deaa57a26bbb8a1
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Tue Jun 23 12:39:46 2026 +0900
[SPARK-57603][SQL][TEST] Dump DB container logs on docker-integration
connection failure and extend Oracle connection timeout
### What changes were proposed in this pull request?
`OracleIntegrationSuite` intermittently aborts in the scheduled
docker-integration
builds with `ORA-12541: No listener` after retrying the JDBC connection for
the
full `connectionTimeout` (e.g. 597 attempts over 10 minutes in run
`27894816608`). The suite is not hanging -- it actively retries ~1/s until
the
deadline -- but when it aborts, `afterAll()` immediately kills the
container, so
there is no record of WHY Oracle never accepted connections.
This PR has two parts:
1. **Diagnostics (primary):** dump the database container's own
stdout/stderr
(via `docker logs`) in the initialization failure path, before the
container
is torn down. This makes it possible to tell apart a genuinely slow
Oracle
bootstrap (where a longer timeout helps) from a container that never
came up
(where it does not), instead of guessing.
2. **Stopgap:** a per-database `connectionTimeout` override, with Oracle
set to
15 min. Evidence supports this being a real slow-bootstrap issue rather
than a
hard hang: Oracle bring-up is slow and highly variable on these runners
(observed ~7-20 min from suite start to first test on passing runs), so
a run
that crosses the 10-min default occasionally aborts. 15 min gives
headroom.
### Why are the changes needed?
To stop the flaky `ORA-12541` aborts from failing the scheduled builds, and
-- more
importantly -- to capture the evidence needed to fix the root cause rather
than
just extend a timeout.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Verified on a fork by running the docker-integration suite (which compiles
this
change and runs all JDBC suites including Oracle):
- GREEN run: https://github.com/HyukjinKwon/spark/actions/runs/27935421131
(job 82663663133) -- `Run Docker integration tests` succeeded, `Suites:
completed 27, aborted 0`, `OracleIntegrationSuite` passed (no
`ORA-12541`).
This confirms the change compiles and the `ORA-12541` failure is
intermittent.
The container-log dump only triggers on an initialization failure and is
best-effort (never throws), so it does not affect the passing path.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #56645 from HyukjinKwon/ci-fix/docker-int.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 9e8b7d356593d00872c3d0f9ef4b11d8ae22b33b)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../sql/jdbc/DockerJDBCIntegrationSuite.scala | 50 +++++++++++++++++++++-
.../spark/sql/jdbc/OracleDatabaseOnDocker.scala | 6 +++
2 files changed, 54 insertions(+), 2 deletions(-)
diff --git
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
index 48e21f0d6c12..27b9debeb1e0 100644
---
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
+++
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala
@@ -17,6 +17,7 @@
package org.apache.spark.sql.jdbc
+import java.nio.charset.StandardCharsets
import java.sql.{Connection, DriverManager}
import java.util.Properties
import java.util.concurrent.TimeUnit
@@ -96,6 +97,15 @@ abstract class DatabaseOnDocker {
def beforeContainerStart(
hostConfigBuilder: HostConfig,
containerConfigBuilder: ContainerConfig): Unit = {}
+
+ /**
+ * Optional per-database override for how long to wait for the database to
accept JDBC
+ * connections after the container has started. Some databases (e.g. Oracle)
take
+ * considerably longer to fully bootstrap their listener, so they can extend
the default.
+ * The value is a duration string such as "15min". When `None`, the suite
default
+ * (`spark.test.docker.connectionTimeout`, "10min") is used.
+ */
+ def connectionTimeout: Option[String] = None
}
abstract class DockerJDBCIntegrationSuite
@@ -111,8 +121,12 @@ abstract class DockerJDBCIntegrationSuite
timeStringAsSeconds(sys.props.getOrElse("spark.test.docker.imagePullTimeout",
"5min"))
protected val startContainerTimeout: Long =
timeStringAsSeconds(sys.props.getOrElse("spark.test.docker.startContainerTimeout",
"5min"))
- protected val connectionTimeout: PatienceConfiguration.Timeout = {
- val timeoutStr =
sys.props.getOrElse("spark.test.docker.connectionTimeout", "10min")
+ // `lazy` so that `db` (abstract, defined by concrete suites) is initialized
before it is read.
+ // A database may extend the connection timeout via `db.connectionTimeout`
(e.g. Oracle, whose
+ // container takes longer to bootstrap its listener); otherwise the suite
default is used.
+ protected lazy val connectionTimeout: PatienceConfiguration.Timeout = {
+ val timeoutStr = db.connectionTimeout.getOrElse(
+ sys.props.getOrElse("spark.test.docker.connectionTimeout", "10min"))
timeout(timeStringAsSeconds(timeoutStr).seconds)
}
@@ -223,6 +237,10 @@ abstract class DockerJDBCIntegrationSuite
case NonFatal(e) =>
logError(log"Failed to initialize Docker container for " +
log"${MDC(CLASS_NAME, this.getClass.getName)}", e)
+ // Dump the container's own logs BEFORE afterAll() tears it down, so
that a
+ // connection timeout (e.g. Oracle "ORA-12541: No listener") can be
diagnosed
+ // from the database's bootstrap output rather than guessed at.
+ dumpContainerLogs()
try {
afterAll()
} finally {
@@ -249,6 +267,34 @@ abstract class DockerJDBCIntegrationSuite
DriverManager.getConnection(jdbcUrl, db.getJdbcProperties())
}
+ /**
+ * Best-effort dump of the database container's stdout/stderr to the test
log. Used when
+ * container initialization fails (e.g. the JDBC connection never succeeds
within
+ * `connectionTimeout`) so the database's own bootstrap output is available
for diagnosis.
+ * Never throws.
+ */
+ private def dumpContainerLogs(): Unit = {
+ if (docker == null || container == null) return
+ try {
+ val logs = new StringBuilder
+ docker.logContainerCmd(container.getId)
+ .withStdOut(true)
+ .withStdErr(true)
+ .withTail(2000)
+ .exec(new ResultCallback.Adapter[Frame]() {
+ override def onNext(frame: Frame): Unit = {
+ logs.append(new String(frame.getPayload, StandardCharsets.UTF_8))
+ }
+ })
+ .awaitCompletion(60, TimeUnit.SECONDS)
+ logWarning(s"Docker container logs for ${this.getClass.getName} " +
+ s"(image=${db.imageName},
container=${container.getId}):\n${logs.toString}")
+ } catch {
+ case NonFatal(e) =>
+ logWarning(s"Failed to capture Docker container logs: ${e.getMessage}")
+ }
+ }
+
/**
* Prepare databases and tables for testing.
*/
diff --git
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleDatabaseOnDocker.scala
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleDatabaseOnDocker.scala
index baed9f5c7a5e..7f56376a812d 100644
---
a/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleDatabaseOnDocker.scala
+++
b/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleDatabaseOnDocker.scala
@@ -33,6 +33,12 @@ class OracleDatabaseOnDocker extends DatabaseOnDocker with
Logging {
override val usesIpc = false
override val jdbcPort: Int = 1521
+ // The Oracle Free container is the heaviest of the JDBC integration test
databases and
+ // intermittently takes longer than the default 10 minutes to fully
bootstrap its listener,
+ // causing the suite to abort with "ORA-12541: No listener". Give it more
headroom so the
+ // test is not flaky on slow CI runners.
+ override def connectionTimeout: Option[String] = Some("15min")
+
override def getJdbcUrl(ip: String, port: Int): String = {
s"jdbc:oracle:thin:system/$oracle_password@//$ip:$port/freepdb1"
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
