[
https://issues.apache.org/jira/browse/FLINK-40473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaodongHuan updated FLINK-40473:
---------------------------------
Description:
h3. Problem
Submitting a YAML pipeline with {{-t yarn-application}} using the official
flink-cdc-3.6.0-1.20 binary distribution always fails:
{code:java}
java.io.FileNotFoundException: Failed to fetch Flink CDC dist jar from path:
/path/to/flink-cdc-3.6.0-1.20/lib
at
org.apache.flink.cdc.composer.flink.deployment.YarnApplicationDeploymentExecutor.getFlinkCDCDistJarFromEnv(YarnApplicationDeploymentExecutor.java:151)
at
org.apache.flink.cdc.composer.flink.deployment.YarnApplicationDeploymentExecutor.deploy(YarnApplicationDeploymentExecutor.java:76)
at org.apache.flink.cdc.cli.CliExecutor.run(CliExecutor.java:75)
{code}
The {{lib}} directory does contain the dist jar. The message is misleading: the
directory is fine, but no file in it matches the expected naming pattern.
h3. Root cause
The dist jar is located by name matching
(YarnApplicationDeploymentExecutor.java:62):
{code:java}
"^flink-cdc-dist-(\\d+(\\.\\d+)*)(-SNAPSHOT)?\\.jar$"
{code}
After the version this only accepts {{-SNAPSHOT}} or {{{}.jar{}}}. Since 3.6.0
the
distribution is released per Flink minor version, and the release script sets
{{revision}} to {{<release-version>-<flink-version>}}
(tools/releasing/deploy_staging_jars.sh:83), so the shipped jar is
{{{}flink-cdc-dist-3.6.0-1.20.jar{}}}. The {{-1.20}} segment never matches.
{{mvn package}} on master keeps {{revision}}
at {{{}3.7-SNAPSHOT{}}}, which does match. The code is identical between
release-3.6.0 and master, so 3.7.0 will reproduce this once released.
!image-2026-08-26-15-45-49-268.png!
was:
h3. Problem
Submitting a YAML pipeline with {{-t yarn-application}} using the official
flink-cdc-3.6.0-1.20 binary distribution always fails:
{code:java}
java.io.FileNotFoundException: Failed to fetch Flink CDC dist jar from path:
/path/to/flink-cdc-3.6.0-1.20/lib
at
org.apache.flink.cdc.composer.flink.deployment.YarnApplicationDeploymentExecutor.getFlinkCDCDistJarFromEnv(YarnApplicationDeploymentExecutor.java:151)
at
org.apache.flink.cdc.composer.flink.deployment.YarnApplicationDeploymentExecutor.deploy(YarnApplicationDeploymentExecutor.java:76)
at org.apache.flink.cdc.cli.CliExecutor.run(CliExecutor.java:75)
{code}
The {{lib}} directory does contain the dist jar. The message is misleading: the
directory is fine, but no file in it matches the expected naming pattern.
h3. Root cause
The dist jar is located by name matching
(YarnApplicationDeploymentExecutor.java:62):
{code:java}
"^flink-cdc-dist-(\\d+(\\.\\d+)*)(-SNAPSHOT)?\\.jar$"
{code}
After the version this only accepts {{-SNAPSHOT}} or {{{}.jar{}}}. Since 3.6.0
the
distribution is released per Flink minor version, and the release script sets
{{revision}} to {{<release-version>-<flink-version>}}
(tools/releasing/deploy_staging_jars.sh:83), so the shipped jar is
{{{}flink-cdc-dist-3.6.0-1.20.jar{}}}. The {{-1.20}} segment never matches.
{{mvn package}} on master keeps {{revision}}
at {{{}3.7-SNAPSHOT{}}}, which does match. The code is identical between
release-3.6.0 and master, so 3.7.0 will reproduce this once released.
h3. Workaround
Set {{pipeline.jars}} explicitly to skip the lookup:
{code:java}
bash ./bin/flink-cdc.sh -t yarn-application \
-Dpipeline.jars=file:///path/to/lib/flink-cdc-dist-3.6.0-1.20.jar job.yaml
{code}
h3. Proposed fix
Allow an optional Flink version suffix, and include the pattern plus the actual
directory listing in the exception message:
{code:java}
"^flink-cdc-dist-(\\d+(\\.\\d+)*)(-\\d+(\\.\\d+)*)?(-SNAPSHOT)?\\.jar$"
{code}
!image-2026-08-26-15-45-49-268.png!
> YARN application mode fails to locate flink-cdc-dist jar when the jar name
> carries a Flink version suffix
> ---------------------------------------------------------------------------------------------------------
>
> Key: FLINK-40473
> URL: https://issues.apache.org/jira/browse/FLINK-40473
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.6.0, cdc-3.7.0
> Reporter: XiaodongHuan
> Priority: Major
> Attachments: image-2026-08-26-15-45-49-268.png
>
>
> h3. Problem
> Submitting a YAML pipeline with {{-t yarn-application}} using the official
> flink-cdc-3.6.0-1.20 binary distribution always fails:
> {code:java}
> java.io.FileNotFoundException: Failed to fetch Flink CDC dist jar from path:
> /path/to/flink-cdc-3.6.0-1.20/lib
> at
> org.apache.flink.cdc.composer.flink.deployment.YarnApplicationDeploymentExecutor.getFlinkCDCDistJarFromEnv(YarnApplicationDeploymentExecutor.java:151)
> at
> org.apache.flink.cdc.composer.flink.deployment.YarnApplicationDeploymentExecutor.deploy(YarnApplicationDeploymentExecutor.java:76)
> at org.apache.flink.cdc.cli.CliExecutor.run(CliExecutor.java:75)
> {code}
> The {{lib}} directory does contain the dist jar. The message is misleading:
> the
> directory is fine, but no file in it matches the expected naming pattern.
> h3. Root cause
> The dist jar is located by name matching
> (YarnApplicationDeploymentExecutor.java:62):
> {code:java}
> "^flink-cdc-dist-(\\d+(\\.\\d+)*)(-SNAPSHOT)?\\.jar$"
> {code}
> After the version this only accepts {{-SNAPSHOT}} or {{{}.jar{}}}. Since
> 3.6.0 the
> distribution is released per Flink minor version, and the release script sets
> {{revision}} to {{<release-version>-<flink-version>}}
> (tools/releasing/deploy_staging_jars.sh:83), so the shipped jar is
> {{{}flink-cdc-dist-3.6.0-1.20.jar{}}}. The {{-1.20}} segment never matches.
> {{mvn package}} on master keeps {{revision}}
> at {{{}3.7-SNAPSHOT{}}}, which does match. The code is identical between
> release-3.6.0 and master, so 3.7.0 will reproduce this once released.
> !image-2026-08-26-15-45-49-268.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
