bamaer opened a new issue, #8041: URL: https://github.com/apache/hop/issues/8041
### What would you like to happen? Problem Marketplace plugins are shipped as zips, and a plugin zip can be missing a jar it needs at runtime without anything in the build noticing. The plugin compiles, its unit tests pass, and the failure only appears when a user installs it and runs a pipeline — as a NoClassDefFoundError at the first row. This happens because plugin dependencies are routed to two different places: a plugin's private lib/ directory, and the shared lib/core. A dependency that is mis-scoped, or dropped by a wildcard <exclusion>*:*</exclusion>, silently lands in neither. Nothing in CI compares what a plugin needs against what its zip actually contains. A recent case (#8036): a technology plugin declared a Hadoop transitive at test scope while needing it at runtime. The assembly component only routes provided-scope dependencies to lib/core, so the jar shipped nowhere, and both of that plugin's transforms failed immediately on use. It had worked previously only because a different plugin happened to place the same jar in lib/core — and that plugin was later moved out of the default client. That last detail is the general shape of the risk: plugins can free-ride on jars another plugin contributes to lib/core, and nothing detects it until that other plugin moves or changes. Root gap PR CI runs mvn clean install ... -Dassemblies=false -Dmodule.zips=false (.github/workflows/pr_build_code.yml). Those two flags disable the assemblies module and every plugin's maven-assembly-plugin execution (pom.xml, the assemblies and assembly profiles). No plugin zip is built on a PR at all. The packaging layer where these bugs live is entirely untested — a change that rewrites every assembly descriptor in the repo can merge green. Fixing this is a prerequisite: add a nightly or pre-release job that builds the plugin zips. Proposals 2 and 3 below depend on it. Proposed guardrails 1. Lint mis-scoped dependencies — cheap, runs at PR time Fail the build when a plugin pom declares a test-scoped dependency that isn't a recognised test library. This is the exact shape of the bug above. It needs no zips, so it fits the current PR build. There are only a handful of such declarations repo-wide today and they're all legitimate (embedded SSH/FTP servers, cross-module test fixtures), so it starts green and stays meaningful. 2. A packaging classpath check — nightly A tools/check-plugin-classpath.sh that, for each plugin listed in optional-plugins.yaml: unpacks the zip, resolves its classes with jdeps against a lean client, and fails on unresolved references. Needs a checked-in allowlist for genuinely optional packages, otherwise it drowns in noise from third-party optional dependencies. This catches gaps on code paths no functional test exercises — for example an SSH library losing algorithms because an optional crypto provider is no longer on the classpath. It fits alongside the existing tools/check-assembly-size.sh and tools/verify-ci-snapshot-zips.sh. 3. A smoke test per marketplace plugin — highest value One minimal pipeline per plugin, run against a lean client with only that plugin installed. The isolation is the entire point, and it is easy to get wrong: a -Pfull build masks this class of bug. With every plugin present, one plugin's lib/core contribution satisfies another's missing dependency, which is exactly how the Parquet gap stayed hidden. A smoke test against a full distribution would have passed while real installs failed. Each plugin must be tested alone. Risk: Several marketplace plugins currently have no integration test at all. Suggested order Nightly job that builds the plugin zips (prerequisite) Proposal 1 — small, immediate, PR-time Proposal 3 — the one that would catch outright breakage Proposal 2 — covers what smoke tests don't reach Proposal 3 alone would catch a plugin that is fundamentally broken on install. Proposal 2 is what catches quieter degradations. Proposal 1 is near-free and is the only one that can run on a PR as CI is configured today. ### Issue Priority Priority: 2 ### Issue Component Component: Marketplace -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
