hansva commented on code in PR #9618:
URL: https://github.com/apache/hop/pull/9618#discussion_r4652602990
##########
plugins/misc/lint/src/main/java/org/apache/hop/lint/CustomRuleExecutor.java:
##########
@@ -1512,6 +1707,52 @@ private static int sizeOf(Object value) {
return -1;
}
+ /**
+ * The map an option condition reads. Anything else is a mistake in the
rule, reported as one
+ * rather than passed: a rule that cannot run must not look like a rule that
found nothing.
+ */
+ private static Map<?, ?> optionsOf(Object value, RuleCondition condition,
CustomLintRule rule) {
+ if (value instanceof Map<?, ?> map) {
+ return map;
+ }
+ throw new IllegalStateException(
+ "Rule '"
+ + rule.generateRuleId()
+ + "' uses "
+ + condition.name()
+ + " on a field that holds no options. It reads a map, such as a
connection's"
+ + " attributes.");
+ }
+
+ /**
+ * The value of an option, or null when it is not set. The option is found
by its full key or by
+ * its own name after a prefix: a connection stores its extra options as
{@code
+ * EXTRA_OPTION_POSTGRESQL.sslmode}, and a rule names it {@code sslmode}.
Comparing the map's text
+ * with a pattern instead broke on any value holding a comma or a brace.
+ *
+ * <p>As Hop builds a connection: an option stored for another database
type, left from before the
+ * type was changed, and an option without a value are not used, so they do
not count.
+ */
+ private static String optionValue(Map<?, ?> options, String name, Object
hopObject) {
+ String ownPrefix =
+ hopObject instanceof DatabaseMeta databaseMeta
+ ? EXTRA_OPTION_PREFIX + databaseMeta.getPluginId() + "."
+ : null;
+ for (Map.Entry<?, ?> entry : options.entrySet()) {
+ String key = String.valueOf(entry.getKey());
+ if (ownPrefix != null && key.startsWith(EXTRA_OPTION_PREFIX) &&
!key.startsWith(ownPrefix)) {
Review Comment:
This skips every `EXTRA_OPTION_<other type>.` key, but `DatabaseMeta` still
appends options stored for a parent type when it builds the URL
(`databaseForBothDbInterfacesIsTheSame`: same plugin id, or the connection's
type is a subclass). Greenplum, Redshift, CockroachDB and CrateDB extend
`PostgreSqlDatabaseMeta`, so a Greenplum connection with
`EXTRA_OPTION_POSTGRESQL.sslmode=require` gets `?sslmode=require` in its URL
while `HAS_OPTION` / `OPTION_MATCHES sslmode` reports it as missing. Could this
use the same test as `DatabaseMeta`?
##########
plugins/misc/lint/src/main/java/org/apache/hop/lint/CustomRuleExecutor.java:
##########
@@ -713,6 +743,84 @@ && hasDefaultGeneratedName(
}
}
+ /**
+ * The actions, other than Start, that lead on when they succeed but have no
hop followed when
+ * they fail: when one of them fails, the workflow stops there without
handling it.
+ *
+ * <p>As the engine runs a workflow: an unconditional hop is followed
whatever the result, so an
+ * action with one carries on. An action that does not evaluate its result,
such as Dummy, follows
+ * only its unconditional hops, so it has no failure to handle and is not
listed.
+ */
+ private static List<String> actionsWithoutFailurePath(WorkflowMeta workflow)
{
+ List<String> names = new ArrayList<>();
+ for (ActionMeta action : workflow.getActions()) {
+ if (action.isStart() || !action.isEvaluation()) {
+ continue;
+ }
+ List<WorkflowHopMeta> hops = outgoingHops(action, workflow);
+ if (!hops.isEmpty()
+ && hops.stream().noneMatch(hop -> hop.isUnconditional() ||
!hop.isEvaluation())) {
+ names.add(action.getName());
+ }
+ }
+ return names;
+ }
+
+ /**
+ * The JDBC URL as Hop builds it for the connection, with its variables left
unresolved: the
+ * manual URL when there is one, otherwise the type's URL with the extra
options appended. Null
+ * when the database type cannot build one.
+ */
+ private static String connectionUrl(DatabaseMeta dbMeta) {
+ // Left as written, variable resolvers included: building a URL must not
reach a vault.
+ Variables unresolved =
+ new Variables() {
+ @Override
+ public synchronized String resolve(String text) {
+ return text;
+ }
+ };
+ try {
+ return dbMeta.getURL(unresolved);
Review Comment:
When `getURL` throws, this returns null, and in `evaluateCondition` a null
value is "not violated" for `MATCHES_PATTERN`. A TLS rule like OPT-002 (`allOf`
attributes + url) then reports nothing for that connection, with only a
`logDetailed` line as a trace. That goes against the principle that a rule that
can't run must not look like one that found nothing. Could this report an error
for the rule/connection, or treat a null `url` as a violation for
`MATCHES_PATTERN`?
##########
plugins/misc/lint/src/main/java/org/apache/hop/lint/LintReportWriter.java:
##########
@@ -147,48 +163,57 @@ private static String renderSarif(
driver.put("version", toolVersion);
}
- // SARIF wants each rule declared once, then referenced by index from
every result. The rule is
+ // SARIF wants each rule declared once, then referenced by index from
every result. Every rule
+ // the run evaluated is declared, so a rule that found nothing reads
differently from one that
+ // was switched off or not installed.
+ Map<String, Integer> ruleIndex = new LinkedHashMap<>();
+ ArrayNode rules = driver.putArray("rules");
+ if (invocation != null) {
+ for (CustomLintRule rule : invocation.enabledRules()) {
+ String id = rule.generateRuleId();
+ if (!ruleIndex.containsKey(id)) {
+ ruleIndex.put(id, rules.size());
+ putRule(
+ rules.addObject(),
+ id,
+ Utils.isEmpty(rule.getName()) ? id : rule.getName(),
+ LintRuleDetails.of(rule),
+ rule.getSeverity());
+ }
+ }
+ }
+
+ // Then the ids only the findings carry, such as the error codes of Hop's
own checks. Each is
// described by the first finding of that id which carries the rule's
details, so a finding
// without them, such as a remark checked without the native rules, cannot
hide the tags.
Map<String, LintResult> describing = new LinkedHashMap<>();
for (LintResult result : results) {
+ if (ruleIndex.containsKey(result.getRuleId())) {
+ continue;
+ }
LintResult current = describing.get(result.getRuleId());
if (current == null
|| (LintRuleDetails.NONE.equals(current.getRuleDetails())
&& !LintRuleDetails.NONE.equals(result.getRuleDetails()))) {
describing.put(result.getRuleId(), result);
}
}
- Map<String, Integer> ruleIndex = new LinkedHashMap<>();
- ArrayNode rules = driver.putArray("rules");
describing.forEach(
(id, result) -> {
ruleIndex.put(id, rules.size());
- ObjectNode rule = rules.addObject();
- rule.put("id", id);
- // The id is the only stable per-rule name available: Hop's own
verify remarks all share
- // one rule id while carrying the transform name as their result
name, so using that
- // here would label the rule after whichever transform happened to
be reported first.
- rule.put("name", id);
LintRuleDetails details = result.getRuleDetails();
- rule.putObject("shortDescription").put("text", shortDescription(id,
result, details));
- if (!details.description().isBlank()) {
- rule.putObject("fullDescription").put("text",
details.description());
- }
- if (details.helpUri() != null && !details.helpUri().isBlank()) {
- rule.put("helpUri", details.helpUri());
- }
- rule.putObject("defaultConfiguration").put("level",
sarifLevel(result.getSeverity()));
- if (details.hasTags()) {
- ObjectNode properties = rule.putObject("properties");
- // GitHub code scanning and Azure DevOps read tags as a flat list
of strings, so a
- // key with several values becomes several key:value entries.
- ArrayNode flat = properties.putArray("tags");
- LintRuleDetails.flatTags(details.tags()).forEach(flat::add);
- putTags(properties.putObject("hopTags"), details);
- }
+ putRule(
+ rules.addObject(),
+ id,
+ shortDescription(id, result, details),
Review Comment:
Minor: a rule declared from the run's enabled rules gets `shortDescription`
= the rule's name/id and `defaultConfiguration.level` = the rule's severity,
but a rule declared only from its findings still uses `shortDescription(id,
result, details)` and the finding's severity. The same rule therefore renders
differently depending on whether an invocation was passed (CLI / Run Linter vs.
the plain `render(...)` overload), which shows up as noise when tools diff
reports. Could both paths build the descriptor the same way?
##########
plugins/misc/lint/src/main/java/org/apache/hop/lint/LintRun.java:
##########
@@ -278,12 +308,101 @@ public static Path baseDirectoryOf(File target) {
return directory != null ? directory.toPath().toAbsolutePath().normalize()
: null;
}
- /** The report in the given format, as {@code hop lint} writes it. */
+ /**
+ * The report of a run in the given format, as {@code hop lint} writes it:
the findings shown, and
+ * in SARIF how the run was set up.
+ */
+ public String renderReport(Outcome outcome, LintReportFormat format)
+ throws JsonProcessingException {
+ return LintReportWriter.render(
+ format, outcome.getShown(), toolVersion(), baseDirectory(),
invocation(outcome));
+ }
+
+ /**
+ * The report of a run's findings, without how the run was set up.
+ *
+ * @deprecated since 2.21; use {@link #renderReport(Outcome,
LintReportFormat)}, which also writes
+ * the run's rules and set-up to SARIF
+ */
+ @Deprecated(since = "2.21")
public String renderReport(List<LintResult> results, LintReportFormat format)
throws JsonProcessingException {
return LintReportWriter.render(format, results, toolVersion(),
baseDirectory());
}
+ /**
+ * A configuration or baseline file as a report records it: relative to the
lint target's folder,
+ * with {@code ../} where it lies outside it. A report is uploaded and kept,
so it must not carry
+ * the absolute path of a developer's home folder or a CI workspace. A local
{@code file:} URL, as
+ * the Run Linter action may be given, is read as the file it names; a
remote VFS location is kept
+ * as written.
+ *
+ * @param name the file as given, relative to the working directory or
absolute
+ * @param base the lint target's folder, or null
+ */
+ static String reportPath(String name, Path base) {
+ if (Utils.isEmpty(name)) {
+ return "";
+ }
+ Path path;
+ try {
+ if (name.regionMatches(true, 0, "file:", 0, 5)) {
+ path = Paths.get(URI.create(name));
+ } else if (name.matches("^[A-Za-z][A-Za-z0-9+.-]+://.*")) {
+ return name;
+ } else {
+ path = Paths.get(name);
+ }
+ path = path.toAbsolutePath().normalize();
+ } catch (Exception e) {
+ // Not a path we can place: the file name tells the reader which file
without the folders.
+ int slash = Math.max(name.lastIndexOf('/'), name.lastIndexOf('\\'));
+ return name.substring(slash + 1);
+ }
+ if (base == null || !Objects.equals(path.getRoot(), base.getRoot())) {
+ return path.getFileName() == null ? "" : path.getFileName().toString();
+ }
+ return base.relativize(path).toString().replace('\\', '/');
+ }
+
+ /**
+ * How this run was set up and how it ended: the rules it evaluated, the
packs they came from, and
+ * the settings that decided the findings and the outcome.
+ */
+ public LintInvocation invocation(Outcome outcome) {
+ Map<String, Object> configuration = new LinkedHashMap<>();
+ if (outcome.getConfigFile() != null) {
+ configuration.put(
+ "configFile", reportPath(outcome.getConfigFile().getAbsolutePath(),
baseDirectory()));
+ }
+ // No filter reports everything, as INFO does: the Run Linter action
defaults to INFO, hop lint
+ // to no filter, and the two runs are the same.
+ configuration.put(
+ "severity",
+ severityFilter == null ? LintSeverity.Level.INFO.name() :
severityFilter.name());
+ configuration.put("failOn", failOn == null ?
LintSeverity.FailOn.NONE.name() : failOn.name());
+ if (maxWarnings >= 0) {
+ configuration.put("maxWarnings", maxWarnings);
+ }
+ configuration.put("includeMetadata", includeMetadata);
+ if (baseline != null) {
+ configuration.put("baseline", reportPath(baselineName, baseDirectory()));
Review Comment:
`reportPath` returns any `scheme://...` location unchanged, and the Run
Linter action passes the baseline already resolved
(`run.setBaselineName(resolve(baselineFile))` in `ActionRunLinter`). With a
baseline such as `sftp://${SFTP_USER}:${SFTP_PASSWORD}@host/proj/baseline.json`
the real password lands in `runs[0].invocations[0].properties.baseline` of a
report that is usually uploaded and kept. Since the PR carefully keeps local
paths out of the report, could remote URLs have their userinfo (`user:pass@`)
stripped here too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]