hansva commented on code in PR #8984:
URL: https://github.com/apache/hop/pull/8984#discussion_r4314718532
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -292,32 +292,70 @@ public void check(
WorkflowMeta workflowMeta,
IVariables variables,
IHopMetadataProvider metadataProvider) {
- // boolean res =
- // ActionValidatorUtils.andValidator()
- // .validate(
- // this,
- // "fileItems",
- // remarks,
- //
AndValidator.putValidators(ActionValidatorUtils.notNullValidator()));
- //
- // if (!res) {
- // return;
- // }
+ // Folder names come from the previous action result at runtime.
+ if (argFromPrevious) {
+ return;
+ }
if (Utils.isEmpty(fileItems)) {
- String message = BaseMessages.getString(PKG,
"CloneRowMeta.CheckResult.NrClonesdMissing");
remarks.add(
- new CheckResult(ICheckResult.TYPE_RESULT_WARNING, "Any folders to
deletes", this));
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_WARNING,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.CheckResult.NoFolders"),
+ this));
+ return;
}
- /* TODO: If we enable action check
- ValidatorContext ctx = new ValidatorContext();
- AbstractFileValidator.putVariableSpace(ctx, getVariables());
- AndValidator.putValidators(
- ctx, ActionValidatorUtils.notNullValidator(),
ActionValidatorUtils.fileExistsValidator());
- for (FileItem item : fileItems) {
- ActionValidatorUtils.andValidator().validate(this, "fileName", remarks,
ctx);
- }*/
+ IVariables space = variables != null ? variables : getVariables();
+ for (int i = 0; i < fileItems.size(); i++) {
+ FileItem item = fileItems.get(i);
+ String folderName = resolveFolderName(item.getFileName(), space);
+ if (folderName.isBlank()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
+ "ActionDeleteFolders.CheckResult.FolderNameMissing",
+ String.valueOf(i + 1)),
+ this));
+ continue;
+ }
+
+ try (FileObject folder = HopVfs.getFileObject(folderName, space)) {
+ if (!folder.exists()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
Review Comment:
**A missing folder is now an ERROR, but at runtime it counts as a success**
`deleteFolder()` treats a folder that does not exist as already deleted and
returns `true`. So does every sibling action: `ActionDeleteFiles.check()` uses
`fileExistsValidator()` without `failIfDoesNotExist`, which adds no remark at
all for a missing path. This check makes it an ERROR.
The usual case is a temp/work folder that an earlier action in the same
workflow creates, or a path that only exists on the server. Design-time
variables also leave a runtime-only `${VAR}` unresolved, and that gets reported
too. These remarks now go through the linter's native checks
(`includeNativeChecks` is on by default) with `failOn=ERROR` as the default for
`hop lint`, the pre-commit lint and `ActionRunLinter`. A workflow that ran fine
before now fails CI and blocks commits.
A WARNING says the same thing to the user without breaking those gates:
```suggestion
ICheckResult.TYPE_RESULT_WARNING,
```
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -155,7 +155,7 @@ public Result execute(Result result, int nr) throws
HopException {
return result;
}
RowMetaAndData resultRow = rows.get(iteration);
- String argsPrevious = resultRow.getString(0, null);
+ String argsPrevious = resolveFolderName(resultRow.getString(0, null),
getVariables());
Review Comment:
**Folder names from the previous result are now variable-expanded before a
recursive delete**
Before this PR, `resultRow.getString(0, null)` was used literally, as every
other action does with result rows (DeleteFiles, CopyFiles, MoveFiles, Unzip,
Zip, …). `Variables.resolve()` now also expands `${...}`, `%%...%%` and
`$[hex]`, and runs `#{resolver:...}` variable resolvers on the row data.
Result rows are data and often come from outside, such as a file list, a
database query or Get File Names. A row value of `${PROJECT_HOME}` or
`${java.io.tmpdir}` now deletes that whole folder tree. A real folder whose
name contains `${x}` is now replaced by a different path. This was only needed
for the `Internal.Entry.Current.Directory` handling that the last commit
removed, and the PR description does not mention it.
```suggestion
String argsPrevious = resultRow.getString(0, null);
```
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -337,6 +375,17 @@ public List<ResourceReference> getResourceDependencies(
return references;
}
+ private String resolveFolderName(String rawName, IVariables space) {
+ if (rawName == null) {
+ return "";
+ }
+ IVariables source = space != null ? space : getVariables();
+ if (source == null) {
+ return rawName;
+ }
+ return source.resolve(rawName);
Review Comment:
**The PR description promises a `${Internal.Entry.Current.Directory}`
fallback that the final commit removed**
The description says the legacy name "is filled from
`Internal.Entry.Current.Folder` (or the workflow filename folder)… The same
resolution is used for execution and resource dependencies." Commit 35cefa67a8
deleted that code, so `resolveFolderName` is now a plain `resolve()`.
A workflow that still uses `${Internal.Entry.Current.Directory}/tmp`
resolves to a relative literal path at runtime. That path does not exist, so
the action logs "already deleted", returns success and deletes nothing. The
Kettle importer already maps this variable (`KettleConst`), so dropping the
special case is reasonable. Please update the description and test plan so
reviewers and release notes don't expect the fallback.
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -292,32 +292,70 @@ public void check(
WorkflowMeta workflowMeta,
IVariables variables,
IHopMetadataProvider metadataProvider) {
- // boolean res =
- // ActionValidatorUtils.andValidator()
- // .validate(
- // this,
- // "fileItems",
- // remarks,
- //
AndValidator.putValidators(ActionValidatorUtils.notNullValidator()));
- //
- // if (!res) {
- // return;
- // }
+ // Folder names come from the previous action result at runtime.
+ if (argFromPrevious) {
+ return;
+ }
if (Utils.isEmpty(fileItems)) {
- String message = BaseMessages.getString(PKG,
"CloneRowMeta.CheckResult.NrClonesdMissing");
remarks.add(
- new CheckResult(ICheckResult.TYPE_RESULT_WARNING, "Any folders to
deletes", this));
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_WARNING,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.CheckResult.NoFolders"),
+ this));
+ return;
}
- /* TODO: If we enable action check
- ValidatorContext ctx = new ValidatorContext();
- AbstractFileValidator.putVariableSpace(ctx, getVariables());
- AndValidator.putValidators(
- ctx, ActionValidatorUtils.notNullValidator(),
ActionValidatorUtils.fileExistsValidator());
- for (FileItem item : fileItems) {
- ActionValidatorUtils.andValidator().validate(this, "fileName", remarks,
ctx);
- }*/
+ IVariables space = variables != null ? variables : getVariables();
+ for (int i = 0; i < fileItems.size(); i++) {
+ FileItem item = fileItems.get(i);
+ String folderName = resolveFolderName(item.getFileName(), space);
+ if (folderName.isBlank()) {
Review Comment:
**A blank folder name is an ERROR here, but runtime handles it differently**
`execute()` uses `Utils.isEmpty(...)`. An empty resolved name only logs
"Empty line..ignoring" and is not counted in `nrErrors`, so check flags as
ERROR a row that never makes the action fail. A whitespace-only name passes
`isEmpty` at runtime and goes to `deleteFolder(" ")`, so runtime has no blank
guard at all. Either use the same predicate in both places (`isBlank` in
`execute()` too), or make the check severity match what execution does.
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -292,32 +292,70 @@ public void check(
WorkflowMeta workflowMeta,
IVariables variables,
IHopMetadataProvider metadataProvider) {
- // boolean res =
- // ActionValidatorUtils.andValidator()
- // .validate(
- // this,
- // "fileItems",
- // remarks,
- //
AndValidator.putValidators(ActionValidatorUtils.notNullValidator()));
- //
- // if (!res) {
- // return;
- // }
+ // Folder names come from the previous action result at runtime.
+ if (argFromPrevious) {
+ return;
+ }
if (Utils.isEmpty(fileItems)) {
- String message = BaseMessages.getString(PKG,
"CloneRowMeta.CheckResult.NrClonesdMissing");
remarks.add(
- new CheckResult(ICheckResult.TYPE_RESULT_WARNING, "Any folders to
deletes", this));
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_WARNING,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.CheckResult.NoFolders"),
+ this));
+ return;
}
- /* TODO: If we enable action check
- ValidatorContext ctx = new ValidatorContext();
- AbstractFileValidator.putVariableSpace(ctx, getVariables());
- AndValidator.putValidators(
- ctx, ActionValidatorUtils.notNullValidator(),
ActionValidatorUtils.fileExistsValidator());
- for (FileItem item : fileItems) {
- ActionValidatorUtils.andValidator().validate(this, "fileName", remarks,
ctx);
- }*/
+ IVariables space = variables != null ? variables : getVariables();
+ for (int i = 0; i < fileItems.size(); i++) {
+ FileItem item = fileItems.get(i);
+ String folderName = resolveFolderName(item.getFileName(), space);
+ if (folderName.isBlank()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
+ "ActionDeleteFolders.CheckResult.FolderNameMissing",
+ String.valueOf(i + 1)),
+ this));
+ continue;
+ }
+
+ try (FileObject folder = HopVfs.getFileObject(folderName, space)) {
+ if (!folder.exists()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
"ActionDeleteFolders.CheckResult.FolderDoesNotExist", folderName),
+ this));
+ } else if (folder.getType() != FileType.FOLDER) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.Error.NotFolder", folderName),
+ this));
+ } else {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_OK,
+ BaseMessages.getString(
+ PKG, "ActionDeleteFolders.CheckResult.FolderOk",
folderName),
+ this));
+ }
+ } catch (Exception e) {
Review Comment:
**The catch also covers `close()` from try-with-resources, so one folder can
get two remarks that contradict each other**
The OK/ERROR remark is added inside the try. If `folder.close()` then throws
a `FileSystemException` (possible on SFTP/S3 providers), the catch adds a
second ERROR, "Could not validate folder", for the same row. `e.getMessage()`
can also be null, which shows as "exception: null". One way to fix this is to
compute the result type and message inside the try and add a single remark
after it.
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -292,32 +292,70 @@ public void check(
WorkflowMeta workflowMeta,
IVariables variables,
IHopMetadataProvider metadataProvider) {
- // boolean res =
- // ActionValidatorUtils.andValidator()
- // .validate(
- // this,
- // "fileItems",
- // remarks,
- //
AndValidator.putValidators(ActionValidatorUtils.notNullValidator()));
- //
- // if (!res) {
- // return;
- // }
+ // Folder names come from the previous action result at runtime.
+ if (argFromPrevious) {
+ return;
+ }
if (Utils.isEmpty(fileItems)) {
- String message = BaseMessages.getString(PKG,
"CloneRowMeta.CheckResult.NrClonesdMissing");
remarks.add(
- new CheckResult(ICheckResult.TYPE_RESULT_WARNING, "Any folders to
deletes", this));
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_WARNING,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.CheckResult.NoFolders"),
+ this));
+ return;
}
- /* TODO: If we enable action check
- ValidatorContext ctx = new ValidatorContext();
- AbstractFileValidator.putVariableSpace(ctx, getVariables());
- AndValidator.putValidators(
- ctx, ActionValidatorUtils.notNullValidator(),
ActionValidatorUtils.fileExistsValidator());
- for (FileItem item : fileItems) {
- ActionValidatorUtils.andValidator().validate(this, "fileName", remarks,
ctx);
- }*/
+ IVariables space = variables != null ? variables : getVariables();
+ for (int i = 0; i < fileItems.size(); i++) {
+ FileItem item = fileItems.get(i);
+ String folderName = resolveFolderName(item.getFileName(), space);
+ if (folderName.isBlank()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
+ "ActionDeleteFolders.CheckResult.FolderNameMissing",
+ String.valueOf(i + 1)),
+ this));
+ continue;
+ }
+
+ try (FileObject folder = HopVfs.getFileObject(folderName, space)) {
+ if (!folder.exists()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
"ActionDeleteFolders.CheckResult.FolderDoesNotExist", folderName),
+ this));
+ } else if (folder.getType() != FileType.FOLDER) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.Error.NotFolder", folderName),
Review Comment:
**The check text reuses the runtime log key
`ActionDeleteFolders.Error.NotFolder`**
That message reads `Error! {0} is not a folder.`. Inside a remark that
already has ERROR type, the problem row then reads "Error! … Error! /path is
not a folder." The other check messages got their own `CheckResult.*` keys. A
`CheckResult.NotAFolder` key would keep the wording consistent and leave the
runtime log line alone.
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -292,32 +292,70 @@ public void check(
WorkflowMeta workflowMeta,
IVariables variables,
IHopMetadataProvider metadataProvider) {
- // boolean res =
- // ActionValidatorUtils.andValidator()
- // .validate(
- // this,
- // "fileItems",
- // remarks,
- //
AndValidator.putValidators(ActionValidatorUtils.notNullValidator()));
- //
- // if (!res) {
- // return;
- // }
+ // Folder names come from the previous action result at runtime.
+ if (argFromPrevious) {
+ return;
+ }
if (Utils.isEmpty(fileItems)) {
- String message = BaseMessages.getString(PKG,
"CloneRowMeta.CheckResult.NrClonesdMissing");
remarks.add(
- new CheckResult(ICheckResult.TYPE_RESULT_WARNING, "Any folders to
deletes", this));
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_WARNING,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.CheckResult.NoFolders"),
+ this));
+ return;
}
- /* TODO: If we enable action check
- ValidatorContext ctx = new ValidatorContext();
- AbstractFileValidator.putVariableSpace(ctx, getVariables());
- AndValidator.putValidators(
- ctx, ActionValidatorUtils.notNullValidator(),
ActionValidatorUtils.fileExistsValidator());
- for (FileItem item : fileItems) {
- ActionValidatorUtils.andValidator().validate(this, "fileName", remarks,
ctx);
- }*/
+ IVariables space = variables != null ? variables : getVariables();
Review Comment:
**This reimplements the shared action validators**
The sibling `ActionDeleteFiles.check()` validates the same `fileItems` shape
with `ActionValidatorUtils.andValidator()` plus
`notNullValidator()`/`notBlankValidator()` and `fileExistsValidator()`. It uses
the property path `"fileItems[" + i + "].fileName"` and
`AbstractFileValidator.putVariableSpace(ctx, variables)`. Here about 50 lines
hand-roll the blank, exists and exception remarks, with a different severity
policy (see the comment on L329). Using the validators keeps the two delete
actions consistent. Only the "is a folder" test would remain custom.
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -337,6 +375,17 @@ public List<ResourceReference> getResourceDependencies(
return references;
}
+ private String resolveFolderName(String rawName, IVariables space) {
+ if (rawName == null) {
+ return "";
+ }
+ IVariables source = space != null ? space : getVariables();
Review Comment:
**`resolveFolderName` is left over from the removed legacy logic**
After the last commit it is a null-safe `resolve()`:
- `source == null` cannot happen: `ActionBase.variables` is initialised with
`new Variables()` and `getVariables()` never returns null.
- The `space != null ? space : getVariables()` fallback is repeated at L309.
- In `execute()`, `resolveFolderName(item.getFileName(), getVariables())`
(L181) is the same as the original `resolve(item.getFileName())`. It is churn
in the delete path for no change in behaviour.
`Const.NVL(space.resolve(rawName), "")` at the two call sites that need the
variables argument (check, resource dependencies) does the same job.
`execute()` can stay on `resolve(...)`.
##########
plugins/actions/deletefolders/src/main/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFolders.java:
##########
@@ -292,32 +292,70 @@ public void check(
WorkflowMeta workflowMeta,
IVariables variables,
IHopMetadataProvider metadataProvider) {
- // boolean res =
- // ActionValidatorUtils.andValidator()
- // .validate(
- // this,
- // "fileItems",
- // remarks,
- //
AndValidator.putValidators(ActionValidatorUtils.notNullValidator()));
- //
- // if (!res) {
- // return;
- // }
+ // Folder names come from the previous action result at runtime.
+ if (argFromPrevious) {
+ return;
+ }
if (Utils.isEmpty(fileItems)) {
- String message = BaseMessages.getString(PKG,
"CloneRowMeta.CheckResult.NrClonesdMissing");
remarks.add(
- new CheckResult(ICheckResult.TYPE_RESULT_WARNING, "Any folders to
deletes", this));
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_WARNING,
+ BaseMessages.getString(PKG,
"ActionDeleteFolders.CheckResult.NoFolders"),
+ this));
+ return;
}
- /* TODO: If we enable action check
- ValidatorContext ctx = new ValidatorContext();
- AbstractFileValidator.putVariableSpace(ctx, getVariables());
- AndValidator.putValidators(
- ctx, ActionValidatorUtils.notNullValidator(),
ActionValidatorUtils.fileExistsValidator());
- for (FileItem item : fileItems) {
- ActionValidatorUtils.andValidator().validate(this, "fileName", remarks,
ctx);
- }*/
+ IVariables space = variables != null ? variables : getVariables();
+ for (int i = 0; i < fileItems.size(); i++) {
+ FileItem item = fileItems.get(i);
+ String folderName = resolveFolderName(item.getFileName(), space);
+ if (folderName.isBlank()) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_ERROR,
+ BaseMessages.getString(
+ PKG,
+ "ActionDeleteFolders.CheckResult.FolderNameMissing",
+ String.valueOf(i + 1)),
+ this));
+ continue;
+ }
+
+ try (FileObject folder = HopVfs.getFileObject(folderName, space)) {
Review Comment:
**The check now makes VFS calls for every row each time it runs**
The lint plugin's `BackgroundLintService` runs native checks
(`checkActions`) in the background as the workflow is edited. Each Delete
folders row now costs `getFileObject` + `exists()` + `getType()`. For
S3/GCS/Azure/SFTP paths that means a network round trip, or a new SFTP session,
per row on every lint pass, and a slow or unreachable endpoint stalls the lint.
Consider skipping existence probes for remote schemes, or caching per check run.
##########
plugins/actions/deletefolders/src/test/java/org/apache/hop/workflow/actions/deletefolders/ActionDeleteFoldersCheckTest.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.workflow.actions.deletefolders;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hop.core.HopClientEnvironment;
+import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+class ActionDeleteFoldersCheckTest {
Review Comment:
**The tests only cover `check()`, not the other two changes in this PR**
- `execute()` now variable-resolves names from previous result rows (L158),
which changes runtime deletion behaviour. No test covers it.
- `getResourceDependencies` now resolves with the `variables` argument
instead of the action's own variables (L368). No test covers it.
- The `CouldNotValidate` exception path is not exercised. An unknown VFS
scheme such as `nosuchscheme://x` would hit it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]