hansva commented on code in PR #9360:
URL: https://github.com/apache/hop/pull/9360#discussion_r4537208463
##########
plugins/tech/neo4j/src/main/java/org/apache/hop/neo4j/actions/propertygraph/ActionCreatePropertyGraph.java:
##########
@@ -225,8 +224,67 @@ public Result execute(Result result, int nr) throws
HopException {
return result;
}
+ /**
+ * Run the statements one by one. When one fails, drop the tables this run
created, newest first,
+ * so a failing CREATE PROPERTY GRAPH doesn't leave them behind. Each
statement commits on its
+ * own, so a rollback can't undo them.
+ */
+ private void executeStatements(Database database, List<PlannedStatement>
statements)
Review Comment:
The cleanup here (drop in reverse order, keep tables that already existed,
report the ones left behind) only gets exercised by the AGE integration test. A
small unit test with a mocked `Database` would catch regressions in the drop
order, the 'stay in the database' message, or an ALTER TABLE accidentally
marked as a created table, in the normal Maven build.
##########
plugins/tech/neo4j/src/test/java/org/apache/hop/neo4j/actions/GraphActionFailuresTest.java:
##########
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.neo4j.actions;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+import java.util.List;
+import org.apache.hop.core.HopClientEnvironment;
+import org.apache.hop.core.Result;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.graph.BaseGraphDatabase;
+import org.apache.hop.core.graph.CypherGraphDialect;
+import org.apache.hop.core.graph.GraphConstraintType;
+import org.apache.hop.core.graph.GraphDatabaseMeta;
+import org.apache.hop.core.graph.IGraphConnection;
+import org.apache.hop.core.graph.IGraphDialect;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
+import org.apache.hop.neo4j.actions.constraint.ConstraintUpdate;
+import org.apache.hop.neo4j.actions.constraint.Neo4jConstraint;
+import org.apache.hop.neo4j.actions.cypherscript.CypherScript;
+import org.apache.hop.neo4j.actions.index.IndexUpdate;
+import org.apache.hop.neo4j.actions.index.Neo4jIndex;
+import org.apache.hop.neo4j.actions.index.ObjectType;
+import org.apache.hop.neo4j.actions.index.UpdateType;
+import org.apache.hop.workflow.action.ActionBase;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Graph index, Graph constraint and Graph script are false with 1 error when
they fail, so a
+ * failure hop can handle it, instead of stopping the workflow (#9219).
+ */
+class GraphActionFailuresTest {
+
+ private static final String REFUSING = "refusing";
+
+ @BeforeAll
+ static void beforeAll() throws HopException {
+ HopClientEnvironment.init();
+ }
+
+ @Test
+ void indexFailures() throws Exception {
+ assertFalseWithOneError(index("does-not-exist", rangeIndex()));
+ assertFalseWithOneError(index("", rangeIndex()));
+ assertFalseWithOneError(index(REFUSING, rangeIndex()));
+
+ IndexUpdate noType = rangeIndex();
+ noType.setType(null);
+ assertFalseWithOneError(index(REFUSING, noType));
+
+ IndexUpdate noProperties = rangeIndex();
+ noProperties.setObjectProperties("");
+ assertFalseWithOneError(index(REFUSING, noProperties));
Review Comment:
These 'no properties' cases (here and at line 87) run against the refusing
connection, which already fails on connect, so they pass whether or not the
empty-property validation works. Using a connection that connects, or asserting
on the logged error, would make them actually test the validation.
##########
plugins/tech/neo4j/src/main/java/org/apache/hop/neo4j/actions/propertygraph/ActionCreatePropertyGraph.java:
##########
@@ -225,8 +224,67 @@ public Result execute(Result result, int nr) throws
HopException {
return result;
}
+ /**
+ * Run the statements one by one. When one fails, drop the tables this run
created, newest first,
+ * so a failing CREATE PROPERTY GRAPH doesn't leave them behind. Each
statement commits on its
+ * own, so a rollback can't undo them.
+ */
+ private void executeStatements(Database database, List<PlannedStatement>
statements)
+ throws HopException {
+ List<String> createdTables = new ArrayList<>();
+ for (PlannedStatement statement : statements) {
+ if (isDetailed()) {
+ logDetailed("Executing: " + statement.sql());
+ }
+ try {
+ database.execStatement(statement.sql());
+ } catch (HopException e) {
+ String dropError = dropCreatedTables(database, createdTables);
+ if (dropError != null) {
+ throw new HopException(
+ "The property graph can't be created and the tables it created
can't be dropped: "
+ + dropError,
+ e);
+ }
+ throw e;
+ }
+ if (statement.createdTable() != null) {
+ createdTables.add(statement.createdTable());
Review Comment:
A table counts as 'created by this run' because `checkTableExists` was false
when the statements were planned, and as far as I can see the generator doesn't
de-duplicate table names. If two node/edge mappings resolve to the same table
(or another process creates it in between), the first CREATE TABLE succeeds,
the second fails with 'already exists', and the cleanup then drops the first
one, so the real cause gets hidden behind the drop. Maybe refuse duplicate
tables in the plan, or skip a CREATE for a table already planned?
##########
plugins/tech/neo4j/src/main/java/org/apache/hop/neo4j/actions/constraint/Neo4jConstraint.java:
##########
@@ -73,40 +73,37 @@ public Result execute(Result result, int nr) throws
HopException {
// Success unless something goes wrong, whatever the result of the
previous action
result.setResult(true);
- connection =
- NeoConnectionUtils.findGraphConnection(getMetadataProvider(),
resolve(connectionName));
-
- if (connection == null) {
- result.setResult(false);
- result.increaseErrors(1L);
- throw new HopException("Please specify a Neo4j connection to use");
- }
-
- // Loop over the constraint updates to see which need deleting...
- //
- for (ConstraintUpdate constraintUpdate : constraintUpdates) {
- if (constraintUpdate.getUpdateType() == null) {
- throw new HopException("Please make sure to always specify a
constraint update type");
+ try {
+ connection =
+ NeoConnectionUtils.getGraphConnection(getMetadataProvider(),
resolve(connectionName));
+
+ for (ConstraintUpdate constraintUpdate : constraintUpdates) {
+ if (constraintUpdate.getUpdateType() == null) {
+ throw new HopException("Please make sure to always specify a
constraint update type");
+ }
}
- switch (constraintUpdate.getUpdateType()) {
- case DROP:
+
+ // Drop the constraints first...
+ //
+ for (ConstraintUpdate constraintUpdate : constraintUpdates) {
+ if (constraintUpdate.getUpdateType() == UpdateType.DROP) {
dropConstraint(constraintUpdate);
- break;
- default:
- break;
+ }
}
- }
- // Create the constraints if needed
- //
- for (ConstraintUpdate constraintUpdate : constraintUpdates) {
- switch (constraintUpdate.getUpdateType()) {
- case CREATE:
+ // ... then create the constraints
+ //
+ for (ConstraintUpdate constraintUpdate : constraintUpdates) {
+ if (constraintUpdate.getUpdateType() == UpdateType.CREATE) {
createConstraint(constraintUpdate);
- break;
- default:
- break;
+ }
}
+ } catch (Exception e) {
+ // The updates before the failing one stay done, the ones after it are
not run
+ //
+ logError("Error updating the graph database constraints", e);
+ result.setResult(false);
+ result.setNrErrors(1);
Review Comment:
The removed code used `result.increaseErrors(1L)`; `setNrErrors(1)`
overwrites any error count already on the incoming Result, so N earlier errors
become 1 instead of N+1. Same in `Neo4jIndex` and `CypherScript`.
`increaseErrors(1L)` would keep the totals right.
##########
plugins/tech/neo4j/src/main/java/org/apache/hop/neo4j/actions/constraint/Neo4jConstraint.java:
##########
@@ -73,40 +73,37 @@ public Result execute(Result result, int nr) throws
HopException {
// Success unless something goes wrong, whatever the result of the
previous action
result.setResult(true);
- connection =
- NeoConnectionUtils.findGraphConnection(getMetadataProvider(),
resolve(connectionName));
-
- if (connection == null) {
- result.setResult(false);
- result.increaseErrors(1L);
- throw new HopException("Please specify a Neo4j connection to use");
- }
-
- // Loop over the constraint updates to see which need deleting...
- //
- for (ConstraintUpdate constraintUpdate : constraintUpdates) {
- if (constraintUpdate.getUpdateType() == null) {
- throw new HopException("Please make sure to always specify a
constraint update type");
+ try {
Review Comment:
Small one: this `execute()` and the one in `Neo4jIndex` are now almost
identical (connection lookup, type check, drops, creates, catch/log/false/1
error). A shared helper would keep future changes to the failure handling in
one place.
##########
plugins/tech/neo4j/src/main/java/org/apache/hop/neo4j/actions/index/Neo4jIndex.java:
##########
@@ -75,40 +75,37 @@ public Result execute(Result result, int nr) throws
HopException {
// Success unless something goes wrong, whatever the result of the
previous action
result.setResult(true);
- connection =
- NeoConnectionUtils.findGraphConnection(getMetadataProvider(),
resolve(connectionName));
-
- if (connection == null) {
- result.setResult(false);
- result.increaseErrors(1L);
- throw new HopException("Please specify a Neo4j connection to use");
- }
+ try {
+ connection =
+ NeoConnectionUtils.getGraphConnection(getMetadataProvider(),
resolve(connectionName));
- // Loop over the index updates to see which need deleting...
- //
- for (IndexUpdate indexUpdate : indexUpdates) {
- if (indexUpdate.getType() == null) {
- throw new HopException("Please make sure to always specify an index
update type");
+ for (IndexUpdate indexUpdate : indexUpdates) {
+ if (indexUpdate.getType() == null) {
+ throw new HopException("Please make sure to always specify an index
update type");
+ }
}
- switch (indexUpdate.getType()) {
- case DROP:
+
+ // Drop the indexes first...
+ //
+ for (IndexUpdate indexUpdate : indexUpdates) {
+ if (indexUpdate.getType() == UpdateType.DROP) {
dropIndex(indexUpdate);
- break;
- default:
- break;
+ }
}
- }
- // Create the indexes if needed
- //
- for (IndexUpdate indexUpdate : indexUpdates) {
- switch (indexUpdate.getType()) {
- case CREATE:
+ // ... then create the indexes
+ //
+ for (IndexUpdate indexUpdate : indexUpdates) {
+ if (indexUpdate.getType() == UpdateType.CREATE) {
createIndex(indexUpdate);
- break;
- default:
- break;
+ }
}
+ } catch (Exception e) {
Review Comment:
More of a question than a request: four actions now each catch everything to
return false instead of throwing, because an exception from `IAction.execute`
aborts the workflow without following the failure hop. Other actions that throw
still have that behaviour. Would it make sense to handle this once in the
engine (where `Workflow` calls `execute()`), so every action follows its
failure hop on an exception? Fine to keep that for a separate issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]