hansva opened a new issue, #9597:
URL: https://github.com/apache/hop/issues/9597

   ### What would you like to happen?
   
   Some transforms do setup work for the whole run, such as truncating a table, 
creating it, or adding columns. Today they do it on the first row that copy 0 
processes. That causes trouble whenever more than one copy runs the transform:
   
   - **Local engine, several copies:** copy 1 can insert rows before copy 0 has 
truncated or changed the table. Those rows are then truncated away, or written 
into a table whose structure is still being changed.
   - **Native Spark and Beam engines:** every partition (or bundle) runs as 
copy 0, so every partition does the setup:
     - partitions truncate each other's rows;
     - two partitions create the same table;
     - adding or dropping columns deadlocks.
   
   Transforms affected:
   - Table output: "Truncate table" and "Automatically update table structure".
   - PGVector upsert: creating the table, the HNSW index and the mapped columns.
   - Neo4j Output and Graph Output: creating indexes and constraints. 
Partitions deadlock on the same statements.
   
   ### Proposal
   
   Give transforms a hook for work that runs once per run, before any copy, 
partition or bundle reads a row:
   
   - `ITransformMeta.needsPrepareOnce()`: the transform says it has such work.
   - `ITransform.prepareOnce()`: does the work. The input row metadata is set 
to what the previous transforms declare.
   - `TransformPrepareOnce` runs it:
     - **Local engine:** once per transform, on copy 0, after init and before 
the threads start. A transactional pipeline (one that uses a connection group) 
leaves it to the first row, so the work stays inside the transaction.
     - **Native Spark and Beam:** on the driver, before the job, for the 
transforms an enabled hop connects. The names of the prepared transforms go to 
the partitions or bundles in `Internal.Pipeline.PreparedOnceTransforms`, so the 
partitions don't do the work again.
   - A transform flagged as prepared (`isPreparedOnce()`) skips the work. A 
transform that doesn't use the hook behaves exactly as before.
   
   Transforms that use the hook:
   
   | Transform | Work done once per run |
   |---|---|
   | Table output | "Truncate table", "Automatically update table structure". 
"Truncate on first row" still waits for the first row. |
   | PGVector upsert | Creates the table, the HNSW index and the mapped 
columns, with autocommit on as before. |
   | Graph Output | Creates the indexes of the model's nodes. |
   | Neo4j Output | Creates the indexes of its fixed labels. Labels that come 
from a field are only known on the first row, so those indexes are still 
created there. |
   
   Neo4j index and constraint statements also run in a write transaction where 
the database allows schema changes in one. The driver then retries a statement 
that deadlocks with the same statement from another copy or partition.
   
   ### Tests
   
   - Unit tests for `TransformPrepareOnce` on the local engine and on native 
Spark.
   - Unit tests for Table output, PGVector upsert, Neo4j Output and Graph 
Output.
   - On native Spark, these database integration tests now pass: 0006, 0020 and 
0033–0036. The two Neo4j tests that deadlocked pass too.
   - The docs for Table output, PGVector upsert, Neo4j Output and Graph Output 
describe when the setup work happens.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: API


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to