bamaer opened a new pull request, #9479:
URL: https://github.com/apache/hop/pull/9479

   Item 3 of #9324: row counts and write operations.
   
   - Engine: a transform's end event carries the row counters of its copy (`I`, 
`O`, `R`, `W`, `U`, rejected); a pipeline's end carries the sums over its 
transforms and copies. Local engine only.
   - OpenLineage: the standard `inputStatistics` / `outputStatistics` facets 
with `rowCount`, only when the count belongs to that dataset (the transforms 
that touched it touched nothing else). A `hopRowCounts` run facet with the 
counters, and on a pipeline's end a `hopWrites` run facet with every dataset 
the run wrote, once.
   - DataHub: an `operation` per dataset when a pipeline completes (`INSERT`, 
`UPDATE`, `DELETE`, ...), with the rows of all transforms and copies. A failed 
pipeline reports none, so a half-written table doesn't look fresh. The run's 
properties get its row counts. Check data catalog's freshness check now works 
for datasets only Hop writes.
   - No count rather than a wrong one: transforms that write several datasets, 
deletes (not rows written), Execute SQL scripts of several statements, and 
transforms that don't keep the counter.
   - The counts are Hop's own counters, so Text File Output/Input count a 
header line in `O`/`I` (documented).
   - Docs: OpenLineage, DataHub, Data Catalog.
   - Tests: unit tests in the engine, OpenLineage and DataHub plugins. Checked 
by hand against DataHub 1.7: the operation and properties are accepted, and a 
freshness check passes after a run.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to