bamaer opened a new pull request, #9479: URL: https://github.com/apache/hop/pull/9479
Item 3 of #9324: row counts and write operations. - Engine: a transform's end event carries the row counters of its copy (`I`, `O`, `R`, `W`, `U`, rejected); a pipeline's end carries the sums over its transforms and copies. Local engine only. - OpenLineage: the standard `inputStatistics` / `outputStatistics` facets with `rowCount`, only when the count belongs to that dataset (the transforms that touched it touched nothing else). A `hopRowCounts` run facet with the counters, and on a pipeline's end a `hopWrites` run facet with every dataset the run wrote, once. - DataHub: an `operation` per dataset when a pipeline completes (`INSERT`, `UPDATE`, `DELETE`, ...), with the rows of all transforms and copies. A failed pipeline reports none, so a half-written table doesn't look fresh. The run's properties get its row counts. Check data catalog's freshness check now works for datasets only Hop writes. - No count rather than a wrong one: transforms that write several datasets, deletes (not rows written), Execute SQL scripts of several statements, and transforms that don't keep the counter. - The counts are Hop's own counters, so Text File Output/Input count a header line in `O`/`I` (documented). - Docs: OpenLineage, DataHub, Data Catalog. - Tests: unit tests in the engine, OpenLineage and DataHub plugins. Checked by hand against DataHub 1.7: the operation and properties are accepted, and a freshness check passes after a run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
