wombatu-kun commented on code in PR #20224:
URL: https://github.com/apache/hudi/pull/20224#discussion_r4375467526
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkStreamingMetadataWriteHandler.java:
##########
@@ -39,6 +39,8 @@ public class SparkStreamingMetadataWriteHandler extends
StreamingMetadataWriteHa
@Override
public HoodieData<WriteStatus> streamWriteToMetadataTable(HoodieTable table,
HoodieData<WriteStatus> dataTableWriteStatuses, String instantTime,
int
coalesceDivisorForDataTableWrites) {
+ table.getContext().setJobStatus(this.getClass().getSimpleName(),
Review Comment:
`SparkRDDTableServiceClient.partialUpdateTableMetadata` calls this same
handler, so in the default config the job that runs a compaction, log
compaction or clustering carries this label too. Setting the description right
before `SlimWriteStats.from(...)` in `SparkRDDWriteClient.commit` and in
`SparkRDDTableServiceClient.triggerWritesAndFetchWriteStats` would name the job
that actually runs, for each caller separately.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseWriteHelper.java:
##########
@@ -64,6 +64,9 @@ public HoodieWriteMetadata<O> write(String instantTime,
try {
HoodieTimer sourceReadAndIndexTimer = HoodieTimer.start();
// De-dupe/merge if needed
+ if (shouldCombine) {
+ context.setJobStatus(this.getClass().getSimpleName(), "Deduplicating
records: " + table.getConfig().getTableName());
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
