wombatu-kun commented on code in PR #20224:
URL: https://github.com/apache/hudi/pull/20224#discussion_r4375464993
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkStreamingMetadataWriteHandler.java:
##########
@@ -39,6 +39,8 @@ public class SparkStreamingMetadataWriteHandler extends
StreamingMetadataWriteHa
@Override
public HoodieData<WriteStatus> streamWriteToMetadataTable(HoodieTable table,
HoodieData<WriteStatus> dataTableWriteStatuses, String instantTime,
int
coalesceDivisorForDataTableWrites) {
+ table.getContext().setJobStatus(this.getClass().getSimpleName(),
+ "Streaming write to metadata table: " +
table.getConfig().getTableName());
Review Comment:
On master, `HudiPhaseResolver` has no rule for any of the five new
descriptions, so a job carrying one resolves to `UNKNOWN`; in the default
config this label would move the whole data-table write from `DATA_TABLE_WRITE`
(via `Committing stats:`) to `UNKNOWN`. Could the resolver rules, and the
analyzer docs and caveat text that call these paths untagged, be updated in
this PR for whichever descriptions you keep?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestSparkStreamingMetadataWriteHandler.java:
##########
@@ -56,6 +57,9 @@ void setUp() {
when(mockHoodieTable.getMetaClient()).thenReturn(metaClient);
HoodieEngineContext engineContext = mock(HoodieEngineContext.class);
when(mockHoodieTable.getContext()).thenReturn(engineContext);
+ HoodieWriteConfig writeConfig = mock(HoodieWriteConfig.class);
Review Comment:
Nothing tests which stage ends up carrying each new description; the only
test change stubs `getConfig()` so the new call does not NPE. Could a test
register a `SparkListener` that reads `spark.job.description` from
`SparkListenerStageSubmitted.properties()` during a small upsert, and assert
each label is on the stage it names?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
