Blazer-007 commented on code in PR #4154:
URL: https://github.com/apache/gobblin/pull/4154#discussion_r2513703205
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/iceberg/IcebergFileStreamExtractor.java:
##########
@@ -40,8 +57,22 @@
@Slf4j
public class IcebergFileStreamExtractor extends FileBasedExtractor<String,
FileAwareInputStream> {
+ private final Map<String, String> fileToPartitionPathMap;
+ private final Gson gson = new Gson();
+
public IcebergFileStreamExtractor(WorkUnitState workUnitState) throws
IOException {
super(workUnitState, new IcebergFileStreamHelper(workUnitState));
+
+ // Load partition path mapping from work unit (set by IcebergSource)
+ String partitionPathJson =
workUnitState.getProp(IcebergSource.ICEBERG_FILE_PARTITION_PATH);
+ if (!StringUtils.isBlank(partitionPathJson)) {
+ this.fileToPartitionPathMap = gson.fromJson(partitionPathJson,
+ new TypeToken<Map<String, String>>() {}.getType());
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
