[
https://issues.apache.org/jira/browse/GOBBLIN-2167?focusedWorklogId=939354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-939354
]
ASF GitHub Bot logged work on GOBBLIN-2167:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 22/Oct/24 03:03
Start Date: 22/Oct/24 03:03
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #4069:
URL: https://github.com/apache/gobblin/pull/4069#discussion_r1809781895
##########
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/hive/HiveDatasetFinder.java:
##########
@@ -294,6 +304,12 @@ protected HiveDataset computeNext() {
};
}
+ protected static boolean shouldAllowTableLocation(Optional<String> regex,
Table table) {
+ if (!regex.isPresent()) {
+ return true;
+ }
+ return
Pattern.compile(regex.get()).matcher(table.getSd().getLocation()).matches();
Review Comment:
I compiled it as part of the class field instead, good callout
Issue Time Tracking
-------------------
Worklog Id: (was: 939354)
Time Spent: 40m (was: 0.5h)
> Add a configuration in Hive dataset finder that filters hive tables to
> copy/retention based off underlying table location
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-2167
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2167
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: William Lo
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> With hive tables sometimes it becomes beneficial for retention and other jobs
> (copy) to be able to only select tables within a database that fall under a
> certain underlying HDFS path. This becomes useful when different HDFS paths
> have differing permissions and thus becomes tricky to group together in a
> single Gobblin job.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
