[
https://issues.apache.org/jira/browse/GOBBLIN-2092?focusedWorklogId=924091&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-924091
]
ASF GitHub Bot logged work on GOBBLIN-2092:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jun/24 09:01
Start Date: 24/Jun/24 09:01
Worklog Time Spent: 10m
Work Description: pratapaditya04 commented on code in PR #3978:
URL: https://github.com/apache/gobblin/pull/3978#discussion_r1650632758
##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStoreTest.java:
##########
@@ -215,6 +246,60 @@ public void testGetSpec() throws Exception {
Assert.assertEquals(specs.size(), 1);
Assert.assertTrue(specs.contains(this.flowSpec4));
}
+ @Test
+ public void testGetSpecsAfterUpdate() throws Exception {
+ this.specStore.addSpec(this.flowSpec5);
+ FlowSpec result = (FlowSpec) this.specStore.getSpec(this.uri5);
+ removeModificationTimestampFromSpecs(result);
+ Assert.assertEquals(result, this.flowSpec5);
+ // Updating flowSpec5
+ this.specStore.addSpec(this.flowSpec5Updated);
+
+ FlowSpecSearchObject flowSpecSearchObject =
FlowSpecSearchObject.builder().flowGroup("fg5").build();
+ Collection<Spec> specs = this.specStore.getSpecs(flowSpecSearchObject);
+ Assert.assertTrue(specs.contains(this.flowSpec5Updated));
+
+ flowSpecSearchObject =
FlowSpecSearchObject.builder().flowName("fn5").build();
+ specs = this.specStore.getSpecs(flowSpecSearchObject);
+ Assert.assertTrue(specs.contains(this.flowSpec5Updated));
+
+ flowSpecSearchObject =
FlowSpecSearchObject.builder().flowName("fn5").flowGroup("fg5").build();
+ specs = this.specStore.getSpecs(flowSpecSearchObject);
+ Assert.assertEquals(specs.size(), 1);
+ Assert.assertTrue(specs.contains(this.flowSpec5Updated));
Review Comment:
addressed
##########
gobblin-runtime/src/test/java/org/apache/gobblin/runtime/spec_store/MysqlSpecStoreTest.java:
##########
@@ -215,6 +246,60 @@ public void testGetSpec() throws Exception {
Assert.assertEquals(specs.size(), 1);
Assert.assertTrue(specs.contains(this.flowSpec4));
}
+ @Test
+ public void testGetSpecsAfterUpdate() throws Exception {
+ this.specStore.addSpec(this.flowSpec5);
+ FlowSpec result = (FlowSpec) this.specStore.getSpec(this.uri5);
+ removeModificationTimestampFromSpecs(result);
+ Assert.assertEquals(result, this.flowSpec5);
+ // Updating flowSpec5
+ this.specStore.addSpec(this.flowSpec5Updated);
+
+ FlowSpecSearchObject flowSpecSearchObject =
FlowSpecSearchObject.builder().flowGroup("fg5").build();
+ Collection<Spec> specs = this.specStore.getSpecs(flowSpecSearchObject);
+ Assert.assertTrue(specs.contains(this.flowSpec5Updated));
+
+ flowSpecSearchObject =
FlowSpecSearchObject.builder().flowName("fn5").build();
+ specs = this.specStore.getSpecs(flowSpecSearchObject);
+ Assert.assertTrue(specs.contains(this.flowSpec5Updated));
Review Comment:
addressed
Issue Time Tracking
-------------------
Worklog Id: (was: 924091)
Time Spent: 0.5h (was: 20m)
> `carbon get flow-configs` search facets not consistently working
> ----------------------------------------------------------------
>
> Key: GOBBLIN-2092
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2092
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Aditya Pratap Singh
> Priority: Minor
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The `carbon get flow-configs (search) seems to inconsistently apply search
> facets. it's possible that the facet indices are correctly set up during
> flow creation, but are not properly maintained during flow update. see
> interaction:
> {code:java}
> $ carbon get flow-configs -f prod-lva1 -s war-oh-iceberg | jq -c . | wc -l
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: None; source_identifier:
> war-oh-iceberg; destination_identifier: None; cron_schedule: None;
> run_immediately: None; owning_group: None; start: None; count: None)
> 63
> $ carbon get flow-configs -f prod-lva1 -s war-tl-iceberg | jq -c . | wc -l
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: None; source_identifier:
> war-tl-iceberg; destination_identifier: None; cron_schedule: None;
> run_immediately: None; owning_group: None; start: None; count: None)
> No flows found
> 0
> {code}
> so (at least some) results for sourceIdentifier of `war-oh-iceberg` do show,
> but none do for `war-tl-iceberg`. that's incorrect, because when I instead
> search by user, there are at least two `war-tl-iceberg` flows in `prod-lva1`:
> {code:java}
> $ carbon get flow-configs -f prod-lva1 -u lyndarel | jq -c '{flowGroup:
> .id.flowGroup, flowName: .id.flowName, user: .properties."user.to.proxy",
> between: (.properties."gobblin.flow.sourceIdentifier" + " => " +
> .properties."gobblin.flow.destinationIdentifier")}' | grep tl-iceberg
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: lyndarel; source_identifier:
> None; destination_identifier: None; cron_schedule: None; run_immediately:
> None; owning_group: None; start: None; count: None)
> {"flowGroup":"iceberg_based_openhouse_replication_u_lyndarel","flowName":"copy_to_holdem_replication_course_features","user":"lyndarel","between":"war-tl-iceberg
> => holdem-tl-iceberg"}
> {"flowGroup":"iceberg_based_openhouse_replication_u_lyndarel","flowName":"copy_to_holdem_replication_member_skill_gap","user":"lyndarel","between":"war-tl-iceberg
> => holdem-tl-iceberg"} {code}
> when the user and sourceId constraint are combined, those two no longer show
> up:
> {code:java}
> $ carbon get flow-configs -f prod-lva1 -u lyndarel -s war-tl-iceberg | jq -c
> '{flowGroup: .id.flowGroup, flowName: .id.flowName, user:
> .properties."user.to.proxy", between:
> (.properties."gobblin.flow.sourceIdentifier" + " => " +
> .properties."gobblin.flow.destinationIdentifier")}'
> Searching [fabric: prod-lva1] for flow matching - (flow_group: None;
> flow_name: None; template_uri: None; proxy_user: lyndarel; source_identifier:
> war-tl-iceberg; destination_identifier: None; cron_schedule: None;
> run_immediately: None; owning_group: None; start: None; count: None)
> No flows found {code}
> the reason I suspect flow update as a possible RC is that I had modified
> these two flows to use that sourceId, when they were originally created with
> another one. e.g. something like:
> {code:java}
> $ carbon update flow -f prod-lva1 -fg
> iceberg_based_openhouse_replication_u_lyndarel -fn
> copy_to_holdem_replication_member_skill_gap
> properties.gobblin.flow.sourceIdentifier=war-tl-iceberg,properties.gobblin.flow.destinationIdentifier=holdem-tl-iceberg{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
