[
https://issues.apache.org/jira/browse/GOBBLIN-2056?focusedWorklogId=917156&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-917156
]
ASF GitHub Bot logged work on GOBBLIN-2056:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 01/May/24 18:55
Start Date: 01/May/24 18:55
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3937:
URL: https://github.com/apache/gobblin/pull/3937#discussion_r1586640513
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/OrchestratorTest.java:
##########
@@ -114,6 +119,8 @@ public void setup() throws Exception {
FlowLaunchHandler mockFlowTriggerHandler = mock(FlowLaunchHandler.class);
DagManager mockDagManager = mock(DagManager.class);
doNothing().when(mockDagManager).setTopologySpecMap(anyMap());
+ TopologySpecFactory mockedTopologySpecFactory =
mock(TopologySpecFactory.class);
+
doReturn(Collections.singleton(this.topologySpec)).when(mockedTopologySpecFactory).getTopologies();
Review Comment:
prefer the `when().thenReturn()` form, which is typesafe
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/core/IdentityFlowToJobSpecCompilerTest.java:
##########
@@ -86,16 +87,11 @@ public void setup() throws Exception {
// Initialize compiler with template catalog
Properties compilerWithTemplateCatalogProperties = new Properties();
compilerWithTemplateCatalogProperties.setProperty(ServiceConfigKeys.TEMPLATE_CATALOGS_FULLY_QUALIFIED_PATH_KEY,
TEST_TEMPLATE_CATALOG_URI);
- this.compilerWithTemplateCalague = new
IdentityFlowToJobSpecCompiler(ConfigUtils.propertiesToConfig(compilerWithTemplateCatalogProperties));
-
- // Add a topology to compiler
- this.compilerWithTemplateCalague.onAddSpec(initTopologySpec());
+ this.compilerWithTemplateCalague = new
IdentityFlowToJobSpecCompiler(ConfigUtils.propertiesToConfig(compilerWithTemplateCatalogProperties),
Review Comment:
catalog(ue) is misspelled
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/BaseFlowToJobSpecCompiler.java:
##########
@@ -219,8 +199,6 @@ private AddSpecResponse onAddFlowSpec(FlowSpec flowSpec) {
public AddSpecResponse onAddSpec(Spec addedSpec) {
if (addedSpec instanceof FlowSpec) {
return onAddFlowSpec((FlowSpec) addedSpec);
- } else if (addedSpec instanceof TopologySpec) {
- return onAddTopologySpec( (TopologySpec) addedSpec);
Review Comment:
is it wise to remove the ability to be a TS listener?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/BaseFlowToJobSpecCompiler.java:
##########
@@ -134,10 +112,12 @@ public BaseFlowToJobSpecCompiler(Config config,
Optional<Logger> log, boolean in
userQuotaManager = Optional.absent();
}
- this.topologySpecMap = Maps.newConcurrentMap();
+ topologySpecSet.forEach(this::onAddTopologySpec);
+
+
this.config = config;
- /***
+ /*
* ETL-5996
Review Comment:
this is not an apache gobblin jira ticket
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/OrchestratorTest.java:
##########
@@ -244,9 +250,10 @@ public void createTopologySpec() {
}
// Make sure TopologyCatalog is empty
Assert.assertTrue(specs.size() == 0, "Spec store should be empty before
addition");
- // Make sure TopologyCatalog Listener is empty
- Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 0,
"SpecCompiler should not know about any Topology "
- + "before addition");
+ Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 1,
"SpecCompiler should know about any Topology "
Review Comment:
desc here makes it sound like the test should be `> 1`. to that end, I
didn't notice any `Preconditions` check or similar to insist on a non-empty
collection. do we want one?
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/OrchestratorTest.java:
##########
@@ -244,9 +250,10 @@ public void createTopologySpec() {
}
// Make sure TopologyCatalog is empty
Assert.assertTrue(specs.size() == 0, "Spec store should be empty before
addition");
- // Make sure TopologyCatalog Listener is empty
- Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 0,
"SpecCompiler should not know about any Topology "
- + "before addition");
+ Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 1,
"SpecCompiler should know about any Topology "
+ + " irrespective of what is there in the topology catalog");
+ // Make sure TopologyCatalog empty
+ Assert.assertTrue(this.topologyCatalog.getSize() == 0, "Topology catalog
should contain 0 Spec before addition");
Review Comment:
seems this asserts that the spec compiler's topos may now potentially be
inconsistent w/ the topo catalog's. that may make the system harder to reason
about... is it really a good thing to drop such an invariant?
Issue Time Tracking
-------------------
Worklog Id: (was: 917156)
Time Spent: 1h 20m (was: 1h 10m)
> initialize topology specs directly from the configs
> ---------------------------------------------------
>
> Key: GOBBLIN-2056
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2056
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Arjun Singh Bora
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> initialize topology specs directly from the configs instead of waiting for
> the callbacks of topology spec catalog listerners
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
