phet commented on code in PR #3937:
URL: https://github.com/apache/gobblin/pull/3937#discussion_r1586640513
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/OrchestratorTest.java:
##########
@@ -114,6 +119,8 @@ public void setup() throws Exception {
FlowLaunchHandler mockFlowTriggerHandler = mock(FlowLaunchHandler.class);
DagManager mockDagManager = mock(DagManager.class);
doNothing().when(mockDagManager).setTopologySpecMap(anyMap());
+ TopologySpecFactory mockedTopologySpecFactory =
mock(TopologySpecFactory.class);
+
doReturn(Collections.singleton(this.topologySpec)).when(mockedTopologySpecFactory).getTopologies();
Review Comment:
prefer the `when().thenReturn()` form, which is typesafe
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/core/IdentityFlowToJobSpecCompilerTest.java:
##########
@@ -86,16 +87,11 @@ public void setup() throws Exception {
// Initialize compiler with template catalog
Properties compilerWithTemplateCatalogProperties = new Properties();
compilerWithTemplateCatalogProperties.setProperty(ServiceConfigKeys.TEMPLATE_CATALOGS_FULLY_QUALIFIED_PATH_KEY,
TEST_TEMPLATE_CATALOG_URI);
- this.compilerWithTemplateCalague = new
IdentityFlowToJobSpecCompiler(ConfigUtils.propertiesToConfig(compilerWithTemplateCatalogProperties));
-
- // Add a topology to compiler
- this.compilerWithTemplateCalague.onAddSpec(initTopologySpec());
+ this.compilerWithTemplateCalague = new
IdentityFlowToJobSpecCompiler(ConfigUtils.propertiesToConfig(compilerWithTemplateCatalogProperties),
Review Comment:
catalog(ue) is misspelled
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/BaseFlowToJobSpecCompiler.java:
##########
@@ -219,8 +199,6 @@ private AddSpecResponse onAddFlowSpec(FlowSpec flowSpec) {
public AddSpecResponse onAddSpec(Spec addedSpec) {
if (addedSpec instanceof FlowSpec) {
return onAddFlowSpec((FlowSpec) addedSpec);
- } else if (addedSpec instanceof TopologySpec) {
- return onAddTopologySpec( (TopologySpec) addedSpec);
Review Comment:
is it wise to remove the ability to be a TS listener?
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/BaseFlowToJobSpecCompiler.java:
##########
@@ -134,10 +112,12 @@ public BaseFlowToJobSpecCompiler(Config config,
Optional<Logger> log, boolean in
userQuotaManager = Optional.absent();
}
- this.topologySpecMap = Maps.newConcurrentMap();
+ topologySpecSet.forEach(this::onAddTopologySpec);
+
+
this.config = config;
- /***
+ /*
* ETL-5996
Review Comment:
this is not an apache gobblin jira ticket
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/OrchestratorTest.java:
##########
@@ -244,9 +250,10 @@ public void createTopologySpec() {
}
// Make sure TopologyCatalog is empty
Assert.assertTrue(specs.size() == 0, "Spec store should be empty before
addition");
- // Make sure TopologyCatalog Listener is empty
- Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 0,
"SpecCompiler should not know about any Topology "
- + "before addition");
+ Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 1,
"SpecCompiler should know about any Topology "
Review Comment:
desc here makes it sound like the test should be `> 1`. to that end, I
didn't notice any `Preconditions` check or similar to insist on a non-empty
collection. do we want one?
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/OrchestratorTest.java:
##########
@@ -244,9 +250,10 @@ public void createTopologySpec() {
}
// Make sure TopologyCatalog is empty
Assert.assertTrue(specs.size() == 0, "Spec store should be empty before
addition");
- // Make sure TopologyCatalog Listener is empty
- Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 0,
"SpecCompiler should not know about any Topology "
- + "before addition");
+ Assert.assertTrue(specCompiler.getTopologySpecMap().size() == 1,
"SpecCompiler should know about any Topology "
+ + " irrespective of what is there in the topology catalog");
+ // Make sure TopologyCatalog empty
+ Assert.assertTrue(this.topologyCatalog.getSize() == 0, "Topology catalog
should contain 0 Spec before addition");
Review Comment:
seems this asserts that the spec compiler's topos may now potentially be
inconsistent w/ the topo catalog's. that may make the system harder to reason
about... is it really a good thing to drop such an invariant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@gobblin.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
