[
https://issues.apache.org/jira/browse/GOBBLIN-1678?focusedWorklogId=802612&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-802612
]
ASF GitHub Bot logged work on GOBBLIN-1678:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 22/Aug/22 20:11
Start Date: 22/Aug/22 20:11
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3536:
URL: https://github.com/apache/gobblin/pull/3536#discussion_r951868284
##########
gobblin-service/src/main/java/org/apache/gobblin/service/monitoring/GitConfigMonitor.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.gobblin.service.monitoring;
+
+import com.google.common.collect.ImmutableMap;
+import com.typesafe.config.Config;
+import com.typesafe.config.ConfigFactory;
+
+import javax.inject.Inject;
+import javax.inject.Singleton;
+import lombok.extern.slf4j.Slf4j;
+
+import org.apache.gobblin.configuration.ConfigurationKeys;
+import org.apache.gobblin.runtime.spec_catalog.FlowCatalog;
+
+/**
+ * Service that monitors for jobs from a git repository.
+ * The git repository must have an initial commit that has no config files
since that is used as a base for getting
+ * the change list.
+ * The config needs to be organized with the following structure:
+ * <root_config_dir>/<flowGroup>/<flowName>.(pull|job|json|conf)
+ * The <flowGroup> and <flowName> is used to generate the URI used to store
the config in the {@link FlowCatalog}
+ */
+@Slf4j
+@Singleton
+public class GitConfigMonitor extends GitMonitoringService {
+ public static final String GIT_CONFIG_MONITOR_PREFIX =
"gobblin.service.gitConfigMonitor";
+
+ private static final String PROPERTIES_EXTENSIONS = "pull,job";
+ private static final String CONF_EXTENSIONS = "json,conf";
+ private static final String DEFAULT_GIT_CONFIG_MONITOR_REPO_DIR =
"git-flow-config";
+ private static final String DEFAULT_GIT_CONFIG_MONITOR_CONFIG_DIR =
"gobblin-config";
+ private static final String DEFAULT_GIT_CONFIG_MONITOR_BRANCH_NAME =
"master";
+
+ private static final int DEFAULT_GIT_CONFIG_MONITOR_POLLING_INTERVAL = 60;
+
+ private static final Config DEFAULT_FALLBACK =
+ ConfigFactory.parseMap(ImmutableMap.<String, Object>builder()
+ .put(ConfigurationKeys.GIT_MONITOR_REPO_DIR,
DEFAULT_GIT_CONFIG_MONITOR_REPO_DIR)
+ .put(ConfigurationKeys.GIT_MONITOR_CONFIG_BASE_DIR,
DEFAULT_GIT_CONFIG_MONITOR_CONFIG_DIR)
+ .put(ConfigurationKeys.GIT_MONITOR_BRANCH_NAME,
DEFAULT_GIT_CONFIG_MONITOR_BRANCH_NAME)
+ .put(ConfigurationKeys.GIT_MONITOR_POLLING_INTERVAL,
DEFAULT_GIT_CONFIG_MONITOR_POLLING_INTERVAL)
+ .put(ConfigurationKeys.JAVA_PROPS_EXTENSIONS, PROPERTIES_EXTENSIONS)
+ .put(ConfigurationKeys.HOCON_FILE_EXTENSIONS, CONF_EXTENSIONS)
+ .build());
+
+ private final FlowCatalog flowCatalog;
+
+ @Inject
+ GitConfigMonitor(Config config, FlowCatalog flowCatalog) {
+
super(config.getConfig(GIT_CONFIG_MONITOR_PREFIX).withFallback(DEFAULT_FALLBACK));
+ this.flowCatalog = flowCatalog;
+ Config configWithFallbacks =
config.getConfig(GIT_CONFIG_MONITOR_PREFIX).withFallback(DEFAULT_FALLBACK);
+ this.listeners.add(new GitConfigListener(flowCatalog,
configWithFallbacks.getString(ConfigurationKeys.GIT_MONITOR_REPO_DIR),
+
configWithFallbacks.getString(ConfigurationKeys.GIT_MONITOR_CONFIG_BASE_DIR),
configWithFallbacks.getString(ConfigurationKeys.JAVA_PROPS_EXTENSIONS),
+
configWithFallbacks.getString(ConfigurationKeys.HOCON_FILE_EXTENSIONS)));
+ }
+
+ @Override
+ public boolean shouldPollGit() {
+ // if not active or if the flow catalog is not up yet then can't process
config changes
+ if (!isActive || !this.flowCatalog.isRunning()) {
Review Comment:
A general question, does the git config monitor works only when that
particular instance is active?
Issue Time Tracking
-------------------
Worklog Id: (was: 802612)
Time Spent: 2h 10m (was: 2h)
> Refactor GaaS Flowgraph Monitor to be extensible
> ------------------------------------------------
>
> Key: GOBBLIN-1678
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1678
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: William Lo
> Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> To support new implementations of a flow graph monitor, which allows for live
> updating of a flowgraph, we should reuse as much implementation from the
> existing git flowgraph monitor as possible.
> The current flowgraph monitor has coupled logic to perform a lot of the
> adding node/edges which can be reused for a file based flowgraph.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
