Hi Jay,
You can use Gobblin-as-a-Service with Gobblin-cluster. Jobs can be created in GaaS through a REST call or through a git monitor. GaaS will then send the job spec to the cluster through kafka. On the cluster side the jobs received through kafka are put into a JobCatalog. If you would like to use the cluster without kafka then you can change the JobCatalog type to one that monitors the job conf path. The default for 'gobblin.cluster.job.catalog' is 'org.apache.gobblin.runtime.job_catalog.NonObservingFSJobCatalog' which does not monitor the path and is intended to be used with the kafka job monitor. You can set 'gobblin.cluster.job.catalog' to 'org.apache.gobblin.runtime.job_catalog.FSJobCatalog' to use a job catalog that monitors an FS path. I haven't tried the config above. We have only used the cluster only with an initial set of files in the job conf dir or dynamic jobs through kafka. Also, there is no way to disable the JobConfigurationManager, so it will submit the initial set of jobs on startup. Hung. ________________________________ From: Jay Sen <jay...@apache.org> Sent: Thursday, February 28, 2019 12:06:05 AM To: dev@gobblin.incubator.apache.org Subject: JobConfigurationManager ? Hi Dev Team, just wanted to confirm that there is no scheduled local job file configuration manager that will periodically check the job conf path for file updates/addition. I only see scheduled and streaming job config manager for kafka only. does LinkedIn store the job config in kafka from where the gobblin picks from ? can someone pls explain ? thanks Jay
