[
https://issues.apache.org/jira/browse/GOBBLIN-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093032#comment-16093032
]
mazhiyong commented on GOBBLIN-12:
----------------------------------
thanks [~Zhixiong Chen]
this is my config file.
#job.conf
gobblin.yarn.work.dir=/user/yarn/gobblin-yarn
fs.uri=hdfs://mzcluster
mr.job.max.mappers=1
mr.job.root.dir=${gobblin.yarn.work.dir}/working
topic.whitelist=test
kafka.brokers=mz-bigdata-32-201:9092,mz-bigdata-32-202:9092,mz-bigdata-32-203:9092
kafka.workunit.packer.type=BI_LEVEL
bootstrap.with.offset=earliest
job.name=GobblinKafkaQuickStart
job.group=GobblinKafka
job.description=Gobblin quick start job for Kafka
job.schedule=0 0/10 * * * ?
source.class=gobblin.source.extractor.extract.kafka.KafkaSimpleSource
extract.namespace=gobblin.extract.kafka
extract.limit.enabled=true
extract.limit.type=time
extract.limit.timeLimitTimeunit=minutes
extract.limit.timeLimit=10
writer.builder.class=gobblin.writer.SimpleDataWriterBuilder
writer.file.path.type=tablename
writer.destination.type=HDFS
writer.output.format=txt
writer.fs.uri=${fs.uri}
metrics.reporting.file.enabled=true
metrics.log.dir=${gobblin.yarn.work.dir}/metrics
metrics.reporting.file.suffix=txt
data.publisher.type=gobblin.publisher.BaseDataPublisher
data.publisher.final.dir=/gobblintest/job-output
data.publisher.replace.final.dir=false
application.conf
gobblin.yarn.app.name=gobblin-yarn
gobblin.yarn.app.master.memory.mbs=512
gobblin.yarn.initial.containers=2
gobblin.yarn.container.memory.mbs=2048
gobblin.yarn.conf.dir=${gobblin.home.dir}/conf/yarn
gobblin.yarn.lib.jars.dir=${gobblin.home.dir}/lib
gobblin.yarn.app.master.files.local=${gobblin.yarn.conf.dir}"/log4j-yarn.properties,"${gobblin.yarn.conf.dir}"/application.conf,"${gobblin.yarn.conf.dir}"/reference.conf"
gobblin.yarn.container.files.local=${gobblin.yarn.app.master.files.local}
gobblin.yarn.logs.sink.root.dir=${gobblin.home.dir}/applogs
gobblin.cluster.helix.cluster.name=gobblin-yarn
gobblin.cluster.job.conf.path=${gobblin.home.dir}/jobconf
writer.fs.uri=${fs.uri}
state.store.fs.uri=${fs.uri}
writer.destination.type=HDFS
writer.output.format=AVRO
writer.staging.dir=${gobblin.yarn.work.dir}/task-staging
writer.output.dir=${gobblin.yarn.work.dir}/task-output
data.publisher.type=gobblin.publisher.BaseDataPublisher
data.publisher.final.dir=${gobblin.yarn.work.dir}/job-output
data.publisher.replace.final.dir=false
state.store.dir=${gobblin.yarn.work.dir}/state-store
qualitychecker.row.err.file=${gobblin.yarn.work.dir}/err
job.lock.enabled=true
job.lock.type=gobblin.runtime.locks.ZookeeperBasedJobLock
metrics.log.dir=${gobblin.yarn.work.dir}/metrics
task.status.reportintervalinms=1000
metrics.enabled=true
#reference.conf
gobblin.home.dir=/opt/gobblin-dist
gobblin.yarn.app.queue=default
gobblin.yarn.app.name=gobblin-yarn
gobblin.yarn.app.master.memory.mbs=512
gobblin.yarn.app.master.cores=1
gobblin.yarn.app.report.interval.minutes=5
gobblin.yarn.max.get.app.report.failures=4
gobblin.yarn.email.notification.on.shutdown=false
gobblin.yarn.initial.containers=1
gobblin.yarn.container.memory.mbs=512
gobblin.yarn.container.cores=1
gobblin.yarn.container.affinity.enabled=true
gobblin.yarn.helix.instance.max.retries=2
gobblin.yarn.keytab.login.interval.minutes=1440
gobblin.yarn.token.renew.interval.minutes=720
gobblin.yarn.work.dir=/gobblin
gobblin.cluster.helix.cluster.name=gobblin-yarn
gobblin.cluster.zk.connection.string=${zookeeper.connection.string}
gobblin.locks.zookeeper.acquire.timeout_milliseconds=10000
gobblin.locks.zookeeper.connection_string=${zookeeper.connection.string}
fs.uri="hdfs://mzcluster"
job.execinfo.server.enabled=false
zookeeper.connection.string="mz-bigdata-32-20:2181,mz-bigdata-32-202:2181,mz-bigdata-32-203:2181"
zookeeper.session.timeout.seconds=180
zookeeper.connection.timeout.seconds=30
zookeeper.retry.backoff.seconds=1
zookeeper.retry.count.max=10
> gobblin on yarn mode,AbstractJobLauncher can't lock and throw Excetion for
> some reson
> -------------------------------------------------------------------------------------
>
> Key: GOBBLIN-12
> URL: https://issues.apache.org/jira/browse/GOBBLIN-12
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: mazhiyong
>
> when i running bin/gobblin-yarn.sh start, the job throw exception,
> ## ApplicationMaster Log
> 2017-06-11 00:45:00 CST WARN [TaskStateCollectorService RUNNING]
> gobblin.runtime.TaskStateCollectorService - No output task state files found
> in
> hdfs://mycluster/user/yarn/GobblinYarn/application_1497109223607_0007/_taskstates/job_GobblinKafkaQuickStart_1497112920089
> 2017-06-11 00:46:00 CST INFO [DefaultQuartzScheduler_Worker-3]
> org.quartz.core.JobRunShell - Job GobblinKafka.GobblinKafkaQuickStart threw
> a JobExecutionException:
> org.quartz.JobExecutionException: gobblin.runtime.JobException: Previous
> instance of job GobblinKafkaQuickStart is still running, skipping this
> scheduled run [See nested exception: gobblin.runtime.JobException: Previous
> instance of job GobblinKafkaQuickStart is still running, skipping this
> scheduled run]
> at gobblin.cluster.GobblinHelixJob.executeImpl(GobblinHelixJob.java:87)
> at gobblin.scheduler.BaseGobblinJob.execute(BaseGobblinJob.java:53)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: gobblin.runtime.JobException: Previous instance of job
> GobblinKafkaQuickStart is still running, skipping this scheduled run
> at
> gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:163)
> at
> gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:144)
> at
> gobblin.cluster.GobblinHelixJobLauncher.<init>(GobblinHelixJobLauncher.java:127)
> at gobblin.cluster.GobblinHelixJob.executeImpl(GobblinHelixJob.java:65)
> ... 3 more
> 2017-06-11 00:46:00 CST WARN [TaskStateCollectorService RUNNING]
> gobblin.runtime.TaskStateCollectorService - No output task state files found
> in
> hdfs://mycluster/user/yarn/GobblinYarn/application_1497109223607_0007/_taskstates/job_GobblinKafkaQuickStart_1497112920089
> 2017-06-11 00:47:00 CST WARN [TaskStateCollectorService RUNNING]
> gobblin.runtime.TaskStateCollectorService - No output task state files found
> in
> hdfs://mycluster/user/yarn/GobblinYarn/application_1497109223607_0007/_taskstates/job_GobblinKafkaQuickStart_1497112920089
> 2017-06-11 00:48:00 CST INFO [DefaultQuartzScheduler_Worker-4]
> org.quartz.core.JobRunShell - Job GobblinKafka.GobblinKafkaQuickStart threw
> a JobExecutionException:
> org.quartz.JobExecutionException: gobblin.runtime.JobException: Previous
> instance of job GobblinKafkaQuickStart is still running, skipping this
> scheduled run [See nested exception: gobblin.runtime.JobException: Previous
> instance of job GobblinKafkaQuickStart is still running, skipping this
> scheduled run]
> at gobblin.cluster.GobblinHelixJob.executeImpl(GobblinHelixJob.java:87)
> at gobblin.scheduler.BaseGobblinJob.execute(BaseGobblinJob.java:53)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: gobblin.runtime.JobException: Previous instance of job
> GobblinKafkaQuickStart is still running, skipping this scheduled run
> at
> gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:163)
> at
> gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:144)
> at
> gobblin.cluster.GobblinHelixJobLauncher.<init>(GobblinHelixJobLauncher.java:127)
> at gobblin.cluster.GobblinHelixJob.executeImpl(GobblinHelixJob.java:65)
> ... 3 more
> public AbstractJobLauncher(Properties jobProps, List<? extends Tag<?>>
> metadataTags,
> @Nullable SharedResourcesBroker<GobblinScopeTypes> instanceBroker)
> throws Exception {
>
> Preconditions.checkArgument(jobProps.containsKey(ConfigurationKeys.JOB_NAME_KEY),
> A job must have a job name specified by job.name);
> // Add clusterIdentifier tag so that it is added to any new TaskState
> created
> List<Tag<?>> clusterNameTags = Lists.newArrayList();
> clusterNameTags.addAll(Tag.fromMap(ClusterNameTags.getClusterNameTags()));
> GobblinMetrics.addCustomTagsToProperties(jobProps, clusterNameTags);
> // Make a copy for both the system and job configuration properties
> this.jobProps = new Properties();
> this.jobProps.putAll(jobProps);
> if (!tryLockJob(this.jobProps)) {
> throw new JobException(String.format(Previous instance of job %s is
> still running, skipping this scheduled run,
> this.jobProps.getProperty(ConfigurationKeys.JOB_NAME_KEY)));
> }
> .......
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/1938
> *Github Reporter* : [~phenixmzy]
> *Github Created At* : 2017-06-10T20:02:43Z
> *Github Updated At* : 2017-07-12T21:49:48Z
> h3. Comments
> ----
> [~phenixmzy] wrote on 2017-06-12T09:13:49Z : i am use ZookeeperBasedJobLock
> and found this.jobLockOptional.isPresent() always is true and
> this.jobLockOptional.get().tryLock() always is false ,in the
> AbstractJobLauncher#tryLockJob, so that can't lock and throws
> gobblin.runtime.JobException: Previous instance of job GobblinKafkaQuickStart
> is still running, skipping this scheduled run.
> i am query zookeeper,
> executor ls /locks/GobblinKafkaQuickStart/leases , return
> [_c_1a81243e-cf64-4839-9187-ef7ee65c729c-lease-0000000001];
> executor ls /locks/GobblinKafkaQuickStart/locks return []
> but what's reason can't be locked?
>
> *Github Url* :
> https://github.com/linkedin/gobblin/issues/1938#issuecomment-307733856
> ----
> [~zxc] wrote on 2017-07-12T21:49:48Z : Did you specify `job.lock.type`
> configuration? Can you provide the job file? It's expected that
> `this.jobLockOptional.isPresent()` is always true as you're using a lock. The
> problem is why `this.jobLockOptional.get().tryLock()` always returns false.
>
> *Github Url* :
> https://github.com/linkedin/gobblin/issues/1938#issuecomment-314907460
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
