[
https://issues.apache.org/jira/browse/ATLAS-628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230268#comment-15230268
]
Hemanth Yamijala commented on ATLAS-628:
----------------------------------------
The exception was occurring because one of the stateful service's lifecycle was
not split into initialization and activation as part of ATLAS-511. This is the
TitanGraphProvider which is initialized by a bunch of other classes either as
part of their class initialization or object initialization. As part of the
initialization of this class, it checks and creates the Titan schema in the
Graph backend, in our case, HBase for the very first time.
The general pattern to solve this problem that we've used in ATLAS-511 is to
split the lifecycle of such stateful services into 2 explicit stages:
initialization and activation, where stateful work is done in the activation
stage, and only happens from the active instance. This can certainly be done
for the TitanGraphProvider, but it is too deeply nested in many other objects
of Atlas core.
A simpler option could be to have an explicit setting up of Atlas schema before
the Atlas server starts the first time. After the schema is initialized, Atlas
service instances can come up at the same time and there would be no problem.
Running this setup could be a post install step for Atlas and could be scripted
easily.
Another such setup operation that could be done as part of such a script is the
setting up of Kafka topics that is the target of ATLAS-515. Given the
advantages of this approach, I am proposing to do this as a fix.
I quickly prototyped this option and tested bringing up multiple instances at
the same time after the setup is run. It works fine and fixes the issue. Please
let me know if there are any concerns anyone sees.
> Starting two Atlas instances at the same time causes exceptions in HA mode.
> ---------------------------------------------------------------------------
>
> Key: ATLAS-628
> URL: https://issues.apache.org/jira/browse/ATLAS-628
> Project: Atlas
> Issue Type: Bug
> Affects Versions: 0.7-incubating
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Priority: Blocker
> Fix For: 0.7-incubating
>
>
> Configure HA parameters for Atlas. Use SOLR and HBase as backends. From two
> terminals start two instances of Atlas at roughly the same time. This causes
> failure exceptions and at least one instance does not start up properly (in
> either active or passive mode). The relevant stack trace is:
> {code}
> Caused by: java.lang.ExceptionInInitializerError
> at
> org.apache.atlas.repository.graph.GraphBackedMetadataRepository.<clinit>(GraphBackedMetadataRepository.java:67)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:278)
> at
> com.google.inject.internal.cglib.core.$ReflectUtils.defineClass(ReflectUtils.java:386)
> at
> com.google.inject.internal.cglib.core.$AbstractClassGenerator.create(AbstractClassGenerator.java:219)
> at
> com.google.inject.internal.cglib.proxy.$Enhancer.createHelper(Enhancer.java:377)
> at
> com.google.inject.internal.cglib.proxy.$Enhancer.createClass(Enhancer.java:317)
> at
> com.google.inject.internal.ProxyFactory$ProxyConstructor.<init>(ProxyFactory.java:252)
> at com.google.inject.internal.ProxyFactory.create(ProxyFactory.java:178)
> at
> com.google.inject.internal.ConstructorInjectorStore.createConstructor(ConstructorInjectorStore.java:90)
> at
> com.google.inject.internal.ConstructorInjectorStore.access$000(ConstructorInjectorStore.java:29)
> at
> com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:37)
> at
> com.google.inject.internal.ConstructorInjectorStore$1.create(ConstructorInjectorStore.java:33)
> at com.google.inject.internal.FailableCache$1.load(FailableCache.java:37)
> at
> com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
> at
> com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
> at
> com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
> at
> com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4880)
> at com.google.inject.internal.FailableCache.get(FailableCache.java:48)
> at
> com.google.inject.internal.ConstructorInjectorStore.get(ConstructorInjectorStore.java:50)
> at
> com.google.inject.internal.ConstructorBindingImpl.initialize(ConstructorBindingImpl.java:136)
> at
> com.google.inject.internal.InjectorImpl.initializeJitBinding(InjectorImpl.java:547)
> at
> com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:884)
> at
> com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:805)
> at
> com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:282)
> at
> com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:214)
> at
> com.google.inject.internal.InjectorImpl.getInternalFactory(InjectorImpl.java:890)
> at com.google.inject.internal.FactoryProxy.notify(FactoryProxy.java:46)
> at
> com.google.inject.internal.ProcessedBindingData.runCreationListeners(ProcessedBindingData.java:50)
> at
> com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:134)
> ... 24 more
> Caused by: java.lang.IllegalArgumentException: Database has already been
> initialized but not frozen
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
> at
> com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1294)
> at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:93)
> at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:73)
> at
> org.apache.atlas.repository.graph.TitanGraphProvider.getGraphInstance(TitanGraphProvider.java:105)
> at
> org.apache.atlas.repository.graph.GraphHelper.<clinit>(GraphHelper.java:64)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
