Thanks! I think the HBase setup with S3 is not working now. Some tests with HBase are failing when I run commands from the HBase shell.
On Fri, Mar 27, 2020 at 10:21 AM [email protected] < [email protected]> wrote: > Hi Krish, > > I got some similar errors when i change `hbase.rootdir` value to the hdfs > filesystem as `hdfs://xxxx/tmp/hbase-root`. > But it's solved when i try to > 1. config Atlas Authentication, see more detail in > https://atlas.apache.org/#/Authentication > My hbase needs kerberos in my company,so add kerberos Authentication > > atlas.authentication.method.kerberos=true > [email protected] > atlas.authentication.method.kerberos.keytab=/home/atlas/atlas.keytab > atlas.authentication.method.kerberos.name.rules=RULE:[2:$1@$0]([email protected] > )s/.*/atlas/ > atlas.authentication.method.kerberos.token.validity=3600 > [email protected] > atlas.authentication.keytab=/home/atlas/atlas.keytab > > > 2. For connecting to hbase ,sometimes you need to add > hdfs-site.xml,core-site.xml, > mapred-site.xml,yarn-site.xml to Atlas classpath, classpath such like ' > HBASE_CONF_DIR' or '{ATLAS_HOME}/conf' > > > > ------------------------------ > [email protected] > > > *From:* Krish I <[email protected]> > *Date:* 2020-03-27 16:26 > *To:* user <[email protected]> > *Subject:* Atlas with JanusGraph on HBase backed with S3 giving errors > Hi, > > I am trying to set up Atlas on a K8s cluster in AWS with HBase backed by > S3. > > Everything works fine and works when I point the `hbase.rootdir` value to > the local filesystem as `file:///tmp/hbase-root`. > > When I change this to some S3 URI `s3://bucket/path`, this fails with a > very generic error when I start Atlas with: > > ``` > > 2020-03-27 08:20:55,293 INFO - [main:] ~ Loading > atlas-application.properties from > file:/apache-atlas-2.0.0/conf/atlas-application.properties > (ApplicationProperties:123) > 2020-03-27 08:20:55,299 INFO - [main:] ~ Using graphdb backend > 'org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase' > (ApplicationProperties:273) > 2020-03-27 08:20:55,299 INFO - [main:] ~ Using storage backend 'hbase2' > (ApplicationProperties:284) > 2020-03-27 08:20:55,299 INFO - [main:] ~ Using index backend 'solr' > (ApplicationProperties:295) > 2020-03-27 08:20:55,300 INFO - [main:] ~ Setting solr-wait-searcher > property 'true' (ApplicationProperties:301) > 2020-03-27 08:20:55,300 INFO - [main:] ~ Setting index.search.map-name > property 'false' (ApplicationProperties:305) > 2020-03-27 08:20:55,301 INFO - [main:] ~ Property (set to default) > atlas.graph.cache.db-cache = true (ApplicationProperties:318) > 2020-03-27 08:20:55,301 INFO - [main:] ~ Property (set to default) > atlas.graph.cache.db-cache-clean-wait = 20 (ApplicationProperties:318) > 2020-03-27 08:20:55,301 INFO - [main:] ~ Property (set to default) > atlas.graph.cache.db-cache-size = 0.5 (ApplicationProperties:318) > 2020-03-27 08:20:55,301 INFO - [main:] ~ Property (set to default) > atlas.graph.cache.tx-cache-size = 15000 (ApplicationProperties:318) > 2020-03-27 08:20:55,301 INFO - [main:] ~ Property (set to default) > atlas.graph.cache.tx-dirty-size = 120 (ApplicationProperties:318) > 2020-03-27 08:20:55,316 INFO - [main:] ~ > > ######################################################################################## > Atlas Server (STARTUP) > > project.name: apache-atlas > project.description: Metadata Management and Data Governance > Platform over Hadoop > build.user: root > build.epoch: 1585085591537 > project.version: 2.0.0 > build.version: 2.0.0 > vc.revision: release > vc.source.url: scm:git:git:// > git.apache.org/atlas.git/atlas-webapp > ######################################################################################## > (Atlas:215) > 2020-03-27 08:20:55,316 INFO - [main:] ~ > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (Atlas:216) > 2020-03-27 08:20:55,316 INFO - [main:] ~ Server starting with TLS ? false > on port 21000 (Atlas:217) > 2020-03-27 08:20:55,316 INFO - [main:] ~ > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< (Atlas:218) > 2020-03-27 08:20:55,961 INFO - [main:] ~ No authentication method > configured. Defaulting to simple authentication (LoginProcessor:102) > 2020-03-27 08:20:56,078 WARN - [main:] ~ Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > (NativeCodeLoader:60) > 2020-03-27 08:20:56,100 INFO - [main:] ~ Logged in user root > (auth:SIMPLE) (LoginProcessor:77) > 2020-03-27 08:20:56,703 INFO - [main:] ~ Not running setup per > configuration atlas.server.run.setup.on.start. > (SetupSteps$SetupRequired:189) > 2020-03-27 08:20:58,679 WARN - [main:] ~ Cannot locate configuration: > tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties > (MetricsConfig:134) > 2020-03-27 08:23:08,702 WARN - [main:] ~ Unexpected exception during > getDeployment() (HBaseStoreManager:399) > java.lang.RuntimeException: > org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in > storage backend > at > org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getDeployment(HBaseStoreManager.java:358) > at > org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getFeatures(HBaseStoreManager.java:397) > at > org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.<init>(GraphDatabaseConfiguration.java:1256) > at > org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:160) > at > org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:131) > at > org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:111) > at > org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraphInstance(AtlasJanusGraphDatabase.java:165) > at > org.apache.atlas.repository.graphdb.janus.AtlasJanusGraphDatabase.getGraph(AtlasJanusGraphDatabase.java:263) > at > org.apache.atlas.repository.graph.AtlasGraphProvider.getGraphInstance(AtlasGraphProvider.java:52) > at > org.apache.atlas.repository.graph.AtlasGraphProvider.get(AtlasGraphProvider.java:98) > at > org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$b936b499.CGLIB$get$1(<generated>) > at > org.apache.atlas.repository.graph.AtlasGraphProvider$$EnhancerBySpringCGLIB$$b936b499$$FastClassBySpringCGLIB$$fd3f07c6.invoke(<generated>) > at > org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) > ... > ... > ... > at > org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) > at > org.apache.atlas.web.service.EmbeddedServer.start(EmbeddedServer.java:98) > at org.apache.atlas.Atlas.main(Atlas.java:133) > Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary > failure in storage backend > at > org.janusgraph.diskstorage.hbase2.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:732) > at > org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:518) > at > org.janusgraph.diskstorage.hbase2.HBaseStoreManager.getDeployment(HBaseStoreManager.java:355) > ... 92 more > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: > Failed after attempts=16, exceptions: > Fri Mar 27 08:20:59 UTC 2020, > RpcRetryingCaller{globalStartTime=1585297258222, pause=100, > maxAttempts=16}, org.apache.hadoop.hbase.PleaseHoldException: > org.apache.hadoop.hbase.PleaseHoldException: Ma > ster is initializing > ... > ... > ... > ``` > > The main bone of contention I think is this line `2020-03-27 08:20:58,679 > WARN - [main:] ~ Cannot locate configuration: tried > hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties > (MetricsConfig:134)` which I do not see when running with the local rootdir. > Please not that I am not running Hadoop/HDFS for now and do not intend to; > we will guarantee consistency to S3 in the future using other tech or move > to DynamoDB or something else. > Please also note that I restart the ZK, Hbase, Atlas instance for testing > numerous time during this setup, so the only persistence I have is the > HBase initializaion data in stored in S3. > > However, the HBase master and regionserver logs themselves show no error > at all when running with S3 as the rootdir. I have also dropped in an hbase > shell and checked the corresponding HBase web interfaces to check the HBase > with S3 is working fine. > > This makes me believe that there is an issue with Janusgraph <-> Hbase > interaction, which I am not sure how to debug. I have solf-linked the > `/atlas/conf/hbase` directory to the actual HBase conf dir at `/hbase/conf`. > > Any pointers will be helpful here as I think I am going in blond over > here. :) > > Best, > Krish > >
