Hi krishna, I guess that you set EMR to use AWS Glue catalog as Hive metadata
and Kylin is missing the AWS lib com.amazonaws.glue in Kylin's classpath. Maybe /usr/lib/hive/auxlib/aws-glue-datacatalog-hive2-client.jar(https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore/blob/master/aws-glue-datacatalog-hive2-client/src/main/java/com/amazonaws/glue/catalog/metastore/AWSGlueDataCatalogHiveClientFactory.java)? You should find the lib in the EMR cluster and add it to yourclass path(maybe $KYLIN_HOME/lib). If you cannot find the right jar, you may package it manually, repo should be this https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore. Maybe ask EMR customer service for help should be considered. ----------------- ----------------- Best wishes to you ! From ：Xiaoxiang Yu At 2019-07-16 10:40:10, "Krishna Bandaru" <krishn...@scanbuy.com> wrote: hi I created Kylin cluster with HA(3 masters and 2 cores) java.lang.RuntimeException: java.io.IOException: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found) at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:97) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:122) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:100) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163) at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:111) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found) at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:97) at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:51) at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:94) ... 10 more Caused by: MetaException(message:Unable to instantiate a metastore client factory com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory due to: java.lang.ClassNotFoundException: Class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory not found) at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClientFactory(HiveUtils.java:525) at org.apache.hadoop.hive.ql.metadata.HiveUtils.createMetaStoreClient(HiveUtils.java:506) at org.apache.hive.hcatalog.common.HiveClientCache.getNonCachedHiveMetastoreClient(HiveClientCache.java:99) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:318) at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:315) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4791) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3584) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250) at com.google.common.cache.LocalCache.get(LocalCache.java:3985) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4788) at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:315) at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:277) at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558) at org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(InitializeInput.java:104) at org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.java:88) at org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.java:95) ... 12 more I am getting below error please help me to get rid off this. even though i set <property> <name>hive.aux.jars.path </name> <value>file:///usr/lib/hive/auxlib/aws-glue-datacatalog-hive2-client.jar</value> </property> in hive-site.xml i am getting above error while cube designing in step 2(#2 Step Name: Extract Fact Table Distinct Columns) thanks, krishna
