The following SQL causes the Hive Hook to fail with the error below: create table bigsql.missy999 as select * from bigsql.brancha;
Question: Is this a new SEVERE BUG in Atlas7rc2 Or has this bug been fixed in .8 but not in .7rc2 ? Excerpt from /var/log/hive/hiveserver2.log ========================================== 2016-12-29 11:52:10,787 INFO bridge.HiveMetaStoreBridge (HiveMetaStoreBridge.ja\ va:createOrUpdateDBInstance(162)) - Importing objects from databaseName : bigsql 2016-12-29 11:52:10,787 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInf\ o(746)) - 6: get_table : db=bigsql tbl=missy999 2016-12-29 11:52:10,787 INFO HiveMetaStore.audit (HiveMetaStore.java:logAuditEv\ ent(371)) - ugi=admin ip=unknown-ip-addr cmd=get_table : db=bigsql tbl=mi\ ssy999 2016-12-29 11:52:10,918 ERROR metadata.Hive (Hive.java:getTable(1119)) - Table m\ issy999 not found: bigsql.missy999 table not found 2016-12-29 11:52:10,919 ERROR hook.HiveHook (HiveHook.java:run(184)) - Atlas hoo\ k failed due to error org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found missy9\ 99 at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1120) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1090) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities (HiveHook.j\ ava:503) at org.apache.atlas.hive.hook.HiveHook.createOrUpdateEntities (HiveHook.j\ ava:524) at org.apache.atlas.hive.hook.HiveHook.processHiveEntity (HiveHook.java:6\ 05) at org.apache.atlas.hive.hook.HiveHook.registerProcess (HiveHook.java:585\ ) at org.apache.atlas.hive.hook.HiveHook.fireAndForget (HiveHook.java:223) at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:78) at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:182) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:51\ 1) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.\ java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor\ .java:617) at java.lang.Thread.run(Thread.java:745) 2016-12-29 11:52:11,221 INFO log.PerfLogger (PerfLogger.java:PerfLogEnd (148)) -\ </PERFLOG method=serializePlan start=1483041130037 end=1483041131221 duration=1\ 184 from=org.apache.hadoop.hive.ql.exec.Utilities> 2016-12-29 11:52:11,348 ERROR mr.ExecDriver (ExecDriver.java:execute(400)) - yar\ n 2016-12-29 11:52:13,528 INFO impl.TimelineClientImpl (TimelineClientImpl.java:s\ erviceInit(296)) - Timeline service address: http://biginsights.ibm.com:8188/ws/\ v1/timeline/ 2016-12-29 11:52:13,641 INFO client.RMProxy (RMProxy.java:createRMProxy (98)) - \ Connecting to ResourceManager at biginsights.ibm.com/192.168.1.132:8050 2016-12-29 11:52:14,269 INFO fs.FSStatsPublisher (FSStatsPublisher.java:init(49\ )) - created : hdfs://biginsights.ibm.com:8020/apps/hive/warehouse/bigsql.db/.hi\ ve-staging_hive_2016-12-29_11-52-08_283_558364161490028907-1/-ext-10002 Russell G. Anderson Senior Technical Consultant From: Hemanth Yamijala <yhema...@gmail.com> To: dev@atlas.incubator.apache.org Cc: Russell Anderson/Worcester/IBM@IBMUS, Barry Rosen/Worcester/IBM@IBMUS Date: 12/29/2016 11:17 AM Subject: Re: Hive Hook - what is missing ? [ RESOLVED } No worries. Glad it helped. On 27-Dec-2016 21:18, "Russell Anderson" <r...@us.ibm.com> wrote: Hi Memanth Yamijala, The magic of your thoughts !!! So I did as you suggested but then I noticed a couple of things that hadn't seemed important at the time : 1) In my Hadoop there were multiple entries for PRE, and FAILURE for Hive Hook. These were left as default - so i changed these to match the one recommended in the documentation. 2) I re-copied the libraries i had built where my AUX_PATH was pointing to. 3) I changed the permissions to match all the other jars in the hive/lib directory - which were 'root'. 4) I then carefully examined the var/log/hive/hiveserver2.log for Hive Hook messages I now have the Hive Hook working as desired !!! Thank you for getting me to look and think about things....!!! Russ. [image: Inactive hide details for Hemanth Yamijala ---12/26/2016 08:11:29 PM---Hi, A few questions to help debug:]Hemanth Yamijala ---12/26/2016 08:11:29 PM---Hi, A few questions to help debug: From: Hemanth Yamijala <yhema...@gmail.com> To: dev@atlas.incubator.apache.org Date: 12/26/2016 08:11 PM Subject: Re: Hive Hook - what is missing ? ------------------------------ Hi, A few questions to help debug: 1) Are you using Hive CLI or Beeline? Best results would be to configure these hooks with HiveServer2 and use Beeline. 2) Could you please check hive logs and look for AtlasHook related messages? Thanks Hemanth On Mon, Dec 26, 2016 at 8:59 PM, Russell Anderson <r...@us.ibm.com> wrote: > > > Hi dev list, > > I have built the Atlas7rc2 - have working the Hive Import, the Dashboard, > and generally things appear to be working as expected. > > However, I have followed the instruction below but cannot seem to get the > Hive Hook to detect table hive creations to be added to the metadata / > lineage. > > Can anyone suggest how to debug what could be missing regardless of > following the instructions below ? Any and all help appreciated !!! > > Regards, > > Russ. > > ============================================================ ================= > > Hive Hook > > > Hive supports listeners on hive command execution using hive hooks. This is > used to add/update/remove entities in Atlas using the model defined in > org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the > request to a thread pool executor to avoid blocking the command execution. > The thread submits the entities as message to the notification server and > atlas server reads these messages and registers the entities. Follow these > instructions in your hive set-up to add hive hook for Atlas: > Set-up atlas hook in hive-site.xml of your hive configuration: > <property> > <name>hive.exec.post.hooks</name> > <value>org.apache.atlas.hive.hook.HiveHook</value> > </property> > > > <property> > <name>atlas.cluster.name</name> > <value>primary</value> > </property> > > > Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in > hive-env.sh of your hive configuration > Copy <atlas-conf>/atlas-application.properties to the hive conf > directory. > > > The following properties in <atlas-conf>/atlas-application.properties > control the thread pool and notification details: > atlas.hook.hive.synchronous - boolean, true to run the hook > synchronously. default false. Recommended to be set to false to avoid > delays in hive query completion. > atlas.hook.hive.numRetries - number of retries for notification > failure. default 3 > atlas.hook.hive.minThreads - core number of threads. default 5 > atlas.hook.hive.maxThreads - maximum number of threads. default 5 > atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10 > atlas.hook.hive.queueSize - queue size for the threadpool. default > 10000