No worries. Glad it helped. On 27-Dec-2016 21:18, "Russell Anderson" <r...@us.ibm.com> wrote:
Hi Memanth Yamijala, The magic of your thoughts !!! So I did as you suggested but then I noticed a couple of things that hadn't seemed important at the time : 1) In my Hadoop there were multiple entries for PRE, and FAILURE for Hive Hook. These were left as default - so i changed these to match the one recommended in the documentation. 2) I re-copied the libraries i had built where my AUX_PATH was pointing to. 3) I changed the permissions to match all the other jars in the hive/lib directory - which were 'root'. 4) I then carefully examined the var/log/hive/hiveserver2.log for Hive Hook messages I now have the Hive Hook working as desired !!! Thank you for getting me to look and think about things....!!! Russ. [image: Inactive hide details for Hemanth Yamijala ---12/26/2016 08:11:29 PM---Hi, A few questions to help debug:]Hemanth Yamijala ---12/26/2016 08:11:29 PM---Hi, A few questions to help debug: From: Hemanth Yamijala <yhema...@gmail.com> To: dev@atlas.incubator.apache.org Date: 12/26/2016 08:11 PM Subject: Re: Hive Hook - what is missing ? ------------------------------ Hi, A few questions to help debug: 1) Are you using Hive CLI or Beeline? Best results would be to configure these hooks with HiveServer2 and use Beeline. 2) Could you please check hive logs and look for AtlasHook related messages? Thanks Hemanth On Mon, Dec 26, 2016 at 8:59 PM, Russell Anderson <r...@us.ibm.com> wrote: > > > Hi dev list, > > I have built the Atlas7rc2 - have working the Hive Import, the Dashboard, > and generally things appear to be working as expected. > > However, I have followed the instruction below but cannot seem to get the > Hive Hook to detect table hive creations to be added to the metadata / > lineage. > > Can anyone suggest how to debug what could be missing regardless of > following the instructions below ? Any and all help appreciated !!! > > Regards, > > Russ. > > ============================================================ ================= > > Hive Hook > > > Hive supports listeners on hive command execution using hive hooks. This is > used to add/update/remove entities in Atlas using the model defined in > org.apache.atlas.hive.model.HiveDataModelGenerator. The hook submits the > request to a thread pool executor to avoid blocking the command execution. > The thread submits the entities as message to the notification server and > atlas server reads these messages and registers the entities. Follow these > instructions in your hive set-up to add hive hook for Atlas: > Set-up atlas hook in hive-site.xml of your hive configuration: > <property> > <name>hive.exec.post.hooks</name> > <value>org.apache.atlas.hive.hook.HiveHook</value> > </property> > > > <property> > <name>atlas.cluster.name</name> > <value>primary</value> > </property> > > > Add 'export HIVE_AUX_JARS_PATH=<atlas package>/hook/hive' in > hive-env.sh of your hive configuration > Copy <atlas-conf>/atlas-application.properties to the hive conf > directory. > > > The following properties in <atlas-conf>/atlas-application.properties > control the thread pool and notification details: > atlas.hook.hive.synchronous - boolean, true to run the hook > synchronously. default false. Recommended to be set to false to avoid > delays in hive query completion. > atlas.hook.hive.numRetries - number of retries for notification > failure. default 3 > atlas.hook.hive.minThreads - core number of threads. default 5 > atlas.hook.hive.maxThreads - maximum number of threads. default 5 > atlas.hook.hive.keepAliveTime - keep alive time in msecs. default 10 > atlas.hook.hive.queueSize - queue size for the threadpool. default > 10000
