Hi Shwetha, much appreciated. I missed /etc/atlas/conf in the class path. Now
the hook works. Herman.
> On Jan 21, 2016, at 9:30 AM, Shwetha Shivalingamurthy
> <sshivalingamur...@hortonworks.com> wrote:
>
> Hive-env.sh should contain
> export
> HADOOP_CLASSPATH=/etc/atlas/conf:/usr/hdp/current/atlas-server/hook/hive:${
> HADOOP_CLASSPATH}
>
>
> So, /etc/atlas/conf/ should contain atlas’ application.properties
>
>
> Regards,
> Shwetha
>
>
>
>
>
>
> On 21/01/16 7:46 pm, "Herman Yu" <herman...@teeupdata.com> wrote:
>
>> Thanks Shwetha. that’s the reason, the hook is getting application and
>> client configurations from jar files under hook/hive. I must be missing
>> some configuration steps? is it supposed to get those two config files
>> from Hive configuration folder?
>>
>> Thanks
>> Herman.
>>
>>
>>> On Jan 20, 2016, at 11:27 PM, Shwetha Shivalingamurthy
>>> <sshivalingamur...@hortonworks.com> wrote:
>>>
>>> Check if the atlas conf is picked up correctly. You can enable DEBUG
>>> logs
>>> for atlas by modifying the hive’s log4j.xml - add log level for package
>>> org.apache.atlas. DEBUG log level will print the location of atlas’
>>> application.properties that’s picked up
>>>
>>> Regards,
>>> Shwetha
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 21/01/16 12:41 am, "Herman Yu" <herman...@teeupdata.com> wrote:
>>>
>>>> Hi Shewtha,
>>>>
>>>> Thanks for the link. I followed the link, but am still getting the
>>>> following error in hiveserver2.log. It seems to me that atlasclient is
>>>> refused when trying to connect to atlas server via http. I already
>>>> have
>>>> atlas.http.authentication.enabled=false set to false in both
>>>> application
>>>> and client properties files.
>>>>
>>>>
>>>> any other places I need to check?
>>>>
>>>> thanks
>>>> Herman.
>>>>
>>>>
>>>> 2016-01-20 13:55:22,568 INFO [Atlas Logger 3]: hook.HiveHook
>>>> (HiveHook.java:fireAndForget(192)) - Entered Atlas hook for hook type
>>>> POST_EXEC_HOOK operation CREATETABLE
>>>> 2016-01-20 13:55:22,572 INFO [HiveServer2-Background-Pool:
>>>> Thread-191]:
>>>> log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG
>>>> method=releaseLocks start=1453316122568 end=1453316122572 duration=4
>>>> from=org.apache.hadoop.hive.ql.Driver>
>>>> 2016-01-20 13:55:22,573 INFO [HiveServer2-Background-Pool:
>>>> Thread-191]:
>>>> log.PerfLogger (PerfLogger.java:PerfLogEnd(148)) - </PERFLOG
>>>> method=Driver.run start=1453316122342 end=1453316122573 duration=231
>>>> from=org.apache.hadoop.hive.ql.Driver>
>>>> 2016-01-20 13:55:22,634 INFO [Atlas Logger 3]:
>>>> security.SecureClientUtils
>>>> (SecureClientUtils.java:getClientConnectionHandler(90)) - Real User:
>>>> hive
>>>> (auth:SIMPLE), is from ticket cache? false
>>>> 2016-01-20 13:55:22,635 INFO [Atlas Logger 3]:
>>>> security.SecureClientUtils
>>>> (SecureClientUtils.java:getClientConnectionHandler(93)) - doAsUser: hue
>>>> 2016-01-20 13:55:22,635 INFO [Atlas Logger 2]: hook.HiveHook
>>>> (HiveHook.java:run(182)) - Atlas hook failed
>>>> com.sun.jersey.api.client.ClientHandlerException: java.io.IOException:
>>>> java.net.ConnectException: Connection refused
>>>> at
>>>>
>>>> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(UR
>>>> LC
>>>> onnectionClientHandler.java:149)
>>>> at com.sun.jersey.api.client.Client.handle(Client.java:648)
>>>> at
>>>> com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
>>>> at
>>>> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>>>> at
>>>>
>>>> com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:62
>>>> 3)
>>>> at
>>>> org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:584)
>>>> at
>>>> org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:579)
>>>> at org.apache.atlas.AtlasClient.getType(AtlasClient.java:257)
>>>> at
>>>>
>>>> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerHiveDataModel(H
>>>> iv
>>>> eMetaStoreBridge.java:487)
>>>> at
>>>> org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:197)
>>>> at
>>>> org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:66)
>>>> at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:180)
>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>> a:
>>>> 1145)
>>>> at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>> va
>>>> :615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: java.io.IOException: java.net.ConnectException: Connection
>>>> refused
>>>> at
>>>>
>>>> org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.ja
>>>> va
>>>> :106)
>>>> at
>>>>
>>>> org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.ja
>>>> va
>>>> :98)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> at
>>>>
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio
>>>> n.
>>>> java:1657)
>>>> at
>>>>
>>>> org.apache.atlas.security.SecureClientUtils$1.getHttpURLConnection(Secur
>>>> eC
>>>> lientUtils.java:98)
>>>> at
>>>>
>>>> com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(U
>>>> RL
>>>> ConnectionClientHandler.java:159)
>>>> at
>>>>
>>>> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(UR
>>>> LC
>>>> onnectionClientHandler.java:147)
>>>> ... 16 more
>>>> Caused by: java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at
>>>>
>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
>>>> 33
>>>> 9)
>>>> at
>>>>
>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImp
>>>> l.
>>>> java:200)
>>>> at
>>>>
>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:18
>>>> 2)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:579)
>>>> at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
>>>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
>>>> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
>>>> at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
>>>> at sun.net.www.http.HttpClient.New(HttpClient.java:308)
>>>> at sun.net.www.http.HttpClient.New(HttpClient.java:326)
>>>> at
>>>>
>>>> sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConn
>>>> ec
>>>> tion.java:998)
>>>> at
>>>>
>>>> sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnecti
>>>> on
>>>> .java:934)
>>>> at
>>>>
>>>> sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.ja
>>>> va
>>>> :852)
>>>> at
>>>>
>>>> org.apache.hadoop.security.authentication.client.PseudoAuthenticator.aut
>>>> he
>>>> nticate(PseudoAuthenticator.java:76)
>>>> at
>>>>
>>>> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthentic
>>>> at
>>>> or.authenticate(DelegationTokenAuthenticator.java:128)
>>>> at
>>>>
>>>> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openCo
>>>> nn
>>>> ection(AuthenticatedURL.java:215)
>>>> at
>>>>
>>>> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthentic
>>>> at
>>>> edURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
>>>> at
>>>>
>>>> org.apache.atlas.security.SecureClientUtils$1$1.run(SecureClientUtils.ja
>>>> va
>>>> :102)
>>>> ... 23 more
>>>>
>>>>
>>>>> On Jan 20, 2016, at 2:35 AM, Shwetha Shivalingamurthy
>>>>> <sshivalingamur...@hortonworks.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/Ambari-2.2.0.0
>>>>> /b
>>>>> k_
>>>>> ambari_reference_guide/content/ch_integrating_atlas.html has the
>>>>> instructions on setting up 0.6 Atlas with Ambari
>>>>>
>>>>>
>>>>> Regards,
>>>>> Shwetha
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 20/01/16 2:26 am, "Herman Yu" <herman...@teeupdata.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> I am having some problems with installing and configuring v0.6 on HDP
>>>>>> 2.3.2 sandbox.
>>>>>>
>>>>>> HDP 2.3.2 comes with v0.5 pre-configured with Ambari. After compiling
>>>>>> v0.6 source code, and updated the link /usr/hdp/current/atlas-server
>>>>>> to
>>>>>> be associated with the new v0.6 folder. I also made necessary changes
>>>>>> in
>>>>>> amber¹s (e.g. those notification related parameters in
>>>>>> application.properties). Atlas starts fine (even though with some
>>>>>> errors
>>>>>> in log file which are kafka related), however, Hive stopped working
>>>>>> with
>>>>>> the following errors with any operations.
>>>>>>
>>>>>> I suspect this is the Hive hook related, after several tries, I
>>>>>> figured
>>>>>> out that if I update atlas.hook.hive.synchronous from true to false,
>>>>>> Hive
>>>>>> starts working fine however the hook doesn¹t capture any hive queries
>>>>>> to
>>>>>> atlas.
>>>>>>
>>>>>> Did anyone experience the same problem? Also, how does hive know
>>>>>> where
>>>>>> the jar file of the hook is located? in HDP 2.3.2, I don¹t see
>>>>>> HIVE_AUX_PATH is appended with <Atlas_home>/bridge/hive, I had to
>>>>>> manually append this in Ambari.
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.HiveSQLException: Error while processing
>>>>>> statement: FAILED: Hive Internal Error:
>>>>>> com.sun.jersey.api.client.ClientHandlerException(java.io.IOException:
>>>>>> java.net.ConnectException: Connection refused)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.operation.Operation.toSQLException(Operati
>>>>>> on
>>>>>> .j
>>>>>> ava:315)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperati
>>>>>> on
>>>>>> .j
>>>>>> ava:156)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOper
>>>>>> at
>>>>>> io
>>>>>> n.java:183)
>>>>>> at
>>>>>>
>>>>>> org.apache.hive.service.cli.operation.Operation.run(Operation.java:257
>>>>>> )
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementIn
>>>>>> te
>>>>>> rn
>>>>>> al(HiveSessionImpl.java:388)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(H
>>>>>> iv
>>>>>> eS
>>>>>> essionImpl.java:369)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.CLIService.executeStatement(CLIService.jav
>>>>>> a:
>>>>>> 26
>>>>>> 1)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(T
>>>>>> hr
>>>>>> if
>>>>>> tCLIService.java:486)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatem
>>>>>> en
>>>>>> t.
>>>>>> getResult(TCLIService.java:1313)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatem
>>>>>> en
>>>>>> t.
>>>>>> getResult(TCLIService.java:1298)
>>>>>> at
>>>>>> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>>>>>> at
>>>>>> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddr
>>>>>> es
>>>>>> sP
>>>>>> rocessor.java:56)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPo
>>>>>> ol
>>>>>> Se
>>>>>> rver.java:285)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
>>>>>> av
>>>>>> a:
>>>>>> 1145)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
>>>>>> ja
>>>>>> va
>>>>>> :615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Caused by: com.sun.jersey.api.client.ClientHandlerException:
>>>>>> java.io.IOException: java.net.ConnectException: Connection refused
>>>>>> at
>>>>>>
>>>>>>
>>>>>> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
>>>>>> UR
>>>>>> LC
>>>>>> onnectionClientHandler.java:149)
>>>>>> at com.sun.jersey.api.client.Client.handle(Client.java:648)
>>>>>> at
>>>>>> com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
>>>>>> at
>>>>>> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> com.sun.jersey.api.client.WebResource$Builder.method(WebResource.java:
>>>>>> 62
>>>>>> 3)
>>>>>> at
>>>>>>
>>>>>> org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:584)
>>>>>> at
>>>>>>
>>>>>> org.apache.atlas.AtlasClient.callAPIWithResource(AtlasClient.java:579)
>>>>>> at org.apache.atlas.AtlasClient.getType(AtlasClient.java:257)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerHiveDataModel
>>>>>> (H
>>>>>> iv
>>>>>> eMetaStoreBridge.java:487)
>>>>>> at
>>>>>> org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:197)
>>>>>> at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:174)
>>>>>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1522)
>>>>>> at
>>>>>> org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
>>>>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
>>>>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1054)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperati
>>>>>> on
>>>>>> .j
>>>>>> ava:154)
>>>>>> ... 15 more
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Herman.
>>>>>>
>>>>>
>>>>
>>>
>>
>
