[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221054#comment-16221054 ] Ramesh Mani commented on ATLAS-1805: commit link https://github.com/apache/atlas/commit/b05b855622166bf98ef6a18198ae94a9414bf21d > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas > --- > > Key: ATLAS-1805 > URL: https://issues.apache.org/jira/browse/ATLAS-1805 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Reporter: Ramesh Mani >Assignee: Ramesh Mani > Attachments: > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch, > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names_002.patch, > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names_003.patch > > > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[
https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145087#comment-16145087
]
Nixon Rodrigues commented on ATLAS-1805:
[~cuijunheng]
Enable HBase Hook for Atlas
* cd hbase/bin
ADD HBase Atlas hook to the HBASE classpath.
* cat hbase
{code}
#!/bin/bash
BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default}
[ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && .
${BIGTOP_DEFAULTS_DIR}/hbase
if [ -d "atlas/hook/hbase" ]; then
if [ -z "${HBASE_CLASSPATH}" ]; then
export HBASE_CLASSPATH=atlas/hook/hbase/*
else
export HBASE_CLASSPATH=${HBASE_CLASSPATH}:atlas/hook/hbase/*
fi
fi
export ATLAS_HOME=${ATLAS_HOME:-~/atlas}
export HBASE_HOME=${HBASE_HOME:-~/hbase}
export HADOOP_HOME=${HADOOP_HOME:-~/hadoop}
export HADOOP_CONF=${HADOOP_HOME}/conf
export ZOOKEEPER_HOME=${ZOOKEEPER_HOME:~/zookeeper}
export
HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$HBASE_CLASSPATH
export HBASE_OPTS="-Dhdp.version=${HDP_VERSION} ${HBASE_OPTS}"
exec hbase/bin/hbase.distro "$@"
{code}
3) create atlas-application.properties in hbase/conf folder
* cat atlas-application.properties
{code}
atlas.authentication.method.kerberos=false
atlas.cluster.name=cl1
atlas.hook.hbase.keepAliveTime=10
atlas.hook.hbase.maxThreads=5
atlas.hook.hbase.minThreads=5
atlas.hook.hbase.numRetries=3
atlas.hook.hbase.queueSize=1000
atlas.hook.hbase.synchronous=false
atlas.jaas.KafkaClient.loginModuleControlFlag=required
atlas.jaas.KafkaClient.loginModuleName=com.sun.security.auth.module.Krb5LoginModule
atlas.jaas.KafkaClient.option.keyTab=/etc/security/keytabs/hbase.service.keytab
atlas.jaas.KafkaClient.option.principal=hbase/_h...@example.com
atlas.jaas.KafkaClient.option.serviceName=cstm-kafka
atlas.jaas.KafkaClient.option.storeKey=True
atlas.jaas.KafkaClient.option.useKeyTab=True
atlas.kafka.bootstrap.servers=:6667
atlas.kafka.hook.group.id=atlas
atlas.kafka.sasl.kerberos.service.name=cstm-kafka
atlas.kafka.security.protocol=PLAINTEXTSASL
atlas.kafka.zookeeper.connect=:2181
atlas.kafka.zookeeper.connection.timeout.ms=3
atlas.kafka.zookeeper.session.timeout.ms=6
atlas.kafka.zookeeper.sync.time.ms=20
atlas.notification.create.topics=True
atlas.notification.replicas=1
atlas.notification.topics=ATLAS_HOOK,ATLAS_ENTITIES
atlas.rest.address=http://:21000
{code}
4) Adding the Atlas Hook to Hbase.
In Hbase Site.xml
{code}
hbase.coprocessor.master.classes=org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor
{code}
This is in addition to what is there in the property.
> Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to
> Atlas
> ---
>
> Key: ATLAS-1805
> URL: https://issues.apache.org/jira/browse/ATLAS-1805
> Project: Atlas
> Issue Type: Bug
> Components: atlas-intg
>Reporter: Ramesh Mani
>Assignee: Ramesh Mani
> Attachments:
> 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch
>
>
> Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to
> Atlas
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16143325#comment-16143325 ] Junheng Cui commented on ATLAS-1805: I compiled it and got the jar file. So could you please tell me how to config and use it? Thanks a lot. > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas > --- > > Key: ATLAS-1805 > URL: https://issues.apache.org/jira/browse/ATLAS-1805 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Reporter: Ramesh Mani >Assignee: Ramesh Mani > Attachments: > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch > > > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139894#comment-16139894 ] Junheng Cui commented on ATLAS-1805: [~nixonrodrigues] https://github.com/apache/atlas don't contain this patch. How do you get this patch java files and compile it? > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas > --- > > Key: ATLAS-1805 > URL: https://issues.apache.org/jira/browse/ATLAS-1805 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Reporter: Ramesh Mani >Assignee: Ramesh Mani > Attachments: > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch > > > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139874#comment-16139874 ] Nixon Rodrigues commented on ATLAS-1805: [~cuijunheng], The patch is attached to this Jira itself. [^0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch] > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas > --- > > Key: ATLAS-1805 > URL: https://issues.apache.org/jira/browse/ATLAS-1805 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Reporter: Ramesh Mani >Assignee: Ramesh Mani > Attachments: > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch > > > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[
https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138402#comment-16138402
]
Nixon Rodrigues commented on ATLAS-1805:
[~rmani] Thanks for the patch.
After adding the hooks jar and configs in hbase end, I was able to verify basic
functionality for create table in hbase to be working and Entities for
Hbase-table and Hbase column family are getting reflected on Atlas UI.
found one warning in atlas-application.log :
{code}
2017-08-23 07:05:13,897 WARN - [NotificationHookConsumer thread-0:] ~ ignored
unknown attribute hbase_table.qualifier (AtlasStructFormatConverter:161)
{code}
*+1* for this patch from my end.
> Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to
> Atlas
> ---
>
> Key: ATLAS-1805
> URL: https://issues.apache.org/jira/browse/ATLAS-1805
> Project: Atlas
> Issue Type: Bug
> Components: atlas-intg
>Reporter: Ramesh Mani
>Assignee: Ramesh Mani
> Attachments:
> 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch
>
>
> Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to
> Atlas
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16136381#comment-16136381 ] Ramesh Mani commented on ATLAS-1805: [~nixonrodrigues] Attached the rebased patch. Thanks. > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas > --- > > Key: ATLAS-1805 > URL: https://issues.apache.org/jira/browse/ATLAS-1805 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Reporter: Ramesh Mani >Assignee: Ramesh Mani > Attachments: > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch > > > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ATLAS-1805) Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to Atlas
[ https://issues.apache.org/jira/browse/ATLAS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135381#comment-16135381 ] Nixon Rodrigues commented on ATLAS-1805: [~rmani], The repo for Atlas has change from incubating to main, can you please rebase the patch and use this repo https://github.com/apache/atlas > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas > --- > > Key: ATLAS-1805 > URL: https://issues.apache.org/jira/browse/ATLAS-1805 > Project: Atlas > Issue Type: Bug > Components: atlas-intg >Reporter: Ramesh Mani >Assignee: Ramesh Mani > Attachments: > 0001-ATLAS-1805-Provide-an-Atlas-hook-to-send-Hbase-Names.patch > > > Provide an Atlas hook to send Hbase Namespace/Table/column family metadata to > Atlas -- This message was sent by Atlassian JIRA (v6.4.14#64029)
