[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[ https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507236#comment-15507236 ] Ayub Khan commented on ATLAS-1145: -- Sure, I can verify this as part of my testing and update this thread. > Storm hook with kafka topic and hive bolt picks wrong hive table name > -- > > Key: ATLAS-1145 > URL: https://issues.apache.org/jira/browse/ATLAS-1145 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Sarath Subramanian > > Storm hook with kafka topic and hive bolt picks kafka topic name as hive > table name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[ https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507157#comment-15507157 ] Sarath Subramanian commented on ATLAS-1145: --- [~ayubkhan] can you confirm this, since you are writing an automation script to create these storm topologies and verify them in atlas. > Storm hook with kafka topic and hive bolt picks wrong hive table name > -- > > Key: ATLAS-1145 > URL: https://issues.apache.org/jira/browse/ATLAS-1145 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Sarath Subramanian > > Storm hook with kafka topic and hive bolt picks kafka topic name as hive > table name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507148#comment-15507148
]
Sarath Subramanian commented on ATLAS-1145:
---
Thanks for the samples [~shwethags], [~yhemanth] and [~ayubkhan]. I was able to
create a storm topology with kafka spout/hive bolt using the samples from
Ayub's link. I'm not able to reproduce this issue. Attaching the entity
definition from REST, the kafka topic entity in atlas gets assigned the right
topic name and not the hive table name. Attaching the kafka topic entity
definition created by the storm topology.
{
"requestId": "qtp1448525331-188 - f7904223-bac3-489b-b593-ffbb3100debd",
"definition": {
"jsonClass":
"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "5669f2ba-8397-44c5-bf0d-0544533979a4",
"version": 0,
"typeName": "kafka_topic",
"state": "ACTIVE"
},
"typeName": "kafka_topic",
"values": {
"name": "kafka_topic_7",
"description": null,
"uri": "localhost:2181",
"qualifiedName": "kafka_topic_7@ErieCluster",
"topic": "kafka_topic_7",
"owner": "storm"
},
"traitNames": [],
"traits": {}
}
}
> Storm hook with kafka topic and hive bolt picks wrong hive table name
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
> Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive
> table name
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[ https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505762#comment-15505762 ] Ayub Khan commented on ATLAS-1145: -- This has some hive-kafka examples: https://github.com/sakserv/storm-topology-examples > Storm hook with kafka topic and hive bolt picks wrong hive table name > -- > > Key: ATLAS-1145 > URL: https://issues.apache.org/jira/browse/ATLAS-1145 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Sarath Subramanian > > Storm hook with kafka topic and hive bolt picks kafka topic name as hive > table name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[
https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505596#comment-15505596
]
Hemanth Yamijala commented on ATLAS-1145:
-
[~sarath.ku...@gmail.com], Some time back I had a working version of Kafka to
Hive storm topology capturing lineage into Atlas etc. Code is here:
https://github.com/yhemanth/storm-samples/blob/master/src/main/java/com/dsinpractice/storm/samples/WordCountTopology.java
However, we shifted focus from Hive to HDFS for later testing.
The main issue AFAIK is to setup Hive for streaming accurately. It took some
time for me to figure that out. Wish I had documented the steps more clearly
:-(
These are properties I have in my hive-site which *I think* were used to get
these to work. Please use these in case they are of help.
{code}
hive.support.concurrency
true
hive.enforce.bucketing
true
hive.exec.dynamic.partition.mode
nonstrict
hive.compactor.initiator.on
true
hive.txn.manager
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
hive.compactor.initiator.on
true
hive.compactor.worker.threads
1
{code}
> Storm hook with kafka topic and hive bolt picks wrong hive table name
> --
>
> Key: ATLAS-1145
> URL: https://issues.apache.org/jira/browse/ATLAS-1145
> Project: Atlas
> Issue Type: Bug
>Reporter: Shwetha G S
>Assignee: Sarath Subramanian
>
> Storm hook with kafka topic and hive bolt picks kafka topic name as hive
> table name
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[ https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505532#comment-15505532 ] Shwetha G S commented on ATLAS-1145: Hemanth has one - https://github.com/yhemanth/storm-samples > Storm hook with kafka topic and hive bolt picks wrong hive table name > -- > > Key: ATLAS-1145 > URL: https://issues.apache.org/jira/browse/ATLAS-1145 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Sarath Subramanian > > Storm hook with kafka topic and hive bolt picks kafka topic name as hive > table name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[ https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505236#comment-15505236 ] Sarath Subramanian commented on ATLAS-1145: --- [~shwethags] Do you have a sample script/command to create a storm topology with kafka spout and hive bolt? I'm using - storm jar /usr/hdp/current/storm-client/contrib/storm-starter/storm-starter-topologies-*.jar storm.starter.StatefulTopology test_topology_1 but it doesn't have any hive bolt or kafka spout. > Storm hook with kafka topic and hive bolt picks wrong hive table name > -- > > Key: ATLAS-1145 > URL: https://issues.apache.org/jira/browse/ATLAS-1145 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Sarath Subramanian > > Storm hook with kafka topic and hive bolt picks kafka topic name as hive > table name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ATLAS-1145) Storm hook with kafka topic and hive bolt picks wrong hive table name
[ https://issues.apache.org/jira/browse/ATLAS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494879#comment-15494879 ] Sarath Subramanian commented on ATLAS-1145: --- Could you specify more info and reproduction steps? > Storm hook with kafka topic and hive bolt picks wrong hive table name > -- > > Key: ATLAS-1145 > URL: https://issues.apache.org/jira/browse/ATLAS-1145 > Project: Atlas > Issue Type: Bug >Reporter: Shwetha G S >Assignee: Sarath Subramanian > > Storm hook with kafka topic and hive bolt picks kafka topic name as hive > table name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
