[
https://issues.apache.org/jira/browse/STORM-2506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Srishty Agrawal updated STORM-2506:
-----------------------------------
Description:
On using storm-kafka KafkaSpout, the generated worker logs have information
about the task index rather than task ID while printing the assigned kafka
partitions. It is difficult to fetch the executor ID reading a particular kafka
partition if only task index is present in the logs.
[Example from generated worker.log while using storm-kafka module's
KafkaSpout|https://gist.github.com/srishtyagrawal/e068bd934094d685a08ece5924e3e71f].
There were a total of 16 tasks and 2 executors assigned for this topology.
The executor IDs as indicated by storm-UI were (5 12) and (13 20).
This problem has been eliminated in the KafkaSpout from storm-kafka-client
module. Logs generated with the new KafkaSpout print the mapping between
executor ID running KafkaSpout and the kafka parititions. The task ID is still
not printed in the log line which prints this mapping.
[Example from worker.log while using storm-kafka module's
KafkaSpout|https://gist.github.com/srishtyagrawal/04939d3ac9931ead24b89405a5f8be6b].
There were a total of 16 tasks and 2 executors for this topology as well.
was:
On using storm-kafka KafkaSpout, the generated worker logs have information
about the task index rather than task ID while printing the assigned kafka
partitions. It is difficult to fetch the executor ID reading a particular kafka
partition if only Task Index is present in the logs.
[Example from generated worker.log while using storm-kafka module's
KafkaSpout|https://gist.github.com/srishtyagrawal/e068bd934094d685a08ece5924e3e71f].
There were a total of 16 tasks and 2 executors assigned for this topology.
The executor IDs as indicated by storm-UI were (5 12) and (13 20).
This problem has been eliminated in the KafkaSpout from storm-kafka-client
module. Logs generated with the new KafkaSpout print the mapping between
executor running KafkaSpout and the kafka parititions it is assigned to read.
The task ID is still not printed in the log line which prints this mapping.
[Example from worker.log while using storm-kafka module's
KafkaSpout|https://gist.github.com/srishtyagrawal/04939d3ac9931ead24b89405a5f8be6b].
There were a total of 16 tasks and 2 executors for the below run:
> Make Kafka Spout log its assigned partition
> -------------------------------------------
>
> Key: STORM-2506
> URL: https://issues.apache.org/jira/browse/STORM-2506
> Project: Apache Storm
> Issue Type: Task
> Components: storm-kafka, storm-kafka-client
> Reporter: Srishty Agrawal
> Priority: Minor
> Labels: information, log
>
> On using storm-kafka KafkaSpout, the generated worker logs have information
> about the task index rather than task ID while printing the assigned kafka
> partitions. It is difficult to fetch the executor ID reading a particular
> kafka partition if only task index is present in the logs.
> [Example from generated worker.log while using storm-kafka module's
> KafkaSpout|https://gist.github.com/srishtyagrawal/e068bd934094d685a08ece5924e3e71f].
> There were a total of 16 tasks and 2 executors assigned for this topology.
> The executor IDs as indicated by storm-UI were (5 12) and (13 20).
> This problem has been eliminated in the KafkaSpout from storm-kafka-client
> module. Logs generated with the new KafkaSpout print the mapping between
> executor ID running KafkaSpout and the kafka parititions. The task ID is
> still not printed in the log line which prints this mapping.
> [Example from worker.log while using storm-kafka module's
> KafkaSpout|https://gist.github.com/srishtyagrawal/04939d3ac9931ead24b89405a5f8be6b].
> There were a total of 16 tasks and 2 executors for this topology as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
