[
https://issues.apache.org/jira/browse/KAFKA-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418131#comment-16418131
]
Matthias J. Sax commented on KAFKA-6437:
----------------------------------------
[~wojda] Thanks for follow up. With regard to the "RUNNING" status, it seems to
be related to KAFKA-6520.
About this ticket and KAFKA-6720: I agree that it is opinion based if failing
of logging is the right approach. Note, that Kafka Streams inherits its
behavior from KafkaConsumer: it also idles if the input topics don't exist.
Therefore, it might even be required to change the consumer, affecting even
more developers. Maybe [~guozhang] can shed some light, why the consumer is
designed in this way (I am sure, there are good reasons for it).
About adding a new config: might be a solution. However, from our experience we
learned that having too many configs can be confusion for users -- thus, we
tend to be conservative about adding new configs if there is better solution. I
am not saying, we should not introduce a "fail-on-missing-topic" configs, I am
just saying we should discuss it in detail before we make a decision. Do you
think that fixing KAFKA-6520 would be an acceptable alternative solution
instead of throwing an exception? Meaning, introducing an new "IDLE" state if
the input topics are missing? Question would be, how to handle partially
available input topics? (Just putting out ideas here...)
> Streams does not warn about missing input topics, but hangs
> -----------------------------------------------------------
>
> Key: KAFKA-6437
> URL: https://issues.apache.org/jira/browse/KAFKA-6437
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 1.0.0
> Environment: Single client on single node broker
> Reporter: Chris Schwarzfischer
> Assignee: Mariam John
> Priority: Minor
> Labels: newbie
>
> *Case*
> Streams application with two input topics being used for a left join.
> When the left side topic is missing upon starting the streams application, it
> hangs "in the middle" of the topology (at …00009, see below). Only parts of
> the intermediate topics are created (up to …00009)
> When the missing input topic is created, the streams application resumes
> processing.
> {noformat}
> Topology:
> StreamsTask taskId: 2_0
> ProcessorTopology:
> KSTREAM-SOURCE-0000000011:
> topics:
> [mystreams_app-KTABLE-AGGREGATE-STATE-STORE-0000000009-repartition]
> children: [KTABLE-AGGREGATE-0000000012]
> KTABLE-AGGREGATE-0000000012:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-0000000009]
> children: [KTABLE-TOSTREAM-0000000020]
> KTABLE-TOSTREAM-0000000020:
> children: [KSTREAM-SINK-0000000021]
> KSTREAM-SINK-0000000021:
> topic: data_udr_month_customer_aggregration
> KSTREAM-SOURCE-0000000017:
> topics:
> [mystreams_app-KSTREAM-MAP-0000000014-repartition]
> children: [KSTREAM-LEFTJOIN-0000000018]
> KSTREAM-LEFTJOIN-0000000018:
> states:
> [KTABLE-AGGREGATE-STATE-STORE-0000000009]
> children: [KSTREAM-SINK-0000000019]
> KSTREAM-SINK-0000000019:
> topic: data_UDR_joined
> Partitions [mystreams_app-KSTREAM-MAP-0000000014-repartition-0,
> mystreams_app-KTABLE-AGGREGATE-STATE-STORE-0000000009-repartition-0]
> {noformat}
> *Why this matters*
> The applications does quite a lot of preprocessing before joining with the
> missing input topic. This preprocessing won't happen without the topic,
> creating a huge backlog of data.
> *Fix*
> Issue an `warn` or `error` level message at start to inform about the missing
> topic and it's consequences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
