Re[2]: Kafka behavior when consuming a topic which doesn't exist?

[M. Manna]

Please ensure that you set the desired minimum partition and replication 
settings in the server.properties file. These are the ones which will be 
used for auto creation of topics.


KR,

-- Original Message --
From: "Ali Akhtar" <ali.rac...@gmail.com>
To: users@kafka.apache.org
Sent: 7/8/2017 4:27:55 PM
Subject: Re: Kafka behavior when consuming a topic which doesn't exist?

Oh gotcha, thanks. So a topic will be created if topic creation is 
enabled.


On Sat, Jul 8, 2017 at 8:14 PM, M. Manna <manme...@gmail.com> wrote:


Please check my previous email.

On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar <ali.rac...@gmail.com> 
wrote:


> What happens if auto creation is enabled but the topic doesn't 
exist.
> Consumers subscribe to that topic which doesn't exist. Then messages 
are

> posted to that topic.
>
> Will the consumer receive those messages in this scenario?
>
>
>
> On 8 Jul 2017 4:38 a.m., "M. Manna" <manme...@gmail.com> wrote:
>
> That depends.
>
> If auto creation of non-existent topic enabled (check docs), then it 
will
> simple use the minimum partiotion and replication settings defined 
in

> broker config to create a topic. If auto creation is disabled, your
> consumer group won't do anything.
>
> With auto creation enable - It's the same if you publish to a topic 
that
> doesn't yet exist. But that means topic will get created first, and 
then
> the messages are sent on the bus. If you subscribe to that topic 
later

you
> will get messages. Check below:
>
> http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
> a-topic-that-does-not-exist
>
>
>
> On 7 Jul 2017 9:46 pm, "Ali Akhtar" <ali.rac...@gmail.com> wrote:
>
> > Sometimes I see warnings in my logs if i create a consumer for a 
topic

> > which doesn't exist. Such as:
> >
> > org.apache.kafka.clients.NetworkClient  - Error while fetching
metadata
> > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
> >
> > If later messages are posted to that topic (which will create it), 
will

> my
> > consumer receive those messages?
> >
>

Re: Kafka behavior when consuming a topic which doesn't exist?

[Ali Akhtar]

Oh gotcha, thanks. So a topic will be created if topic creation is enabled.

On Sat, Jul 8, 2017 at 8:14 PM, M. Manna  wrote:

> Please check my previous email.
>
> On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar  wrote:
>
> > What happens if auto creation is enabled but the topic doesn't exist.
> > Consumers subscribe to that topic which doesn't exist. Then messages are
> > posted to that topic.
> >
> > Will the consumer receive those messages in this scenario?
> >
> >
> >
> > On 8 Jul 2017 4:38 a.m., "M. Manna"  wrote:
> >
> > That depends.
> >
> > If auto creation of non-existent topic enabled (check docs), then it will
> > simple use the minimum partiotion and replication settings defined in
> > broker config to create a topic. If auto creation is disabled, your
> > consumer group won't do anything.
> >
> > With auto creation enable - It's the same if you publish to a topic that
> > doesn't yet exist. But that means topic will get created first, and then
> > the messages are sent on the bus. If you subscribe to that topic later
> you
> > will get messages. Check below:
> >
> > http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
> > a-topic-that-does-not-exist
> >
> >
> >
> > On 7 Jul 2017 9:46 pm, "Ali Akhtar"  wrote:
> >
> > > Sometimes I see warnings in my logs if i create a consumer for a topic
> > > which doesn't exist. Such as:
> > >
> > > org.apache.kafka.clients.NetworkClient  - Error while fetching
> metadata
> > > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
> > >
> > > If later messages are posted to that topic (which will create it), will
> > my
> > > consumer receive those messages?
> > >
> >
>

Re: Kafka behavior when consuming a topic which doesn't exist?

[M. Manna]

Please check my previous email.

On Sat, 8 Jul 2017 at 2:32 am, Ali Akhtar  wrote:

> What happens if auto creation is enabled but the topic doesn't exist.
> Consumers subscribe to that topic which doesn't exist. Then messages are
> posted to that topic.
>
> Will the consumer receive those messages in this scenario?
>
>
>
> On 8 Jul 2017 4:38 a.m., "M. Manna"  wrote:
>
> That depends.
>
> If auto creation of non-existent topic enabled (check docs), then it will
> simple use the minimum partiotion and replication settings defined in
> broker config to create a topic. If auto creation is disabled, your
> consumer group won't do anything.
>
> With auto creation enable - It's the same if you publish to a topic that
> doesn't yet exist. But that means topic will get created first, and then
> the messages are sent on the bus. If you subscribe to that topic later you
> will get messages. Check below:
>
> http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
> a-topic-that-does-not-exist
>
>
>
> On 7 Jul 2017 9:46 pm, "Ali Akhtar"  wrote:
>
> > Sometimes I see warnings in my logs if i create a consumer for a topic
> > which doesn't exist. Such as:
> >
> > org.apache.kafka.clients.NetworkClient  - Error while fetching metadata
> > with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
> >
> > If later messages are posted to that topic (which will create it), will
> my
> > consumer receive those messages?
> >
>

Re: Kafka behavior when consuming a topic which doesn't exist?

[Ali Akhtar]

What happens if auto creation is enabled but the topic doesn't exist.
Consumers subscribe to that topic which doesn't exist. Then messages are
posted to that topic.

Will the consumer receive those messages in this scenario?



On 8 Jul 2017 4:38 a.m., "M. Manna"  wrote:

That depends.

If auto creation of non-existent topic enabled (check docs), then it will
simple use the minimum partiotion and replication settings defined in
broker config to create a topic. If auto creation is disabled, your
consumer group won't do anything.

With auto creation enable - It's the same if you publish to a topic that
doesn't yet exist. But that means topic will get created first, and then
the messages are sent on the bus. If you subscribe to that topic later you
will get messages. Check below:

http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-
a-topic-that-does-not-exist



On 7 Jul 2017 9:46 pm, "Ali Akhtar"  wrote:

> Sometimes I see warnings in my logs if i create a consumer for a topic
> which doesn't exist. Such as:
>
> org.apache.kafka.clients.NetworkClient  - Error while fetching metadata
> with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
>
> If later messages are posted to that topic (which will create it), will my
> consumer receive those messages?
>

Re: Kafka behavior when consuming a topic which doesn't exist?

[M. Manna]

That depends.

If auto creation of non-existent topic enabled (check docs), then it will
simple use the minimum partiotion and replication settings defined in
broker config to create a topic. If auto creation is disabled, your
consumer group won't do anything.

With auto creation enable - It's the same if you publish to a topic that
doesn't yet exist. But that means topic will get created first, and then
the messages are sent on the bus. If you subscribe to that topic later you
will get messages. Check below:

http://grokbase.com/t/kafka/users/1648kbr04c/subscribe-on-a-topic-that-does-not-exist



On 7 Jul 2017 9:46 pm, "Ali Akhtar"  wrote:

> Sometimes I see warnings in my logs if i create a consumer for a topic
> which doesn't exist. Such as:
>
> org.apache.kafka.clients.NetworkClient  - Error while fetching metadata
> with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}
>
> If later messages are posted to that topic (which will create it), will my
> consumer receive those messages?
>

Kafka behavior when consuming a topic which doesn't exist?

[Ali Akhtar]

Sometimes I see warnings in my logs if i create a consumer for a topic
which doesn't exist. Such as:

org.apache.kafka.clients.NetworkClient  - Error while fetching metadata
with correlation id 1 : {example_topic=LEADER_NOT_AVAILABLE}

If later messages are posted to that topic (which will create it), will my
consumer receive those messages?