delete topic ?

[Shlomi Hazan]

Hi,

Doing some evaluation testing, and accidently create a queue with wrong
replication factor.

Trying to delete as in:

kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete
--topic replicated-topic

Yeilded:

Command must include exactly one action: --list, --describe, --create or
-alter

Event though this page (https://kafka.apache.org/documentation.html) says:

 

And finally deleting a topic:

 > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
my_topic_name

WARNING: Delete topic functionality is beta in 0.8.1. Please report any bugs
that you encounter on the <mailto:%20us...@kafka.apache.org> mailing list or
<https://issues.apache.org/jira/browse/KAFKA> JIRA.

Kafka does not currently support reducing the number of partitions for a
topic or changing the replication factor.

What should I do?

Shlomi

Delete topic

[Armando Martinez Briones]

Hi.

What is the best way to delete a topic into production environment?

-- 
[image: Tralix][image: 1]José Armando Martínez Briones
*Arquitecto de software*
Tralix México
Tel: +52 442 161 1002  ext. 2920
www.tralix.com.mx

*TRALIX MÉXICO S. DE R.L.
DE C.V* como responsable del tratamiento de sus datos personales, hace de
su conocimiento que la información obtenida por este medio es tratada de
forma estrictamente confidencial por lo que recabaremos y trataremos de sus
datos según los lineamientos de nuestro Aviso de Privacidad
. Para
conocer la versión completa podrá hacerlo a través de la página Aviso de
privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
oposición al tratamiento de sus Datos Personales, se entiende que usted
conoció la versión completa y acepta los términos del mismo, siempre de
conformidad a la Ley Federal de Protección de Datos Personales en Posesión
de los Particulares.

Delete Topic - BadVersionException

[Drew Goya]

Just tried my first topic delete today and it looks like something went
wrong on the controller.  I issued the command on a test topic and shortly
after that a describe looked like:

Topic:TimeoutQueueTest PartitionCount:256 ReplicationFactor:3 Configs:
Topic: TimeoutQueueTest Partition: 0 Leader: -1 Replicas: 9,14,15 Isr:
Topic: TimeoutQueueTest Partition: 1 Leader: -1 Replicas: 10,15,1 Isr:
Topic: TimeoutQueueTest Partition: 2 Leader: -1 Replicas: 11,1,2 Isr:
Topic: TimeoutQueueTest Partition: 3 Leader: -1 Replicas: 12,2,3 Isr:
Topic: TimeoutQueueTest Partition: 4 Leader: -1 Replicas: 13,3,4 Isr:
Topic: TimeoutQueueTest Partition: 5 Leader: -1 Replicas: 14,4,5 Isr:

It stayed that way for quite a while so I hit zookeeper and went looking
for who was the controller, I found these in that brokers logs:

[2014-04-24 16:27:42,498] ERROR Conditional update of path
/brokers/topics/TimeoutQueueTest/partitions/170/state with data
{"controller_epoch":18,"leader":2,"version":1,"leader_epoch":14,"isr":[2,14]}
and expected version 30 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/TimeoutQueueTest/partitions/170/state
(kafka.utils.ZkUtils$)
[2014-04-24 16:27:42,504] ERROR Conditional update of path
/brokers/topics/TimeoutQueueTest/partitions/113/state with data
{"controller_epoch":18,"leader":2,"version":1,"leader_epoch":4,"isr":[2,15,14]}
and expected version 17 failed due to
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /brokers/topics/TimeoutQueueTest/partitions/113/state
(kafka.utils.ZkUtils$)

Any ideas?

Re: delete topic ?

[Neha Narkhede]

Kafka allows increasing the replication factor of a topic. You can read
about it here
<http://kafka.apache.org/081/documentation.html#basic_ops_increase_replication_factor>.
We do not support reducing the number of partitions, so you either have to
create a new topic or delete the existing one. We fixed a number of bugs in
delete topic but those fixes are on trunk. You can give it a spin.


On Wed, Jun 18, 2014 at 4:06 AM, Shlomi Hazan  wrote:

> Hi,
>
> Doing some evaluation testing, and accidently create a queue with wrong
> replication factor.
>
> Trying to delete as in:
>
> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete
> --topic replicated-topic
>
> Yeilded:
>
> Command must include exactly one action: --list, --describe, --create or
> -alter
>
> Event though this page (https://kafka.apache.org/documentation.html) says:
>
>
>
> And finally deleting a topic:
>
>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> my_topic_name
>
> WARNING: Delete topic functionality is beta in 0.8.1. Please report any
> bugs
> that you encounter on the <mailto:%20us...@kafka.apache.org> mailing list
> or
> <https://issues.apache.org/jira/browse/KAFKA> JIRA.
>
> Kafka does not currently support reducing the number of partitions for a
> topic or changing the replication factor.
>
> What should I do?
>
> Shlomi
>
>

Re: delete topic ?

[Mark Roberts]

When we were in testing phase, we would either create a new topic with the 
correct details or shut the cluster down and hard kill the topic in zookeeper + 
local disk.  In prod we have the cluster configured via configuration 
management and auto create turned off.

The ability to delete a topic in a live, running kafka cluster is tricky, and 
the implementations of it have been subtly incorrect (and therefore dangerous). 
I know that there is work happening around that, but haven't kept up with the 
status of it.  Maybe in 8.2? It sounds conceptually simpler to implement with 
the new metadata API.

-Mark

> On Jun 18, 2014, at 4:06, "Shlomi Hazan"  wrote:
> 
> Hi,
> 
> Doing some evaluation testing, and accidently create a queue with wrong
> replication factor.
> 
> Trying to delete as in:
> 
> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete
> --topic replicated-topic
> 
> Yeilded:
> 
> Command must include exactly one action: --list, --describe, --create or
> -alter
> 
> Event though this page (https://kafka.apache.org/documentation.html) says:
> 
> 
> 
> And finally deleting a topic:
> 
>> bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> my_topic_name
> 
> WARNING: Delete topic functionality is beta in 0.8.1. Please report any bugs
> that you encounter on the <mailto:%20us...@kafka.apache.org> mailing list or
> <https://issues.apache.org/jira/browse/KAFKA> JIRA.
> 
> Kafka does not currently support reducing the number of partitions for a
> topic or changing the replication factor.
> 
> What should I do?
> 
> Shlomi
> 

Re: delete topic ?

[hsy...@gmail.com]

I'm using 0.8.1.1
I use DeleteTopicCommand to delete topic
args[0] = "--topic";
args[1] = the topic you want to delete
args[2] = "--zookeeper";
args[3] = kafkaZookeepers;
DeleteTopicCommand.main(args);

You can write your own script to delete the topic, I guess. And I think it
only deletes the entry in zookeeper

Best,
Siyuan



On Wed, Jun 18, 2014 at 9:13 AM, Mark Roberts  wrote:

> When we were in testing phase, we would either create a new topic with the
> correct details or shut the cluster down and hard kill the topic in
> zookeeper + local disk.  In prod we have the cluster configured via
> configuration management and auto create turned off.
>
> The ability to delete a topic in a live, running kafka cluster is tricky,
> and the implementations of it have been subtly incorrect (and therefore
> dangerous). I know that there is work happening around that, but haven't
> kept up with the status of it.  Maybe in 8.2? It sounds conceptually
> simpler to implement with the new metadata API.
>
> -Mark
>
> > On Jun 18, 2014, at 4:06, "Shlomi Hazan"  wrote:
> >
> > Hi,
> >
> > Doing some evaluation testing, and accidently create a queue with wrong
> > replication factor.
> >
> > Trying to delete as in:
> >
> > kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181
> --delete
> > --topic replicated-topic
> >
> > Yeilded:
> >
> > Command must include exactly one action: --list, --describe, --create or
> > -alter
> >
> > Event though this page (https://kafka.apache.org/documentation.html)
> says:
> >
> >
> >
> > And finally deleting a topic:
> >
> >> bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> > my_topic_name
> >
> > WARNING: Delete topic functionality is beta in 0.8.1. Please report any
> bugs
> > that you encounter on the <mailto:%20us...@kafka.apache.org> mailing
> list or
> > <https://issues.apache.org/jira/browse/KAFKA> JIRA.
> >
> > Kafka does not currently support reducing the number of partitions for a
> > topic or changing the replication factor.
> >
> > What should I do?
> >
> > Shlomi
> >
>

Re: delete topic ?

[Timothy Chen]

Yes the existing delete topic command just cleans up the topic entry in zk, but 
not really deleting the topic from the cluster.

I have a patch that enables kafka-topics.sh to delete topic but not sure if 
it's merged to trunk.

Tim

> On Jun 18, 2014, at 1:39 PM, "hsy...@gmail.com"  wrote:
> 
> I'm using 0.8.1.1
> I use DeleteTopicCommand to delete topic
>args[0] = "--topic";
>args[1] = the topic you want to delete
>args[2] = "--zookeeper";
>args[3] = kafkaZookeepers;
>DeleteTopicCommand.main(args);
> 
> You can write your own script to delete the topic, I guess. And I think it
> only deletes the entry in zookeeper
> 
> Best,
> Siyuan
> 
> 
> 
>> On Wed, Jun 18, 2014 at 9:13 AM, Mark Roberts  wrote:
>> 
>> When we were in testing phase, we would either create a new topic with the
>> correct details or shut the cluster down and hard kill the topic in
>> zookeeper + local disk.  In prod we have the cluster configured via
>> configuration management and auto create turned off.
>> 
>> The ability to delete a topic in a live, running kafka cluster is tricky,
>> and the implementations of it have been subtly incorrect (and therefore
>> dangerous). I know that there is work happening around that, but haven't
>> kept up with the status of it.  Maybe in 8.2? It sounds conceptually
>> simpler to implement with the new metadata API.
>> 
>> -Mark
>> 
>>> On Jun 18, 2014, at 4:06, "Shlomi Hazan"  wrote:
>>> 
>>> Hi,
>>> 
>>> Doing some evaluation testing, and accidently create a queue with wrong
>>> replication factor.
>>> 
>>> Trying to delete as in:
>>> 
>>> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181
>> --delete
>>> --topic replicated-topic
>>> 
>>> Yeilded:
>>> 
>>> Command must include exactly one action: --list, --describe, --create or
>>> -alter
>>> 
>>> Event though this page (https://kafka.apache.org/documentation.html)
>> says:
>>> 
>>> 
>>> 
>>> And finally deleting a topic:
>>> 
>>>> bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
>>> my_topic_name
>>> 
>>> WARNING: Delete topic functionality is beta in 0.8.1. Please report any
>> bugs
>>> that you encounter on the <mailto:%20us...@kafka.apache.org> mailing
>> list or
>>> <https://issues.apache.org/jira/browse/KAFKA> JIRA.
>>> 
>>> Kafka does not currently support reducing the number of partitions for a
>>> topic or changing the replication factor.
>>> 
>>> What should I do?
>>> 
>>> Shlomi
>> 

Re: delete topic ?

[Shlomi Hazan]

if the answer is pointing out the 'chroot', as a word, it makes no
difference. the result is the same:

kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete
--topic topic-3

gives the same:

"Command must include exactly one action: --list, --describe, --create or
--alter..."

or should I write something instead of "chroot"?



On Wed, Jun 18, 2014 at 2:06 PM, Shlomi Hazan  wrote:

> Hi,
>
> Doing some evaluation testing, and accidently create a queue with wrong
> replication factor.
>
> Trying to delete as in:
>
> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete
> --topic replicated-topic
>
> Yeilded:
>
> Command must include exactly one action: --list, --describe, --create or
> –alter
>
> Event though this page (https://kafka.apache.org/documentation.html) says:
>
>
>
> And finally deleting a topic:
>
>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic 
> my_topic_name
>
> WARNING: Delete topic functionality is beta in 0.8.1. Please report any
> bugs that you encounter on themailing list <%20us...@kafka.apache.org> or
> JIRA <https://issues.apache.org/jira/browse/KAFKA>.
>
> Kafka does not currently support reducing the number of partitions for a
> topic or changing the replication factor.
>
> What should I do?
>
> Shlomi
>

Re: delete topic ?

[Timothy Chen]

Is this the latest master? I've added the delete option in trunk, but
it's not in any release yet.

We used to have the delete option flag but I believe we removed it
that's why the documentation difference.

Tim

On Wed, Aug 6, 2014 at 10:53 PM, Shlomi Hazan  wrote:
> if the answer is pointing out the 'chroot', as a word, it makes no
> difference. the result is the same:
>
> kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete
> --topic topic-3
>
> gives the same:
>
> "Command must include exactly one action: --list, --describe, --create or
> --alter..."
>
> or should I write something instead of "chroot"?
>
>
>
> On Wed, Jun 18, 2014 at 2:06 PM, Shlomi Hazan  wrote:
>
>> Hi,
>>
>> Doing some evaluation testing, and accidently create a queue with wrong
>> replication factor.
>>
>> Trying to delete as in:
>>
>> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete
>> --topic replicated-topic
>>
>> Yeilded:
>>
>> Command must include exactly one action: --list, --describe, --create or
>> –alter
>>
>> Event though this page (https://kafka.apache.org/documentation.html) says:
>>
>>
>>
>> And finally deleting a topic:
>>
>>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic 
>> my_topic_name
>>
>> WARNING: Delete topic functionality is beta in 0.8.1. Please report any
>> bugs that you encounter on themailing list <%20us...@kafka.apache.org> or
>> JIRA <https://issues.apache.org/jira/browse/KAFKA>.
>>
>> Kafka does not currently support reducing the number of partitions for a
>> topic or changing the replication factor.
>>
>> What should I do?
>>
>> Shlomi
>>

Re: delete topic ?

[Gwen Shapira]

Hi Timothy,

While we are on the subject, few questions/comments (based on the
trunk implementation of delete topic command):

* After deleting a topic, I still see it when listing topics. Is the
expected behavior? Should it disappear after some time?
* When does the actual deletion gets triggered?
* If I try to delete a topic twice I get a pretty confusing exception
(Node exists from zkclient). It will be nice to catch this and say
"Topic is being deleted" or something to this effect.
* Even nicer if list topics command will mark topics as "being deleted".

I'll probably open a separate Jira for the "nice" behavior, but
interested in hearing your thoughts.

Gwen

On Wed, Aug 6, 2014 at 11:01 PM, Timothy Chen  wrote:
> Is this the latest master? I've added the delete option in trunk, but
> it's not in any release yet.
>
> We used to have the delete option flag but I believe we removed it
> that's why the documentation difference.
>
> Tim
>
> On Wed, Aug 6, 2014 at 10:53 PM, Shlomi Hazan  wrote:
>> if the answer is pointing out the 'chroot', as a word, it makes no
>> difference. the result is the same:
>>
>> kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete
>> --topic topic-3
>>
>> gives the same:
>>
>> "Command must include exactly one action: --list, --describe, --create or
>> --alter..."
>>
>> or should I write something instead of "chroot"?
>>
>>
>>
>> On Wed, Jun 18, 2014 at 2:06 PM, Shlomi Hazan  wrote:
>>
>>> Hi,
>>>
>>> Doing some evaluation testing, and accidently create a queue with wrong
>>> replication factor.
>>>
>>> Trying to delete as in:
>>>
>>> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete
>>> --topic replicated-topic
>>>
>>> Yeilded:
>>>
>>> Command must include exactly one action: --list, --describe, --create or
>>> –alter
>>>
>>> Event though this page (https://kafka.apache.org/documentation.html) says:
>>>
>>>
>>>
>>> And finally deleting a topic:
>>>
>>>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic 
>>> my_topic_name
>>>
>>> WARNING: Delete topic functionality is beta in 0.8.1. Please report any
>>> bugs that you encounter on themailing list <%20us...@kafka.apache.org> or
>>> JIRA <https://issues.apache.org/jira/browse/KAFKA>.
>>>
>>> Kafka does not currently support reducing the number of partitions for a
>>> topic or changing the replication factor.
>>>
>>> What should I do?
>>>
>>> Shlomi
>>>

Re: delete topic ?

[Timothy Chen]

Hi Gwen,

That is a very confusing error message for sure, feel free to file a
jira for both the experience cases.

But in general how delete topic works is that it creates a entry in
the delete_topic zk path, and the leader has a delete topic thread
that watches that path and starts the topic deletion once it receives
the message. It then requires rounds of coordination among all the
brokers that has partitions for the topic to delete all the
partitions, then finally delete the topic from zk.

Therefore once the deletion finishes it will also deleted from zk. The
topic command can definitely however join the topic list with the
delete topic list and mark the ones being deleted with a special
status.

Tim

On Wed, Aug 6, 2014 at 11:20 PM, Gwen Shapira  wrote:
> Hi Timothy,
>
> While we are on the subject, few questions/comments (based on the
> trunk implementation of delete topic command):
>
> * After deleting a topic, I still see it when listing topics. Is the
> expected behavior? Should it disappear after some time?
> * When does the actual deletion gets triggered?
> * If I try to delete a topic twice I get a pretty confusing exception
> (Node exists from zkclient). It will be nice to catch this and say
> "Topic is being deleted" or something to this effect.
> * Even nicer if list topics command will mark topics as "being deleted".
>
> I'll probably open a separate Jira for the "nice" behavior, but
> interested in hearing your thoughts.
>
> Gwen
>
> On Wed, Aug 6, 2014 at 11:01 PM, Timothy Chen  wrote:
>> Is this the latest master? I've added the delete option in trunk, but
>> it's not in any release yet.
>>
>> We used to have the delete option flag but I believe we removed it
>> that's why the documentation difference.
>>
>> Tim
>>
>> On Wed, Aug 6, 2014 at 10:53 PM, Shlomi Hazan  wrote:
>>> if the answer is pointing out the 'chroot', as a word, it makes no
>>> difference. the result is the same:
>>>
>>> kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete
>>> --topic topic-3
>>>
>>> gives the same:
>>>
>>> "Command must include exactly one action: --list, --describe, --create or
>>> --alter..."
>>>
>>> or should I write something instead of "chroot"?
>>>
>>>
>>>
>>> On Wed, Jun 18, 2014 at 2:06 PM, Shlomi Hazan  wrote:
>>>
>>>> Hi,
>>>>
>>>> Doing some evaluation testing, and accidently create a queue with wrong
>>>> replication factor.
>>>>
>>>> Trying to delete as in:
>>>>
>>>> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181 --delete
>>>> --topic replicated-topic
>>>>
>>>> Yeilded:
>>>>
>>>> Command must include exactly one action: --list, --describe, --create or
>>>> –alter
>>>>
>>>> Event though this page (https://kafka.apache.org/documentation.html) says:
>>>>
>>>>
>>>>
>>>> And finally deleting a topic:
>>>>
>>>>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic 
>>>> my_topic_name
>>>>
>>>> WARNING: Delete topic functionality is beta in 0.8.1. Please report any
>>>> bugs that you encounter on themailing list <%20us...@kafka.apache.org> or
>>>> JIRA <https://issues.apache.org/jira/browse/KAFKA>.
>>>>
>>>> Kafka does not currently support reducing the number of partitions for a
>>>> topic or changing the replication factor.
>>>>
>>>> What should I do?
>>>>
>>>> Shlomi
>>>>

Re: delete topic ?

[Jason Rosenberg]

Since the deletion stuff is now in trunk, would be compatible to issue the
command from a jar built from trunk, against a running 0.8.1.1 cluster?  Or
does the cluster also have to be running trunk?  (I'm guessing it does :)).

I have some topics I'd like to delete, but don't want to wait for 0.8.2
(but will probably have to, I'm guessing).

Jason


On Thu, Aug 7, 2014 at 2:53 AM, Timothy Chen  wrote:

> Hi Gwen,
>
> That is a very confusing error message for sure, feel free to file a
> jira for both the experience cases.
>
> But in general how delete topic works is that it creates a entry in
> the delete_topic zk path, and the leader has a delete topic thread
> that watches that path and starts the topic deletion once it receives
> the message. It then requires rounds of coordination among all the
> brokers that has partitions for the topic to delete all the
> partitions, then finally delete the topic from zk.
>
> Therefore once the deletion finishes it will also deleted from zk. The
> topic command can definitely however join the topic list with the
> delete topic list and mark the ones being deleted with a special
> status.
>
> Tim
>
> On Wed, Aug 6, 2014 at 11:20 PM, Gwen Shapira 
> wrote:
> > Hi Timothy,
> >
> > While we are on the subject, few questions/comments (based on the
> > trunk implementation of delete topic command):
> >
> > * After deleting a topic, I still see it when listing topics. Is the
> > expected behavior? Should it disappear after some time?
> > * When does the actual deletion gets triggered?
> > * If I try to delete a topic twice I get a pretty confusing exception
> > (Node exists from zkclient). It will be nice to catch this and say
> > "Topic is being deleted" or something to this effect.
> > * Even nicer if list topics command will mark topics as "being deleted".
> >
> > I'll probably open a separate Jira for the "nice" behavior, but
> > interested in hearing your thoughts.
> >
> > Gwen
> >
> > On Wed, Aug 6, 2014 at 11:01 PM, Timothy Chen  wrote:
> >> Is this the latest master? I've added the delete option in trunk, but
> >> it's not in any release yet.
> >>
> >> We used to have the delete option flag but I believe we removed it
> >> that's why the documentation difference.
> >>
> >> Tim
> >>
> >> On Wed, Aug 6, 2014 at 10:53 PM, Shlomi Hazan  wrote:
> >>> if the answer is pointing out the 'chroot', as a word, it makes no
> >>> difference. the result is the same:
> >>>
> >>> kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete
> >>> --topic topic-3
> >>>
> >>> gives the same:
> >>>
> >>> "Command must include exactly one action: --list, --describe, --create
> or
> >>> --alter..."
> >>>
> >>> or should I write something instead of "chroot"?
> >>>
> >>>
> >>>
> >>> On Wed, Jun 18, 2014 at 2:06 PM, Shlomi Hazan 
> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> Doing some evaluation testing, and accidently create a queue with
> wrong
> >>>> replication factor.
> >>>>
> >>>> Trying to delete as in:
> >>>>
> >>>> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181
> --delete
> >>>> --topic replicated-topic
> >>>>
> >>>> Yeilded:
> >>>>
> >>>> Command must include exactly one action: --list, --describe, --create
> or
> >>>> –alter
> >>>>
> >>>> Event though this page (https://kafka.apache.org/documentation.html)
> says:
> >>>>
> >>>>
> >>>>
> >>>> And finally deleting a topic:
> >>>>
> >>>>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete
> --topic my_topic_name
> >>>>
> >>>> WARNING: Delete topic functionality is beta in 0.8.1. Please report
> any
> >>>> bugs that you encounter on themailing list <%20us...@kafka.apache.org>
> or
> >>>> JIRA <https://issues.apache.org/jira/browse/KAFKA>.
> >>>>
> >>>> Kafka does not currently support reducing the number of partitions
> for a
> >>>> topic or changing the replication factor.
> >>>>
> >>>> What should I do?
> >>>>
> >>>> Shlomi
> >>>>
>

Re: delete topic ?

[Gwen Shapira]

Looking at the delete topic patch, it looks like there were
significant modifications in the controller code to support that, so I
think you are out of luck.

(https://reviews.apache.org/r/20745)

On Thu, Aug 7, 2014 at 8:18 AM, Jason Rosenberg  wrote:
> Since the deletion stuff is now in trunk, would be compatible to issue the
> command from a jar built from trunk, against a running 0.8.1.1 cluster?  Or
> does the cluster also have to be running trunk?  (I'm guessing it does :)).
>
> I have some topics I'd like to delete, but don't want to wait for 0.8.2
> (but will probably have to, I'm guessing).
>
> Jason
>
>
> On Thu, Aug 7, 2014 at 2:53 AM, Timothy Chen  wrote:
>
>> Hi Gwen,
>>
>> That is a very confusing error message for sure, feel free to file a
>> jira for both the experience cases.
>>
>> But in general how delete topic works is that it creates a entry in
>> the delete_topic zk path, and the leader has a delete topic thread
>> that watches that path and starts the topic deletion once it receives
>> the message. It then requires rounds of coordination among all the
>> brokers that has partitions for the topic to delete all the
>> partitions, then finally delete the topic from zk.
>>
>> Therefore once the deletion finishes it will also deleted from zk. The
>> topic command can definitely however join the topic list with the
>> delete topic list and mark the ones being deleted with a special
>> status.
>>
>> Tim
>>
>> On Wed, Aug 6, 2014 at 11:20 PM, Gwen Shapira 
>> wrote:
>> > Hi Timothy,
>> >
>> > While we are on the subject, few questions/comments (based on the
>> > trunk implementation of delete topic command):
>> >
>> > * After deleting a topic, I still see it when listing topics. Is the
>> > expected behavior? Should it disappear after some time?
>> > * When does the actual deletion gets triggered?
>> > * If I try to delete a topic twice I get a pretty confusing exception
>> > (Node exists from zkclient). It will be nice to catch this and say
>> > "Topic is being deleted" or something to this effect.
>> > * Even nicer if list topics command will mark topics as "being deleted".
>> >
>> > I'll probably open a separate Jira for the "nice" behavior, but
>> > interested in hearing your thoughts.
>> >
>> > Gwen
>> >
>> > On Wed, Aug 6, 2014 at 11:01 PM, Timothy Chen  wrote:
>> >> Is this the latest master? I've added the delete option in trunk, but
>> >> it's not in any release yet.
>> >>
>> >> We used to have the delete option flag but I believe we removed it
>> >> that's why the documentation difference.
>> >>
>> >> Tim
>> >>
>> >> On Wed, Aug 6, 2014 at 10:53 PM, Shlomi Hazan  wrote:
>> >>> if the answer is pointing out the 'chroot', as a word, it makes no
>> >>> difference. the result is the same:
>> >>>
>> >>> kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete
>> >>> --topic topic-3
>> >>>
>> >>> gives the same:
>> >>>
>> >>> "Command must include exactly one action: --list, --describe, --create
>> or
>> >>> --alter..."
>> >>>
>> >>> or should I write something instead of "chroot"?
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jun 18, 2014 at 2:06 PM, Shlomi Hazan 
>> wrote:
>> >>>
>> >>>> Hi,
>> >>>>
>> >>>> Doing some evaluation testing, and accidently create a queue with
>> wrong
>> >>>> replication factor.
>> >>>>
>> >>>> Trying to delete as in:
>> >>>>
>> >>>> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181
>> --delete
>> >>>> --topic replicated-topic
>> >>>>
>> >>>> Yeilded:
>> >>>>
>> >>>> Command must include exactly one action: --list, --describe, --create
>> or
>> >>>> –alter
>> >>>>
>> >>>> Event though this page (https://kafka.apache.org/documentation.html)
>> says:
>> >>>>
>> >>>>
>> >>>>
>> >>>> And finally deleting a topic:
>> >>>>
>> >>>>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete
>> --topic my_topic_name
>> >>>>
>> >>>> WARNING: Delete topic functionality is beta in 0.8.1. Please report
>> any
>> >>>> bugs that you encounter on themailing list <%20us...@kafka.apache.org>
>> or
>> >>>> JIRA <https://issues.apache.org/jira/browse/KAFKA>.
>> >>>>
>> >>>> Kafka does not currently support reducing the number of partitions
>> for a
>> >>>> topic or changing the replication factor.
>> >>>>
>> >>>> What should I do?
>> >>>>
>> >>>> Shlomi
>> >>>>
>>

Re: delete topic ?

[Timothy Chen]

Hi Jason,

You do want to wait for the next release as a lot of stability fixes
are going into that.

Tim

On Thu, Aug 7, 2014 at 10:25 AM, Gwen Shapira  wrote:
> Looking at the delete topic patch, it looks like there were
> significant modifications in the controller code to support that, so I
> think you are out of luck.
>
> (https://reviews.apache.org/r/20745)
>
> On Thu, Aug 7, 2014 at 8:18 AM, Jason Rosenberg  wrote:
>> Since the deletion stuff is now in trunk, would be compatible to issue the
>> command from a jar built from trunk, against a running 0.8.1.1 cluster?  Or
>> does the cluster also have to be running trunk?  (I'm guessing it does :)).
>>
>> I have some topics I'd like to delete, but don't want to wait for 0.8.2
>> (but will probably have to, I'm guessing).
>>
>> Jason
>>
>>
>> On Thu, Aug 7, 2014 at 2:53 AM, Timothy Chen  wrote:
>>
>>> Hi Gwen,
>>>
>>> That is a very confusing error message for sure, feel free to file a
>>> jira for both the experience cases.
>>>
>>> But in general how delete topic works is that it creates a entry in
>>> the delete_topic zk path, and the leader has a delete topic thread
>>> that watches that path and starts the topic deletion once it receives
>>> the message. It then requires rounds of coordination among all the
>>> brokers that has partitions for the topic to delete all the
>>> partitions, then finally delete the topic from zk.
>>>
>>> Therefore once the deletion finishes it will also deleted from zk. The
>>> topic command can definitely however join the topic list with the
>>> delete topic list and mark the ones being deleted with a special
>>> status.
>>>
>>> Tim
>>>
>>> On Wed, Aug 6, 2014 at 11:20 PM, Gwen Shapira 
>>> wrote:
>>> > Hi Timothy,
>>> >
>>> > While we are on the subject, few questions/comments (based on the
>>> > trunk implementation of delete topic command):
>>> >
>>> > * After deleting a topic, I still see it when listing topics. Is the
>>> > expected behavior? Should it disappear after some time?
>>> > * When does the actual deletion gets triggered?
>>> > * If I try to delete a topic twice I get a pretty confusing exception
>>> > (Node exists from zkclient). It will be nice to catch this and say
>>> > "Topic is being deleted" or something to this effect.
>>> > * Even nicer if list topics command will mark topics as "being deleted".
>>> >
>>> > I'll probably open a separate Jira for the "nice" behavior, but
>>> > interested in hearing your thoughts.
>>> >
>>> > Gwen
>>> >
>>> > On Wed, Aug 6, 2014 at 11:01 PM, Timothy Chen  wrote:
>>> >> Is this the latest master? I've added the delete option in trunk, but
>>> >> it's not in any release yet.
>>> >>
>>> >> We used to have the delete option flag but I believe we removed it
>>> >> that's why the documentation difference.
>>> >>
>>> >> Tim
>>> >>
>>> >> On Wed, Aug 6, 2014 at 10:53 PM, Shlomi Hazan  wrote:
>>> >>> if the answer is pointing out the 'chroot', as a word, it makes no
>>> >>> difference. the result is the same:
>>> >>>
>>> >>> kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete
>>> >>> --topic topic-3
>>> >>>
>>> >>> gives the same:
>>> >>>
>>> >>> "Command must include exactly one action: --list, --describe, --create
>>> or
>>> >>> --alter..."
>>> >>>
>>> >>> or should I write something instead of "chroot"?
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Wed, Jun 18, 2014 at 2:06 PM, Shlomi Hazan 
>>> wrote:
>>> >>>
>>> >>>> Hi,
>>> >>>>
>>> >>>> Doing some evaluation testing, and accidently create a queue with
>>> wrong
>>> >>>> replication factor.
>>> >>>>
>>> >>>> Trying to delete as in:
>>> >>>>
>>> >>>> kafka_2.10-0.8.1.1/bin/kafka-topics.sh --zookeeper localhost:2181
>>> --delete
>>> >>>> --topic replicated-topic
>>> >>>>
>>> >>>> Yeilded:
>>> >>>>
>>> >>>> Command must include exactly one action: --list, --describe, --create
>>> or
>>> >>>> –alter
>>> >>>>
>>> >>>> Event though this page (https://kafka.apache.org/documentation.html)
>>> says:
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> And finally deleting a topic:
>>> >>>>
>>> >>>>  > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete
>>> --topic my_topic_name
>>> >>>>
>>> >>>> WARNING: Delete topic functionality is beta in 0.8.1. Please report
>>> any
>>> >>>> bugs that you encounter on themailing list <%20us...@kafka.apache.org>
>>> or
>>> >>>> JIRA <https://issues.apache.org/jira/browse/KAFKA>.
>>> >>>>
>>> >>>> Kafka does not currently support reducing the number of partitions
>>> for a
>>> >>>> topic or changing the replication factor.
>>> >>>>
>>> >>>> What should I do?
>>> >>>>
>>> >>>> Shlomi
>>> >>>>
>>>

Re: Delete topic

[Gwen Shapira]

At the moment, the best way would be:

* Wait about two weeks
* Upgrade to 0.8.2
* Use kafka-topic.sh --delete

:)

2015-01-14 9:26 GMT-08:00 Armando Martinez Briones :
> Hi.
>
> What is the best way to delete a topic into production environment?
>
> --
> [image: Tralix][image: 1]José Armando Martínez Briones
> *Arquitecto de software*
> Tralix México
> Tel: +52 442 161 1002 ext. 2920
> www.tralix.com.mx
> 
> *TRALIX MÉXICO S. DE R.L.
> DE C.V* como responsable del tratamiento de sus datos personales, hace de
> su conocimiento que la información obtenida por este medio es tratada de
> forma estrictamente confidencial por lo que recabaremos y trataremos de sus
> datos según los lineamientos de nuestro Aviso de Privacidad
> . Para
> conocer la versión completa podrá hacerlo a través de la página Aviso de
> privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> oposición al tratamiento de sus Datos Personales, se entiende que usted
> conoció la versión completa y acepta los términos del mismo, siempre de
> conformidad a la Ley Federal de Protección de Datos Personales en Posesión
> de los Particulares.

Re: Delete topic

[Armando Martinez Briones]

thanks Gwen Shapira ;)

El 14 de enero de 2015, 11:31, Gwen Shapira 
escribió:

> At the moment, the best way would be:
>
> * Wait about two weeks
> * Upgrade to 0.8.2
> * Use kafka-topic.sh --delete
>
> :)
>
> 2015-01-14 9:26 GMT-08:00 Armando Martinez Briones :
> > Hi.
> >
> > What is the best way to delete a topic into production environment?
> >
> > --
> > [image: Tralix][image: 1]José Armando Martínez Briones
> > *Arquitecto de software*
> > Tralix México
> > Tel: +52 442 161 1002 ext. 2920
> > www.tralix.com.mx
> > 
> > *TRALIX MÉXICO S. DE
> R.L.
> > DE C.V* como responsable del tratamiento de sus datos personales, hace de
> > su conocimiento que la información obtenida por este medio es tratada de
> > forma estrictamente confidencial por lo que recabaremos y trataremos de
> sus
> > datos según los lineamientos de nuestro Aviso de Privacidad
> > .
> Para
> > conocer la versión completa podrá hacerlo a través de la página Aviso de
> > privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> > oposición al tratamiento de sus Datos Personales, se entiende que usted
> > conoció la versión completa y acepta los términos del mismo, siempre de
> > conformidad a la Ley Federal de Protección de Datos Personales en
> Posesión
> > de los Particulares.
>



-- 
[image: Tralix][image: 1]José Armando Martínez Briones
*Arquitecto de software*
Tralix México
Tel: +52 442 161 1002  ext. 2920
www.tralix.com.mx

*TRALIX MÉXICO S. DE R.L.
DE C.V* como responsable del tratamiento de sus datos personales, hace de
su conocimiento que la información obtenida por este medio es tratada de
forma estrictamente confidencial por lo que recabaremos y trataremos de sus
datos según los lineamientos de nuestro Aviso de Privacidad
. Para
conocer la versión completa podrá hacerlo a través de la página Aviso de
privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
oposición al tratamiento de sus Datos Personales, se entiende que usted
conoció la versión completa y acepta los términos del mismo, siempre de
conformidad a la Ley Federal de Protección de Datos Personales en Posesión
de los Particulares.

Re: Delete topic

[Jayesh Thakrar]

Does one also need to set the config parameter "delete.topic.enable" to true ?I 
am using 8.2 beta and I had to set it to true to enable topic deletion.
  From: Armando Martinez Briones 
 To: users@kafka.apache.org 
 Sent: Wednesday, January 14, 2015 11:33 AM
 Subject: Re: Delete topic
   
thanks Gwen Shapira ;)

El 14 de enero de 2015, 11:31, Gwen Shapira 
escribió:

> At the moment, the best way would be:
>
> * Wait about two weeks
> * Upgrade to 0.8.2
> * Use kafka-topic.sh --delete
>
> :)
>
> 2015-01-14 9:26 GMT-08:00 Armando Martinez Briones :
> > Hi.
> >
> > What is the best way to delete a topic into production environment?
> >
> > --
> > [image: Tralix][image: 1]José Armando Martínez Briones
> > *Arquitecto de software*
> > Tralix México
> > Tel: +52 442 161 1002 ext. 2920
> > www.tralix.com.mx
> > <https://twitter.com/#!/tralix>
> > <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE
> R.L.
> > DE C.V* como responsable del tratamiento de sus datos personales, hace de
> > su conocimiento que la información obtenida por este medio es tratada de
> > forma estrictamente confidencial por lo que recabaremos y trataremos de
> sus
> > datos según los lineamientos de nuestro Aviso de Privacidad
> > <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>.
> Para
> > conocer la versión completa podrá hacerlo a través de la página Aviso de
> > privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> > oposición al tratamiento de sus Datos Personales, se entiende que usted
> > conoció la versión completa y acepta los términos del mismo, siempre de
> > conformidad a la Ley Federal de Protección de Datos Personales en
> Posesión
> > de los Particulares.


>



-- 
[image: Tralix][image: 1]José Armando Martínez Briones
*Arquitecto de software*
Tralix México
Tel: +52 442 161 1002  ext. 2920
www.tralix.com.mx
<https://twitter.com/#!/tralix>
<http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE R.L.
DE C.V* como responsable del tratamiento de sus datos personales, hace de
su conocimiento que la información obtenida por este medio es tratada de
forma estrictamente confidencial por lo que recabaremos y trataremos de sus
datos según los lineamientos de nuestro Aviso de Privacidad
<http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>. Para
conocer la versión completa podrá hacerlo a través de la página Aviso de
privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
oposición al tratamiento de sus Datos Personales, se entiende que usted
conoció la versión completa y acepta los términos del mismo, siempre de
conformidad a la Ley Federal de Protección de Datos Personales en Posesión
de los Particulares.

  

Re: Delete topic

[Armando Martinez Briones]

thanks Jayesh,  i'm working with the version 8.1.1 because is the version
on production environment.

El 14 de enero de 2015, 12:16, Jayesh Thakrar 
escribió:

> Does one also need to set the config parameter "delete.topic.enable" to
> true ?I am using 8.2 beta and I had to set it to true to enable topic
> deletion.
>   From: Armando Martinez Briones 
>  To: users@kafka.apache.org
>  Sent: Wednesday, January 14, 2015 11:33 AM
>  Subject: Re: Delete topic
>
> thanks Gwen Shapira ;)
>
> El 14 de enero de 2015, 11:31, Gwen Shapira 
> escribió:
>
> > At the moment, the best way would be:
> >
> > * Wait about two weeks
> > * Upgrade to 0.8.2
> > * Use kafka-topic.sh --delete
> >
> > :)
> >
> > 2015-01-14 9:26 GMT-08:00 Armando Martinez Briones :
> > > Hi.
> > >
> > > What is the best way to delete a topic into production environment?
> > >
> > > --
> > > [image: Tralix][image: 1]José Armando Martínez Briones
> > > *Arquitecto de software*
> > > Tralix México
> > > Tel: +52 442 161 1002 ext. 2920
> > > www.tralix.com.mx
> > > <https://twitter.com/#!/tralix>
> > > <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE
> > R.L.
> > > DE C.V* como responsable del tratamiento de sus datos personales, hace
> de
> > > su conocimiento que la información obtenida por este medio es tratada
> de
> > > forma estrictamente confidencial por lo que recabaremos y trataremos de
> > sus
> > > datos según los lineamientos de nuestro Aviso de Privacidad
> > > <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>.
> > Para
> > > conocer la versión completa podrá hacerlo a través de la página Aviso
> de
> > > privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> > > oposición al tratamiento de sus Datos Personales, se entiende que usted
> > > conoció la versión completa y acepta los términos del mismo, siempre de
> > > conformidad a la Ley Federal de Protección de Datos Personales en
> > Posesión
> > > de los Particulares.
>
>
> >
>
>
>
> --
> [image: Tralix][image: 1]José Armando Martínez Briones
> *Arquitecto de software*
> Tralix México
> Tel: +52 442 161 1002  ext. 2920
> www.tralix.com.mx
> <https://twitter.com/#!/tralix>
> <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE
> R.L.
> DE C.V* como responsable del tratamiento de sus datos personales, hace de
> su conocimiento que la información obtenida por este medio es tratada de
> forma estrictamente confidencial por lo que recabaremos y trataremos de sus
> datos según los lineamientos de nuestro Aviso de Privacidad
> <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>. Para
> conocer la versión completa podrá hacerlo a través de la página Aviso de
> privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> oposición al tratamiento de sus Datos Personales, se entiende que usted
> conoció la versión completa y acepta los términos del mismo, siempre de
> conformidad a la Ley Federal de Protección de Datos Personales en Posesión
> de los Particulares.
>
>
>



-- 
[image: Tralix][image: 1]José Armando Martínez Briones
*Arquitecto de software*
Tralix México
Tel: +52 442 161 1002  ext. 2920
www.tralix.com.mx
<https://twitter.com/#!/tralix>
<http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE R.L.
DE C.V* como responsable del tratamiento de sus datos personales, hace de
su conocimiento que la información obtenida por este medio es tratada de
forma estrictamente confidencial por lo que recabaremos y trataremos de sus
datos según los lineamientos de nuestro Aviso de Privacidad
<http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>. Para
conocer la versión completa podrá hacerlo a través de la página Aviso de
privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
oposición al tratamiento de sus Datos Personales, se entiende que usted
conoció la versión completa y acepta los términos del mismo, siempre de
conformidad a la Ley Federal de Protección de Datos Personales en Posesión
de los Particulares.

Re: Delete topic

[Gwen Shapira]

Yep, you need to set delete.topic.enable=true.

Forgot that step :)

2015-01-14 10:16 GMT-08:00 Jayesh Thakrar :
> Does one also need to set the config parameter "delete.topic.enable" to true 
> ?I am using 8.2 beta and I had to set it to true to enable topic deletion.
>   From: Armando Martinez Briones 
>  To: users@kafka.apache.org
>  Sent: Wednesday, January 14, 2015 11:33 AM
>  Subject: Re: Delete topic
>
> thanks Gwen Shapira ;)
>
> El 14 de enero de 2015, 11:31, Gwen Shapira 
> escribió:
>
>> At the moment, the best way would be:
>>
>> * Wait about two weeks
>> * Upgrade to 0.8.2
>> * Use kafka-topic.sh --delete
>>
>> :)
>>
>> 2015-01-14 9:26 GMT-08:00 Armando Martinez Briones :
>> > Hi.
>> >
>> > What is the best way to delete a topic into production environment?
>> >
>> > --
>> > [image: Tralix][image: 1]José Armando Martínez Briones
>> > *Arquitecto de software*
>> > Tralix México
>> > Tel: +52 442 161 1002 ext. 2920
>> > www.tralix.com.mx
>> > <https://twitter.com/#!/tralix>
>> > <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE
>> R.L.
>> > DE C.V* como responsable del tratamiento de sus datos personales, hace de
>> > su conocimiento que la información obtenida por este medio es tratada de
>> > forma estrictamente confidencial por lo que recabaremos y trataremos de
>> sus
>> > datos según los lineamientos de nuestro Aviso de Privacidad
>> > <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>.
>> Para
>> > conocer la versión completa podrá hacerlo a través de la página Aviso de
>> > privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
>> > oposición al tratamiento de sus Datos Personales, se entiende que usted
>> > conoció la versión completa y acepta los términos del mismo, siempre de
>> > conformidad a la Ley Federal de Protección de Datos Personales en
>> Posesión
>> > de los Particulares.
>
>
>>
>
>
>
> --
> [image: Tralix][image: 1]José Armando Martínez Briones
> *Arquitecto de software*
> Tralix México
> Tel: +52 442 161 1002  ext. 2920
> www.tralix.com.mx
> <https://twitter.com/#!/tralix>
> <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE R.L.
> DE C.V* como responsable del tratamiento de sus datos personales, hace de
> su conocimiento que la información obtenida por este medio es tratada de
> forma estrictamente confidencial por lo que recabaremos y trataremos de sus
> datos según los lineamientos de nuestro Aviso de Privacidad
> <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>. Para
> conocer la versión completa podrá hacerlo a través de la página Aviso de
> privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> oposición al tratamiento de sus Datos Personales, se entiende que usted
> conoció la versión completa y acepta los términos del mismo, siempre de
> conformidad a la Ley Federal de Protección de Datos Personales en Posesión
> de los Particulares.
>
>

Re: Delete topic

[Manikumar Reddy]

I think now we should delete this config property and allow topic deletion
in 0.8.2
 Yep, you need to set delete.topic.enable=true.

Forgot that step :)

2015-01-14 10:16 GMT-08:00 Jayesh Thakrar :
> Does one also need to set the config parameter "delete.topic.enable" to
true ?I am using 8.2 beta and I had to set it to true to enable topic
deletion.
>   From: Armando Martinez Briones 
>  To: users@kafka.apache.org
>  Sent: Wednesday, January 14, 2015 11:33 AM
>  Subject: Re: Delete topic
>
> thanks Gwen Shapira ;)
>
> El 14 de enero de 2015, 11:31, Gwen Shapira 
> escribió:
>
>> At the moment, the best way would be:
>>
>> * Wait about two weeks
>> * Upgrade to 0.8.2
>> * Use kafka-topic.sh --delete
>>
>> :)
>>
>> 2015-01-14 9:26 GMT-08:00 Armando Martinez Briones :
>> > Hi.
>> >
>> > What is the best way to delete a topic into production environment?
>> >
>> > --
>> > [image: Tralix][image: 1]José Armando Martínez Briones
>> > *Arquitecto de software*
>> > Tralix México
>> > Tel: +52 442 161 1002 ext. 2920
>> > www.tralix.com.mx
>> > <https://twitter.com/#!/tralix>
>> > <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE
>> R.L.
>> > DE C.V* como responsable del tratamiento de sus datos personales, hace
de
>> > su conocimiento que la información obtenida por este medio es tratada
de
>> > forma estrictamente confidencial por lo que recabaremos y trataremos de
>> sus
>> > datos según los lineamientos de nuestro Aviso de Privacidad
>> > <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>.
>> Para
>> > conocer la versión completa podrá hacerlo a través de la página Aviso
de
>> > privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
>> > oposición al tratamiento de sus Datos Personales, se entiende que usted
>> > conoció la versión completa y acepta los términos del mismo, siempre de
>> > conformidad a la Ley Federal de Protección de Datos Personales en
>> Posesión
>> > de los Particulares.
>
>
>>
>
>
>
> --
> [image: Tralix][image: 1]José Armando Martínez Briones
> *Arquitecto de software*
> Tralix México
> Tel: +52 442 161 1002  ext. 2920
> www.tralix.com.mx
> <https://twitter.com/#!/tralix>
> <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE
R.L.
> DE C.V* como responsable del tratamiento de sus datos personales, hace de
> su conocimiento que la información obtenida por este medio es tratada de
> forma estrictamente confidencial por lo que recabaremos y trataremos de
sus
> datos según los lineamientos de nuestro Aviso de Privacidad
> <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>. Para
> conocer la versión completa podrá hacerlo a través de la página Aviso de
> privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> oposición al tratamiento de sus Datos Personales, se entiende que usted
> conoció la versión completa y acepta los términos del mismo, siempre de
> conformidad a la Ley Federal de Protección de Datos Personales en Posesión
> de los Particulares.
>
>

Re: Delete topic

[Armando Martinez Briones]

I think that it's for securiry.

El 14 de enero de 2015, 12:44, Manikumar Reddy 
escribió:

> I think now we should delete this config property and allow topic deletion
> in 0.8.2
>  Yep, you need to set delete.topic.enable=true.
>
> Forgot that step :)
>
> 2015-01-14 10:16 GMT-08:00 Jayesh Thakrar :
> > Does one also need to set the config parameter "delete.topic.enable" to
> true ?I am using 8.2 beta and I had to set it to true to enable topic
> deletion.
> >   From: Armando Martinez Briones 
> >  To: users@kafka.apache.org
> >  Sent: Wednesday, January 14, 2015 11:33 AM
> >  Subject: Re: Delete topic
> >
> > thanks Gwen Shapira ;)
> >
> > El 14 de enero de 2015, 11:31, Gwen Shapira 
> > escribió:
> >
> >> At the moment, the best way would be:
> >>
> >> * Wait about two weeks
> >> * Upgrade to 0.8.2
> >> * Use kafka-topic.sh --delete
> >>
> >> :)
> >>
> >> 2015-01-14 9:26 GMT-08:00 Armando Martinez Briones  >:
> >> > Hi.
> >> >
> >> > What is the best way to delete a topic into production environment?
> >> >
> >> > --
> >> > [image: Tralix][image: 1]José Armando Martínez Briones
> >> > *Arquitecto de software*
> >> > Tralix México
> >> > Tel: +52 442 161 1002 ext. 2920
> >> > www.tralix.com.mx
> >> > <https://twitter.com/#!/tralix>
> >> > <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S.
> DE
> >> R.L.
> >> > DE C.V* como responsable del tratamiento de sus datos personales, hace
> de
> >> > su conocimiento que la información obtenida por este medio es tratada
> de
> >> > forma estrictamente confidencial por lo que recabaremos y trataremos
> de
> >> sus
> >> > datos según los lineamientos de nuestro Aviso de Privacidad
> >> > <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>.
> >> Para
> >> > conocer la versión completa podrá hacerlo a través de la página Aviso
> de
> >> > privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> >> > oposición al tratamiento de sus Datos Personales, se entiende que
> usted
> >> > conoció la versión completa y acepta los términos del mismo, siempre
> de
> >> > conformidad a la Ley Federal de Protección de Datos Personales en
> >> Posesión
> >> > de los Particulares.
> >
> >
> >>
> >
> >
> >
> > --
> > [image: Tralix][image: 1]José Armando Martínez Briones
> > *Arquitecto de software*
> > Tralix México
> > Tel: +52 442 161 1002  ext. 2920
> > www.tralix.com.mx
> > <https://twitter.com/#!/tralix>
> > <http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE
> R.L.
> > DE C.V* como responsable del tratamiento de sus datos personales, hace de
> > su conocimiento que la información obtenida por este medio es tratada de
> > forma estrictamente confidencial por lo que recabaremos y trataremos de
> sus
> > datos según los lineamientos de nuestro Aviso de Privacidad
> > <http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>.
> Para
> > conocer la versión completa podrá hacerlo a través de la página Aviso de
> > privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
> > oposición al tratamiento de sus Datos Personales, se entiende que usted
> > conoció la versión completa y acepta los términos del mismo, siempre de
> > conformidad a la Ley Federal de Protección de Datos Personales en
> Posesión
> > de los Particulares.
> >
> >
>



-- 
[image: Tralix][image: 1]José Armando Martínez Briones
*Arquitecto de software*
Tralix México
Tel: +52 442 161 1002  ext. 2920
www.tralix.com.mx
<https://twitter.com/#!/tralix>
<http://www.facebook.com/pages/Tralix/47723192646>*TRALIX MÉXICO S. DE R.L.
DE C.V* como responsable del tratamiento de sus datos personales, hace de
su conocimiento que la información obtenida por este medio es tratada de
forma estrictamente confidencial por lo que recabaremos y trataremos de sus
datos según los lineamientos de nuestro Aviso de Privacidad
<http://tralix.com.mx/politicasdeprivacidad/avisodeprivacidad.html>. Para
conocer la versión completa podrá hacerlo a través de la página Aviso de
privacidad. Al leer el presente Aviso de Privacidad y no manifestar su
oposición al tratamiento de sus Datos Personales, se entiende que usted
conoció la versión completa y acepta los términos del mismo, siempre de
conformidad a la Ley Federal de Protección de Datos Personales en Posesión
de los Particulares.

Delete topic pending

[Dillian Murphey]

If a broker doesn't have the topic, and I run delete topic, that topic will
be in a "pending delete" state forever.

What am I doing wrong here?

Also, what if I have data loss and I just want to delete the dang topic
form zookeeper directly with non of this pending stuff.

Thanks

Re: Delete Topic - BadVersionException

[Jun Rao]

Delete topic doesn't quite work yet and we will try to fix it in the next
release. https://issues.apache.org/jira/browse/KAFKA-1397

Thanks,

Jun


On Thu, Apr 24, 2014 at 9:49 AM, Drew Goya  wrote:

> Just tried my first topic delete today and it looks like something went
> wrong on the controller.  I issued the command on a test topic and shortly
> after that a describe looked like:
>
> Topic:TimeoutQueueTest PartitionCount:256 ReplicationFactor:3 Configs:
> Topic: TimeoutQueueTest Partition: 0 Leader: -1 Replicas: 9,14,15 Isr:
> Topic: TimeoutQueueTest Partition: 1 Leader: -1 Replicas: 10,15,1 Isr:
> Topic: TimeoutQueueTest Partition: 2 Leader: -1 Replicas: 11,1,2 Isr:
> Topic: TimeoutQueueTest Partition: 3 Leader: -1 Replicas: 12,2,3 Isr:
> Topic: TimeoutQueueTest Partition: 4 Leader: -1 Replicas: 13,3,4 Isr:
> Topic: TimeoutQueueTest Partition: 5 Leader: -1 Replicas: 14,4,5 Isr:
>
> It stayed that way for quite a while so I hit zookeeper and went looking
> for who was the controller, I found these in that brokers logs:
>
> [2014-04-24 16:27:42,498] ERROR Conditional update of path
> /brokers/topics/TimeoutQueueTest/partitions/170/state with data
>
> {"controller_epoch":18,"leader":2,"version":1,"leader_epoch":14,"isr":[2,14]}
> and expected version 30 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
> BadVersion for /brokers/topics/TimeoutQueueTest/partitions/170/state
> (kafka.utils.ZkUtils$)
> [2014-04-24 16:27:42,504] ERROR Conditional update of path
> /brokers/topics/TimeoutQueueTest/partitions/113/state with data
>
> {"controller_epoch":18,"leader":2,"version":1,"leader_epoch":4,"isr":[2,15,14]}
> and expected version 17 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
> BadVersion for /brokers/topics/TimeoutQueueTest/partitions/113/state
> (kafka.utils.ZkUtils$)
>
> Any ideas?
>

How to delete Topic ?

[rafeeq s]

I am using Kafka 0.8.1 and I tried to execute Topic delete command but
still Topic list showing that specific Topic.

Topic Delete commad which i used:

*bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
my_topic_name*

How can i verify that my_topic is deleted ?

Is there specific steps to delete Topic in Kafka 0.8.1 ?

Thanks in advance!

Delete Topic with 0.8?

[Jason Huang]

Hello,

Do you know if there is any support to delete topics with Kafka 0.8?

Since I couldn't find any, I tried to manually delete the log folders
(topic name + partition). However, if I restart the broker (we are
testing with one single broker), it automatically regenerates those
log folders again (although the actual message content is gone).

Is there any way to remove existing topics completely? (Either by some
manual deletion or by some API call)?

thanks,

Jason

Delete Topic in 0.8.2

[Hema Bhatia]

I upgraded my kafka server to 0.8.2 and client to use 0.8.2 as well..

I am trying to test delete topic feature and I see that delete topic do not 
work consistently.
I saw it working fine a first few times but after a while I saw that deleting 
topics add them to delete_topic node in admin folder but does not remove it 
from topics in brokers.
Also, to make things simple, no other application server was running, and no 
consumer created for these topics. Just create topic followed by delete topic.

Anything I am missing?

Here is the snapshot of createTopic followed by deleteTopic:


[cid:image001.png@01D0541C.E87D7D20]




This message is private and confidential. If you have received it in error, 
please notify the sender and remove it from your system.

Re: Delete topic pending

[Sriharsha Chintalapani]

May be thats a bug. We should probably alert the user saying the topic doesn’t 
exist rather than go and add it to the zookeeper for deletion trigger. Also to 
delete any topic you need to set “delete.topic.enable” to true

-- 
Harsha


On May 18, 2015 at 10:16:46 AM, Dillian Murphey (crackshotm...@gmail.com) wrote:

If a broker doesn't have the topic, and I run delete topic, that topic will  
be in a "pending delete" state forever.  

What am I doing wrong here?  

Also, what if I have data loss and I just want to delete the dang topic  
form zookeeper directly with non of this pending stuff.  

Thanks  

Re: Delete topic pending

[Jiangjie Qin]

Hmm, which Kafka version are you running? From the code it looks we should
have already ignored the deletion of a topic if it does not exist.

Jiangjie (Becket) Qin

On 5/18/15, 10:15 AM, "Dillian Murphey"  wrote:

>If a broker doesn't have the topic, and I run delete topic, that topic
>will
>be in a "pending delete" state forever.
>
>What am I doing wrong here?
>
>Also, what if I have data loss and I just want to delete the dang topic
>form zookeeper directly with non of this pending stuff.
>
>Thanks

Delete topic and logs

[Neelesh]

I searched for this and could not get a definitive answer- when do logs get
deleted , after a topic is deleted? What happens if I delete a topic and
recreate it immediately?

So far I know that deleting a topic marks it for deleted in zk, and some
sweeper eventually deletes the topic meta and logs. It's quite not clear if
there is a state in between that may   reattach old logs to the newly
created topic with the same name?
Thank you!

Re: How to delete Topic ?

[François Langelier]

The delete topic isn't working ATM

I think it will be available in the next release
https://issues.apache.org/jira/browse/KAFKA-1397



François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Argentier Fraternité du Piranha <http://fraternitedupiranha.com/> 2012-2014
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior


On 3 June 2014 07:46, rafeeq s  wrote:

> I am using Kafka 0.8.1 and I tried to execute Topic delete command but
> still Topic list showing that specific Topic.
>
> Topic Delete commad which i used:
>
> *bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> my_topic_name*
>
> How can i verify that my_topic is deleted ?
>
> Is there specific steps to delete Topic in Kafka 0.8.1 ?
>
> Thanks in advance!
>

Re: Delete Topic with 0.8?

[Neha Narkhede]

The JIRA tracking delete topic is here -
https://issues.apache.org/jira/browse/KAFKA-330

Until that is resolved, what you can do is shut the cluster down, delete on
disk data as well as zookeeper data, and then restart everything.
Understand that it is cumbersome, but until the feature is in, I can't
think of an easier way to delete a topic

Thanks,
Neha


On Fri, Feb 1, 2013 at 7:25 AM, Jason Huang  wrote:

> Hello,
>
> Do you know if there is any support to delete topics with Kafka 0.8?
>
> Since I couldn't find any, I tried to manually delete the log folders
> (topic name + partition). However, if I restart the broker (we are
> testing with one single broker), it automatically regenerates those
> log folders again (although the actual message content is gone).
>
> Is there any way to remove existing topics completely? (Either by some
> manual deletion or by some API call)?
>
> thanks,
>
> Jason
>

Re: Delete Topic with 0.8?

[Jason Huang]

I see.

thanks Neha.

Jason

On Fri, Feb 1, 2013 at 10:43 AM, Neha Narkhede  wrote:
> The JIRA tracking delete topic is here -
> https://issues.apache.org/jira/browse/KAFKA-330
>
> Until that is resolved, what you can do is shut the cluster down, delete on
> disk data as well as zookeeper data, and then restart everything.
> Understand that it is cumbersome, but until the feature is in, I can't
> think of an easier way to delete a topic
>
> Thanks,
> Neha
>
>
> On Fri, Feb 1, 2013 at 7:25 AM, Jason Huang  wrote:
>
>> Hello,
>>
>> Do you know if there is any support to delete topics with Kafka 0.8?
>>
>> Since I couldn't find any, I tried to manually delete the log folders
>> (topic name + partition). However, if I restart the broker (we are
>> testing with one single broker), it automatically regenerates those
>> log folders again (although the actual message content is gone).
>>
>> Is there any way to remove existing topics completely? (Either by some
>> manual deletion or by some API call)?
>>
>> thanks,
>>
>> Jason
>>

Delete topic API in 0.8.2

[Sumit Rangwala]

I am trying to find if there is a supported API to delete topic (from
within my code) specifically in 0.8.2. One method that I can think of is
calling kafka.admin.TopicCommand.main with the same parameters as one gives
on the command line. Is this the recommended way or is there a better way
of doing it?

Furthermore, are the details of deleting a topic in 0.8.2 documented, since
I would like to understand if delete topic only deletes the topic in
zookeeper or it also cleans the logs in kafka broker as well.


Sumit

Re: Delete Topic in 0.8.2

[Harsha]

Hi Hema, Can you attach controller.log and state-change.log. Image is
not showing up at least for me. Can you also give us details on how big
the cluster is and topic's partitions and replication-factor and any
steps on reproducing this. Thanks, Harsha


On Sun, Mar 1, 2015, at 12:40 PM, Hema Bhatia wrote:
> I upgraded my kafka server to 0.8.2 and client to use 0.8.2 as well..


>


> I am trying to test delete topic feature and I see that delete topic
> do not work consistently.


> I saw it working fine a first few times but after a while I saw that
> deleting topics add them to delete_topic node in admin folder but does
> not remove it from topics in brokers.


> Also, to make things simple, no other application server was running,
> and no consumer created for these topics. Just create topic followed
> by delete topic.


>


> Anything I am missing?


>


> Here is the snapshot of createTopic followed by deleteTopic:


>


>


>


>


>


>
> This message is private and confidential. If you have received it in
> error, please notify the sender and remove it from your system.

RE: Delete Topic in 0.8.2

[Hema Bhatia]

Grr.. delete.topic.enable=true was wiped out at kafka restart from 
server.properties. I have it working now.. thanks.

-Original Message-
From: Harsha [mailto:ka...@harsha.io]
Sent: Sunday, March 01, 2015 12:53 PM
To: users@kafka.apache.org
Subject: Re: Delete Topic in 0.8.2


Hi Hema, Can you attach controller.log and state-change.log. Image is not 
showing up at least for me. Can you also give us details on how big the cluster 
is and topic's partitions and replication-factor and any steps on reproducing 
this. Thanks, Harsha


On Sun, Mar 1, 2015, at 12:40 PM, Hema Bhatia wrote:
> I upgraded my kafka server to 0.8.2 and client to use 0.8.2 as well..


>


> I am trying to test delete topic feature and I see that delete topic
> do not work consistently.


> I saw it working fine a first few times but after a while I saw that
> deleting topics add them to delete_topic node in admin folder but does
> not remove it from topics in brokers.


> Also, to make things simple, no other application server was running,
> and no consumer created for these topics. Just create topic followed
> by delete topic.


>


> Anything I am missing?


>


> Here is the snapshot of createTopic followed by deleteTopic:


>


>


>


>


>


>
> This message is private and confidential. If you have received it in
> error, please notify the sender and remove it from your system.


This message is private and confidential. If you have received it in error, 
please notify the sender and remove it from your system.

Delete topic / Recreate = No leader

[Dillian Murphey]

I am trying to reproduce this. But if I create a topic, then delete it,
then re-create it, no leader is getting assigned.

I can still produce/consume messages (via command line, basic testing).

Is there some additional cleanup I need to do?

Thanks for your time!

Failed attempt to delete topic

[Rakesh Vidyadharan]

Hello,

We are on an older kafka (0.8.1) version.  While a number of consumers were 
running, we attempted to delete a few topics using the kafka-topics.sh file 
(basically want to remove all messages in that topic and restart, since our 
entities went through some incompatible changes).  We noticed logs saying the 
topic has been queued for deletion.  After stopping all processes accessing 
kafka, we restarted kafka and then our processes.  The old topics do not seem 
to have been deleted (I can still see the log directories corresponding to the 
topics), and none of the clients are able to either publish or read to the 
topics that we attempted to delete.  Attempting to read gives us the following 
type of error:

Attempting to access an invalid KafkaTopic ( are you operating on a closed 
KafkaTopic ?)

Attempting to publish gives us a more general type of error:

kafka.common.FailedToSendMessageException: Failed to send messages after 3 
tries.
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at kafka.producer.Producer.send(Producer.scala:76)

How can be get around this issue and start using the topics that we tried to 
clean up?  There may have been better ways to achieve what we wanted, if so 
please suggest recommendations as well.

Thanks
Rakesh

kafka 0.8.2.2 delete topic issue

[Ivan Dyachkov]

Hi!

We're running cluster of 3 kafka-0.8.2.2 nodes and delete.topic.enable is set 
to true on all nodes.

Today we tried to delete one of the topics. I waited ~20 minutes after the 
kafka-topics.sh --delete was executed but the topic was still there.

--describe showed Leader: -1 and only one of three brokers (1, 2 or 3) in ISR 
for all partitions.

The topic had the following configuration:
partitions: 50
replication factor: 3
unclean.leader.election.enable: false
min.insync.replicas: 2

Then I restarted one of the nodes with the hope to trigger retry on topic 
deletion. After that things went nuts and we had short cluster downtime before 
I managed to recover it and get to the same point when we were before the 
restart. The topic was still there.

Since we successfully deleted topics before on the same cluster some time ago, 
the only thing which could be different is someone could hold an open 
connection or a consumer group linked to the topic. According to previous 
similar issues described in this group this could make topic deletion to fail. 
We had shut down the service responsible for this topic, triggered preferred 
replica election, but that didn't help. Also we set retention.bytes topic 
config to 1, waited for retention.interval.ms, nothing changed.

The only option to recover which we see is shut down all nodes, delete the 
topic from file system, remove the topic from zookeeper on /config/topics/ and 
/admin/delete_topics and start the nodes again.

So my question is really is it possible to recover from this situation, either 
cancel topic deletion or finally delete it, without cluster downtime?

Thanks.

/Ivan

Re: Delete topic and logs

[Feroze Daud]

unsubscribe 

On Saturday, February 20, 2016 9:22 AM, Neelesh  wrote:
 

 I searched for this and could not get a definitive answer- when do logs get
deleted , after a topic is deleted? What happens if I delete a topic and
recreate it immediately?

So far I know that deleting a topic marks it for deleted in zk, and some
sweeper eventually deletes the topic meta and logs. It's quite not clear if
there is a state in between that may  reattach old logs to the newly
created topic with the same name?
Thank you!


  

Kafka 0.8: delete topic with data

[Yury Ruchin]

Hi,

I'm using Kafka 0.8 which does not have a command to delete topic. However,
I need the functionality and I'm trying to adopt this approach:
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/DeleteTopicCommand.scala.
I see it simply deletes the topic node from ZK. My question is what will
happen with the topic data after removing the node? Will Kafka brokers be
able to detect that topic does not exist anymore and purge its data?

Thanks,
Yury

Unable to delete topic - kafka 0.8.0

[Virendra Pratap Singh]

I am aware of lack of programmatic way of deleting topics in kafka 0.8.0. So 
using the sledge hammer approach.
This is what I am doing:

1. Bring whole of my kafka cluster down.
2. Delete all the content on all the kafka clusters pointed via logs.dir 
setting.
3. Delete the topic metadata from zookeeper : rmr /brokers (note I am not 
wiping off the whole zookeeper but the znode /brokers where the kafka broker 
ids and topic metadata is stored)
4. Restart the kafka cluster again.

One would expect that the kafka cluster will come up with no memory of any 
topic from previous.

But guess what, and this is the place where I need help and need to understand, 
when the kafka cluster comes back, it somehow is able to obtain the info of the 
previous topics. It promptly goes ahead creating and assigning 
partitions/replicas to the brokers for the previous topics. Now I am completely 
at loss to understand where exactly is kafka able to get the info of previous 
topics when I have wiped it off the zookeeper and also dropped the logs.dir 
locations across the kafka cluster.

An insight is much needed here. Where else is the topic meta data store which 
the kafka server is getting hold of after coming back alive?

Regards,
Virendra

Re: Delete topic API in 0.8.2

[Timothy Chen]

I believe that's the only way it's supported from the CLI.

Delete topic actually fully removes the topic from the cluster, which
also includes cleaning the logs and removing it from zookeeper (once
it is fully deleted).

Tim

On Fri, Jan 23, 2015 at 12:13 PM, Sumit Rangwala
 wrote:
> I am trying to find if there is a supported API to delete topic (from
> within my code) specifically in 0.8.2. One method that I can think of is
> calling kafka.admin.TopicCommand.main with the same parameters as one gives
> on the command line. Is this the recommended way or is there a better way
> of doing it?
>
> Furthermore, are the details of deleting a topic in 0.8.2 documented, since
> I would like to understand if delete topic only deletes the topic in
> zookeeper or it also cleans the logs in kafka broker as well.
>
>
> Sumit

Re: Delete topic API in 0.8.2

[Harsha]

Sumit,
   You can use AdminUtils.deleteTopic(zkClient, topicName) . This
   will initiate the deleteTopic process by zookeeper notification
   to KafkaController.deleteTopicManager.  It deletes log files
   along with zookeeper topic path as Timothy mentioned.
-Harsha

On Fri, Jan 23, 2015, at 12:18 PM, Timothy Chen wrote:
> I believe that's the only way it's supported from the CLI.
> 
> Delete topic actually fully removes the topic from the cluster, which
> also includes cleaning the logs and removing it from zookeeper (once
> it is fully deleted).
> 
> Tim
> 
> On Fri, Jan 23, 2015 at 12:13 PM, Sumit Rangwala
>  wrote:
> > I am trying to find if there is a supported API to delete topic (from
> > within my code) specifically in 0.8.2. One method that I can think of is
> > calling kafka.admin.TopicCommand.main with the same parameters as one gives
> > on the command line. Is this the recommended way or is there a better way
> > of doing it?
> >
> > Furthermore, are the details of deleting a topic in 0.8.2 documented, since
> > I would like to understand if delete topic only deletes the topic in
> > zookeeper or it also cleans the logs in kafka broker as well.
> >
> >
> > Sumit

Re: Delete topic API in 0.8.2

[Sumit Rangwala]

Thanks Harsha, exactly what I was looking for.

Sumit


On Fri, Jan 23, 2015 at 12:24 PM, Harsha  wrote:

> Sumit,
>You can use AdminUtils.deleteTopic(zkClient, topicName) . This
>will initiate the deleteTopic process by zookeeper notification
>to KafkaController.deleteTopicManager.  It deletes log files
>along with zookeeper topic path as Timothy mentioned.
> -Harsha
>
> On Fri, Jan 23, 2015, at 12:18 PM, Timothy Chen wrote:
> > I believe that's the only way it's supported from the CLI.
> >
> > Delete topic actually fully removes the topic from the cluster, which
> > also includes cleaning the logs and removing it from zookeeper (once
> > it is fully deleted).
> >
> > Tim
> >
> > On Fri, Jan 23, 2015 at 12:13 PM, Sumit Rangwala
> >  wrote:
> > > I am trying to find if there is a supported API to delete topic (from
> > > within my code) specifically in 0.8.2. One method that I can think of
> is
> > > calling kafka.admin.TopicCommand.main with the same parameters as one
> gives
> > > on the command line. Is this the recommended way or is there a better
> way
> > > of doing it?
> > >
> > > Furthermore, are the details of deleting a topic in 0.8.2 documented,
> since
> > > I would like to understand if delete topic only deletes the topic in
> > > zookeeper or it also cleans the logs in kafka broker as well.
> > >
> > >
> > > Sumit
>

Re: Delete topic / Recreate = No leader

[马哲超]

I've came accross the same issue with kafka 0.8.10.

2015-05-01 2:51 GMT+08:00 Dillian Murphey :

> I am trying to reproduce this. But if I create a topic, then delete it,
> then re-create it, no leader is getting assigned.
>
> I can still produce/consume messages (via command line, basic testing).
>
> Is there some additional cleanup I need to do?
>
> Thanks for your time!
>

RE: Delete topic / Recreate = No leader

[Aditya Auradkar]

I believe there were some issues with delete topic in 0.8.10 (other correct me 
if I am wrong).
Can you try with the most recent release ?

Thanks,
Aditya


From: 马哲超 [mazhechaomaill...@gmail.com]
Sent: Sunday, May 03, 2015 7:51 PM
To: users@kafka.apache.org
Subject: Re: Delete topic / Recreate = No leader

I've came accross the same issue with kafka 0.8.10.

2015-05-01 2:51 GMT+08:00 Dillian Murphey :

> I am trying to reproduce this. But if I create a topic, then delete it,
> then re-create it, no leader is getting assigned.
>
> I can still produce/consume messages (via command line, basic testing).
>
> Is there some additional cleanup I need to do?
>
> Thanks for your time!
>

can i delete topic like this ?

[Laxmi Narayan NIT DGP]

/bin/kafka-topics.sh --zookeeper localhost:9092 --delete --topic topicName


I am getting exception saying :

[2017-01-13 00:01:45,101] WARN Client session timed out, have not heard
from server in 15016ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
[2017-01-13 00:02:00,902] WARN Client session timed out, have not heard
from server in 15001ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
Exception in thread "main"
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
zookeeper server within timeout: 3
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:156)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:130)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:76)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
at kafka.admin.TopicCommand.main(TopicCommand.scala)





**

*Regards,*
*Laxmi Narayan Patel*
*MCA NIT Durgapur (2011-2014)*
*Mob:-9741292048,8345847473*

Re: Failed attempt to delete topic

[Mayuresh Gharat]

Can you paste some logs from the controller, when you deleted the topic?

Thanks,

Mayuresh

On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
rvidyadha...@gracenote.com> wrote:

> Hello,
>
> We are on an older kafka (0.8.1) version.  While a number of consumers
> were running, we attempted to delete a few topics using the kafka-topics.sh
> file (basically want to remove all messages in that topic and restart,
> since our entities went through some incompatible changes).  We noticed
> logs saying the topic has been queued for deletion.  After stopping all
> processes accessing kafka, we restarted kafka and then our processes.  The
> old topics do not seem to have been deleted (I can still see the log
> directories corresponding to the topics), and none of the clients are able
> to either publish or read to the topics that we attempted to delete.
> Attempting to read gives us the following type of error:
>
> Attempting to access an invalid KafkaTopic ( are you operating on a closed
> KafkaTopic ?)
>
> Attempting to publish gives us a more general type of error:
>
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at kafka.producer.Producer.send(Producer.scala:76)
>
> How can be get around this issue and start using the topics that we tried
> to clean up?  There may have been better ways to achieve what we wanted, if
> so please suggest recommendations as well.
>
> Thanks
> Rakesh
>
>


-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: Failed attempt to delete topic

[Stevo Slavić]

Delete was actually considered to be working since Kafka 0.8.2 (although
there are still not easily reproducible edge cases when it doesn't work
well even in in 0.8.2 or newer).
In 0.8.1 one could request topic to be deleted (request gets stored as
entry in ZooKeeper), because of presence of the request for topic to be
deleted topic would become unusable (cannot publish or read), but broker
would actually never (work on the request to) delete topic.

Maybe it will be enough to delete from ZooKeeper entry for the topic
deletion request under /admin/delete_topics to have topic usable again.

Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
broker should work with old clients so maybe you don't have to upgrade
client side immediately.

Kind regards,
Stevo Slavic.


On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat  wrote:

> Can you paste some logs from the controller, when you deleted the topic?
>
> Thanks,
>
> Mayuresh
>
> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
> rvidyadha...@gracenote.com> wrote:
>
> > Hello,
> >
> > We are on an older kafka (0.8.1) version.  While a number of consumers
> > were running, we attempted to delete a few topics using the
> kafka-topics.sh
> > file (basically want to remove all messages in that topic and restart,
> > since our entities went through some incompatible changes).  We noticed
> > logs saying the topic has been queued for deletion.  After stopping all
> > processes accessing kafka, we restarted kafka and then our processes.
> The
> > old topics do not seem to have been deleted (I can still see the log
> > directories corresponding to the topics), and none of the clients are
> able
> > to either publish or read to the topics that we attempted to delete.
> > Attempting to read gives us the following type of error:
> >
> > Attempting to access an invalid KafkaTopic ( are you operating on a
> closed
> > KafkaTopic ?)
> >
> > Attempting to publish gives us a more general type of error:
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> > at
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at kafka.producer.Producer.send(Producer.scala:76)
> >
> > How can be get around this issue and start using the topics that we tried
> > to clean up?  There may have been better ways to achieve what we wanted,
> if
> > so please suggest recommendations as well.
> >
> > Thanks
> > Rakesh
> >
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

Re: Failed attempt to delete topic

[Steve Robenalt]

Hi Rakesh,

Topic deletion didn't really work properly until 0.8.2. Here's a
stackoverflow link that summarizes how to work around this limitation:

http://stackoverflow.com/questions/24287900/delete-topic-in-kafka-0-8-1-1

HTH,
Steve

On Thu, Dec 3, 2015 at 3:33 PM, Mayuresh Gharat 
wrote:

> Can you paste some logs from the controller, when you deleted the topic?
>
> Thanks,
>
> Mayuresh
>
> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
> rvidyadha...@gracenote.com> wrote:
>
> > Hello,
> >
> > We are on an older kafka (0.8.1) version.  While a number of consumers
> > were running, we attempted to delete a few topics using the
> kafka-topics.sh
> > file (basically want to remove all messages in that topic and restart,
> > since our entities went through some incompatible changes).  We noticed
> > logs saying the topic has been queued for deletion.  After stopping all
> > processes accessing kafka, we restarted kafka and then our processes.
> The
> > old topics do not seem to have been deleted (I can still see the log
> > directories corresponding to the topics), and none of the clients are
> able
> > to either publish or read to the topics that we attempted to delete.
> > Attempting to read gives us the following type of error:
> >
> > Attempting to access an invalid KafkaTopic ( are you operating on a
> closed
> > KafkaTopic ?)
> >
> > Attempting to publish gives us a more general type of error:
> >
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> > at
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at kafka.producer.Producer.send(Producer.scala:76)
> >
> > How can be get around this issue and start using the topics that we tried
> > to clean up?  There may have been better ways to achieve what we wanted,
> if
> > so please suggest recommendations as well.
> >
> > Thanks
> > Rakesh
> >
> >
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>



-- 
Steve Robenalt
Software Architect
sroben...@highwire.org 
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Re: Failed attempt to delete topic

[Rakesh Vidyadharan]

Hi Mayuresh

These are some of the relevant logs that I could find

[2015-12-03 16:04:23,594] INFO Loading log 'merckx.raw.event.type-0' 
(kafka.log.LogManager)
[2015-12-03 16:04:23,595] INFO Completed load of log merckx.raw.event.type-0 
with log end offset 2 (kafka.log.Log)
[2015-12-03 16:04:23,641] INFO Loading log 'merckx.raw.event-0' 
(kafka.log.LogManager)
[2015-12-03 16:04:23,643] INFO Completed load of log merckx.raw.event-0 with 
log end offset 25149 (kafka.log.Log)
[2015-12-03 16:04:25,208] INFO [ReplicaFetcherManager on broker 0] Removed 
fetcher for partitions 
[merckx.raw.venue.capacity,0],[merckx.raw.sport.franchise,0],[merckx.raw.college,0],[merckx.raw.venue.formerName,0],[merckx.raw.venue,0],[merckx.raw.event,0],[merckx.raw.organization,0],[merckx.raw.sport.franchise.season,0],[merckx.raw.organization.season,0],[merckx.raw.sport,0],[merckx.raw.event.type,0]
 (kafka.server.ReplicaFetcherManager)
[2015-12-03 16:04:25,279] INFO Truncating log merckx.raw.event-0 to offset 
25149. (kafka.log.Log)
[2015-12-03 16:04:25,280] INFO Truncating log merckx.raw.event.type-0 to offset 
2. (kafka.log.Log)
[2015-12-03 16:09:04,709] ERROR [KafkaApi-0] Error while fetching metadata for 
partition [merckx.raw.event,0] (kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for partition 
[merckx.raw.event,0]
[2015-12-03 16:09:05,926] ERROR [KafkaApi-0] Error while fetching metadata for 
partition [merckx.raw.event,0] (kafka.server.KafkaApis)


Thanks

Rakesh




On 03/12/2015 17:33, "Mayuresh Gharat"  wrote:

>Can you paste some logs from the controller, when you deleted the topic?
>
>Thanks,
>
>Mayuresh
>
>On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
>rvidyadha...@gracenote.com> wrote:
>
>> Hello,
>>
>> We are on an older kafka (0.8.1) version.  While a number of consumers
>> were running, we attempted to delete a few topics using the kafka-topics.sh
>> file (basically want to remove all messages in that topic and restart,
>> since our entities went through some incompatible changes).  We noticed
>> logs saying the topic has been queued for deletion.  After stopping all
>> processes accessing kafka, we restarted kafka and then our processes.  The
>> old topics do not seem to have been deleted (I can still see the log
>> directories corresponding to the topics), and none of the clients are able
>> to either publish or read to the topics that we attempted to delete.
>> Attempting to read gives us the following type of error:
>>
>> Attempting to access an invalid KafkaTopic ( are you operating on a closed
>> KafkaTopic ?)
>>
>> Attempting to publish gives us a more general type of error:
>>
>> kafka.common.FailedToSendMessageException: Failed to send messages after 3
>> tries.
>> at
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>> at kafka.producer.Producer.send(Producer.scala:76)
>>
>> How can be get around this issue and start using the topics that we tried
>> to clean up?  There may have been better ways to achieve what we wanted, if
>> so please suggest recommendations as well.
>>
>> Thanks
>> Rakesh
>>
>>
>
>
>-- 
>-Regards,
>Mayuresh R. Gharat
>(862) 250-7125

Re: Failed attempt to delete topic

[Rakesh Vidyadharan]

Thanks Stevo.  I did see some messages related to /admin/delete_topics.  Will 
do some research on how I can clean up zookeeper.

Thanks
Rakesh




On 03/12/2015 17:55, "Stevo Slavić"  wrote:

>Delete was actually considered to be working since Kafka 0.8.2 (although
>there are still not easily reproducible edge cases when it doesn't work
>well even in in 0.8.2 or newer).
>In 0.8.1 one could request topic to be deleted (request gets stored as
>entry in ZooKeeper), because of presence of the request for topic to be
>deleted topic would become unusable (cannot publish or read), but broker
>would actually never (work on the request to) delete topic.
>
>Maybe it will be enough to delete from ZooKeeper entry for the topic
>deletion request under /admin/delete_topics to have topic usable again.
>
>Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
>broker should work with old clients so maybe you don't have to upgrade
>client side immediately.
>
>Kind regards,
>Stevo Slavic.
>
>
>On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat > wrote:
>
>> Can you paste some logs from the controller, when you deleted the topic?
>>
>> Thanks,
>>
>> Mayuresh
>>
>> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
>> rvidyadha...@gracenote.com> wrote:
>>
>> > Hello,
>> >
>> > We are on an older kafka (0.8.1) version.  While a number of consumers
>> > were running, we attempted to delete a few topics using the
>> kafka-topics.sh
>> > file (basically want to remove all messages in that topic and restart,
>> > since our entities went through some incompatible changes).  We noticed
>> > logs saying the topic has been queued for deletion.  After stopping all
>> > processes accessing kafka, we restarted kafka and then our processes.
>> The
>> > old topics do not seem to have been deleted (I can still see the log
>> > directories corresponding to the topics), and none of the clients are
>> able
>> > to either publish or read to the topics that we attempted to delete.
>> > Attempting to read gives us the following type of error:
>> >
>> > Attempting to access an invalid KafkaTopic ( are you operating on a
>> closed
>> > KafkaTopic ?)
>> >
>> > Attempting to publish gives us a more general type of error:
>> >
>> > kafka.common.FailedToSendMessageException: Failed to send messages after
>> 3
>> > tries.
>> > at
>> >
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>> > at kafka.producer.Producer.send(Producer.scala:76)
>> >
>> > How can be get around this issue and start using the topics that we tried
>> > to clean up?  There may have been better ways to achieve what we wanted,
>> if
>> > so please suggest recommendations as well.
>> >
>> > Thanks
>> > Rakesh
>> >
>> >
>>
>>
>> --
>> -Regards,
>> Mayuresh R. Gharat
>> (862) 250-7125
>>

Re: Failed attempt to delete topic

[Mayuresh Gharat]

you can use the zookeeper shell inside the bin directory for that.

Thanks,

Mayuresh

On Thu, Dec 3, 2015 at 4:04 PM, Rakesh Vidyadharan <
rvidyadha...@gracenote.com> wrote:

> Thanks Stevo.  I did see some messages related to /admin/delete_topics.
> Will do some research on how I can clean up zookeeper.
>
> Thanks
> Rakesh
>
>
>
>
> On 03/12/2015 17:55, "Stevo Slavić"  wrote:
>
> >Delete was actually considered to be working since Kafka 0.8.2 (although
> >there are still not easily reproducible edge cases when it doesn't work
> >well even in in 0.8.2 or newer).
> >In 0.8.1 one could request topic to be deleted (request gets stored as
> >entry in ZooKeeper), because of presence of the request for topic to be
> >deleted topic would become unusable (cannot publish or read), but broker
> >would actually never (work on the request to) delete topic.
> >
> >Maybe it will be enough to delete from ZooKeeper entry for the topic
> >deletion request under /admin/delete_topics to have topic usable again.
> >
> >Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
> >broker should work with old clients so maybe you don't have to upgrade
> >client side immediately.
> >
> >Kind regards,
> >Stevo Slavic.
> >
> >
> >On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat <
> gharatmayures...@gmail.com
> >> wrote:
> >
> >> Can you paste some logs from the controller, when you deleted the topic?
> >>
> >> Thanks,
> >>
> >> Mayuresh
> >>
> >> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
> >> rvidyadha...@gracenote.com> wrote:
> >>
> >> > Hello,
> >> >
> >> > We are on an older kafka (0.8.1) version.  While a number of consumers
> >> > were running, we attempted to delete a few topics using the
> >> kafka-topics.sh
> >> > file (basically want to remove all messages in that topic and restart,
> >> > since our entities went through some incompatible changes).  We
> noticed
> >> > logs saying the topic has been queued for deletion.  After stopping
> all
> >> > processes accessing kafka, we restarted kafka and then our processes.
> >> The
> >> > old topics do not seem to have been deleted (I can still see the log
> >> > directories corresponding to the topics), and none of the clients are
> >> able
> >> > to either publish or read to the topics that we attempted to delete.
> >> > Attempting to read gives us the following type of error:
> >> >
> >> > Attempting to access an invalid KafkaTopic ( are you operating on a
> >> closed
> >> > KafkaTopic ?)
> >> >
> >> > Attempting to publish gives us a more general type of error:
> >> >
> >> > kafka.common.FailedToSendMessageException: Failed to send messages
> after
> >> 3
> >> > tries.
> >> > at
> >> >
> >>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> >> > at kafka.producer.Producer.send(Producer.scala:76)
> >> >
> >> > How can be get around this issue and start using the topics that we
> tried
> >> > to clean up?  There may have been better ways to achieve what we
> wanted,
> >> if
> >> > so please suggest recommendations as well.
> >> >
> >> > Thanks
> >> > Rakesh
> >> >
> >> >
> >>
> >>
> >> --
> >> -Regards,
> >> Mayuresh R. Gharat
> >> (862) 250-7125
> >>
>



-- 
-Regards,
Mayuresh R. Gharat
(862) 250-7125

Re: Failed attempt to delete topic

[Rakesh Vidyadharan]

Thanks Mayuresh.  I was able to use the shell to delete the entries and things 
are working fine now.




On 03/12/2015 18:22, "Mayuresh Gharat"  wrote:

>you can use the zookeeper shell inside the bin directory for that.
>
>Thanks,
>
>Mayuresh
>
>On Thu, Dec 3, 2015 at 4:04 PM, Rakesh Vidyadharan <
>rvidyadha...@gracenote.com> wrote:
>
>> Thanks Stevo.  I did see some messages related to /admin/delete_topics.
>> Will do some research on how I can clean up zookeeper.
>>
>> Thanks
>> Rakesh
>>
>>
>>
>>
>> On 03/12/2015 17:55, "Stevo Slavić"  wrote:
>>
>> >Delete was actually considered to be working since Kafka 0.8.2 (although
>> >there are still not easily reproducible edge cases when it doesn't work
>> >well even in in 0.8.2 or newer).
>> >In 0.8.1 one could request topic to be deleted (request gets stored as
>> >entry in ZooKeeper), because of presence of the request for topic to be
>> >deleted topic would become unusable (cannot publish or read), but broker
>> >would actually never (work on the request to) delete topic.
>> >
>> >Maybe it will be enough to delete from ZooKeeper entry for the topic
>> >deletion request under /admin/delete_topics to have topic usable again.
>> >
>> >Otherwise, just upgrade broker side to 0.8.2.x or latest 0.9.0.0 - new
>> >broker should work with old clients so maybe you don't have to upgrade
>> >client side immediately.
>> >
>> >Kind regards,
>> >Stevo Slavic.
>> >
>> >
>> >On Fri, Dec 4, 2015 at 12:33 AM, Mayuresh Gharat <
>> gharatmayures...@gmail.com
>> >> wrote:
>> >
>> >> Can you paste some logs from the controller, when you deleted the topic?
>> >>
>> >> Thanks,
>> >>
>> >> Mayuresh
>> >>
>> >> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan <
>> >> rvidyadha...@gracenote.com> wrote:
>> >>
>> >> > Hello,
>> >> >
>> >> > We are on an older kafka (0.8.1) version.  While a number of consumers
>> >> > were running, we attempted to delete a few topics using the
>> >> kafka-topics.sh
>> >> > file (basically want to remove all messages in that topic and restart,
>> >> > since our entities went through some incompatible changes).  We
>> noticed
>> >> > logs saying the topic has been queued for deletion.  After stopping
>> all
>> >> > processes accessing kafka, we restarted kafka and then our processes.
>> >> The
>> >> > old topics do not seem to have been deleted (I can still see the log
>> >> > directories corresponding to the topics), and none of the clients are
>> >> able
>> >> > to either publish or read to the topics that we attempted to delete.
>> >> > Attempting to read gives us the following type of error:
>> >> >
>> >> > Attempting to access an invalid KafkaTopic ( are you operating on a
>> >> closed
>> >> > KafkaTopic ?)
>> >> >
>> >> > Attempting to publish gives us a more general type of error:
>> >> >
>> >> > kafka.common.FailedToSendMessageException: Failed to send messages
>> after
>> >> 3
>> >> > tries.
>> >> > at
>> >> >
>> >>
>> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
>> >> > at kafka.producer.Producer.send(Producer.scala:76)
>> >> >
>> >> > How can be get around this issue and start using the topics that we
>> tried
>> >> > to clean up?  There may have been better ways to achieve what we
>> wanted,
>> >> if
>> >> > so please suggest recommendations as well.
>> >> >
>> >> > Thanks
>> >> > Rakesh
>> >> >
>> >> >
>> >>
>> >>
>> >> --
>> >> -Regards,
>> >> Mayuresh R. Gharat
>> >> (862) 250-7125
>> >>
>>
>
>
>
>-- 
>-Regards,
>Mayuresh R. Gharat
>(862) 250-7125

Re: kafka 0.8.2.2 delete topic issue

[John Holland]

I just ran into this issue in our load environment, unfortunately I came up
with the same options outlined above.  Any better solutions would be most
appreciated otherwise I'm now considering the use of delete topic in any
critical environment off the table.

On Wed, Feb 3, 2016 at 10:10 AM Ivan Dyachkov  wrote:

> Hi!
>
> We're running cluster of 3 kafka-0.8.2.2 nodes and delete.topic.enable is
> set to true on all nodes.
>
> Today we tried to delete one of the topics. I waited ~20 minutes after the
> kafka-topics.sh --delete was executed but the topic was still there.
>
> --describe showed Leader: -1 and only one of three brokers (1, 2 or 3) in
> ISR for all partitions.
>
> The topic had the following configuration:
> partitions: 50
> replication factor: 3
> unclean.leader.election.enable: false
> min.insync.replicas: 2
>
> Then I restarted one of the nodes with the hope to trigger retry on topic
> deletion. After that things went nuts and we had short cluster downtime
> before I managed to recover it and get to the same point when we were
> before the restart. The topic was still there.
>
> Since we successfully deleted topics before on the same cluster some time
> ago, the only thing which could be different is someone could hold an open
> connection or a consumer group linked to the topic. According to previous
> similar issues described in this group this could make topic deletion to
> fail. We had shut down the service responsible for this topic, triggered
> preferred replica election, but that didn't help. Also we set
> retention.bytes topic config to 1, waited for retention.interval.ms,
> nothing changed.
>
> The only option to recover which we see is shut down all nodes, delete the
> topic from file system, remove the topic from zookeeper on /config/topics/
> and /admin/delete_topics and start the nodes again.
>
> So my question is really is it possible to recover from this situation,
> either cancel topic deletion or finally delete it, without cluster downtime?
>
> Thanks.
>
> /Ivan
>

Can we delete topic in kafka

[Snehalata Nagaje]

Hi , 

Can we delete certain topic in kafka? 

I have deleted using command 

./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181 

It says topic marked as deletion, but it does not actually delete topic. 

Thanks, 
Snehalata 

Re: Kafka 0.8: delete topic with data

[Neha Narkhede]

No, delete topic support doesn't exist in 0.8 and the inclusion of
DeleteTopicCommand was overlooked when we cut the release. So using that
command can cause unexpected issues in the cluster and we don't recommend
you use it. Delete topic is available in beta in the upcoming 0.8.1 release
but I think it will be stable only in 0.8.2.

Thanks,
Neha


On Thu, Feb 27, 2014 at 1:16 AM, Yury Ruchin  wrote:

> Hi,
>
> I'm using Kafka 0.8 which does not have a command to delete topic. However,
> I need the functionality and I'm trying to adopt this approach:
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/DeleteTopicCommand.scala
> .
> I see it simply deletes the topic node from ZK. My question is what will
> happen with the topic data after removing the node? Will Kafka brokers be
> able to detect that topic does not exist anymore and purge its data?
>
> Thanks,
> Yury
>

How to delete Topic in kafka 0.8.1 ?

[rafeeq s]

I am using Kafka 0.8.1 and I tried to execute Topic delete command but
still Topic list showing that specific Topic.

Topic Delete commad which i used:

*bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
my_topic_name*

How can i verify that my_topic is deleted ?

Is there specific steps to delete Topic in Kafka 0.8.1 ?

Thanks in advance!

Re: Unable to delete topic - kafka 0.8.0

[Guozhang Wang]

Hello Virendra,

Did you have any producer/consumer clients running during the whole process?

Guozhang


On Wed, Jun 25, 2014 at 11:53 PM, Virendra Pratap Singh <
vpsi...@yahoo-inc.com.invalid> wrote:

> I am aware of lack of programmatic way of deleting topics in kafka 0.8.0.
> So using the sledge hammer approach.
> This is what I am doing:
>
> 1. Bring whole of my kafka cluster down.
> 2. Delete all the content on all the kafka clusters pointed via logs.dir
> setting.
> 3. Delete the topic metadata from zookeeper : rmr /brokers (note I am not
> wiping off the whole zookeeper but the znode /brokers where the kafka
> broker ids and topic metadata is stored)
> 4. Restart the kafka cluster again.
>
> One would expect that the kafka cluster will come up with no memory of any
> topic from previous.
>
> But guess what, and this is the place where I need help and need to
> understand, when the kafka cluster comes back, it somehow is able to obtain
> the info of the previous topics. It promptly goes ahead creating and
> assigning partitions/replicas to the brokers for the previous topics. Now I
> am completely at loss to understand where exactly is kafka able to get the
> info of previous topics when I have wiped it off the zookeeper and also
> dropped the logs.dir locations across the kafka cluster.
>
> An insight is much needed here. Where else is the topic meta data store
> which the kafka server is getting hold of after coming back alive?
>
> Regards,
> Virendra
>
>


-- 
-- Guozhang

Re: Unable to delete topic - kafka 0.8.0

[Neha Narkhede]

Firstly, I'm not sure this process of deleting topics completely works,
especially in corner cases. Having said that, for #3, you should just be
deleting /brokers/topics/. If producers are sending
data to the Kafka cluster, it will recreate the topics once the brokers
come up.

Thanks,
Neha


On Wed, Jun 25, 2014 at 11:53 PM, Virendra Pratap Singh <
vpsi...@yahoo-inc.com.invalid> wrote:

> I am aware of lack of programmatic way of deleting topics in kafka 0.8.0.
> So using the sledge hammer approach.
> This is what I am doing:
>
> 1. Bring whole of my kafka cluster down.
> 2. Delete all the content on all the kafka clusters pointed via logs.dir
> setting.
> 3. Delete the topic metadata from zookeeper : rmr /brokers (note I am not
> wiping off the whole zookeeper but the znode /brokers where the kafka
> broker ids and topic metadata is stored)
> 4. Restart the kafka cluster again.
>
> One would expect that the kafka cluster will come up with no memory of any
> topic from previous.
>
> But guess what, and this is the place where I need help and need to
> understand, when the kafka cluster comes back, it somehow is able to obtain
> the info of the previous topics. It promptly goes ahead creating and
> assigning partitions/replicas to the brokers for the previous topics. Now I
> am completely at loss to understand where exactly is kafka able to get the
> info of previous topics when I have wiped it off the zookeeper and also
> dropped the logs.dir locations across the kafka cluster.
>
> An insight is much needed here. Where else is the topic meta data store
> which the kafka server is getting hold of after coming back alive?
>
> Regards,
> Virendra
>
>

Re: Unable to delete topic - kafka 0.8.0

[Virendra Pratap Singh]

No producers were active. The only producer in our pipeline is a storm
topology and had made it a point that the whole cluster was down and no
topology running.
What I did was nothing short of doing a fresh kafka cluster setup (barring
the fact that I didn¹t wiped the zookeepers as its the same setup which
our storm uses. But given the fact that I had dropped /brokers, it should
not be concern).

So the question is where exactly is kafka brokers getting the info about
the topic to recreate when they come up when there is no producers and all
previous info has been wiped out.

Regards,
Virendra

On 6/26/14, 9:29 AM, "Neha Narkhede"  wrote:

>Firstly, I'm not sure this process of deleting topics completely works,
>especially in corner cases. Having said that, for #3, you should just be
>deleting /brokers/topics/. If producers are sending
>data to the Kafka cluster, it will recreate the topics once the brokers
>come up.
>
>Thanks,
>Neha
>
>
>On Wed, Jun 25, 2014 at 11:53 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> I am aware of lack of programmatic way of deleting topics in kafka
>>0.8.0.
>> So using the sledge hammer approach.
>> This is what I am doing:
>>
>> 1. Bring whole of my kafka cluster down.
>> 2. Delete all the content on all the kafka clusters pointed via logs.dir
>> setting.
>> 3. Delete the topic metadata from zookeeper : rmr /brokers (note I am
>>not
>> wiping off the whole zookeeper but the znode /brokers where the kafka
>> broker ids and topic metadata is stored)
>> 4. Restart the kafka cluster again.
>>
>> One would expect that the kafka cluster will come up with no memory of
>>any
>> topic from previous.
>>
>> But guess what, and this is the place where I need help and need to
>> understand, when the kafka cluster comes back, it somehow is able to
>>obtain
>> the info of the previous topics. It promptly goes ahead creating and
>> assigning partitions/replicas to the brokers for the previous topics.
>>Now I
>> am completely at loss to understand where exactly is kafka able to get
>>the
>> info of previous topics when I have wiped it off the zookeeper and also
>> dropped the logs.dir locations across the kafka cluster.
>>
>> An insight is much needed here. Where else is the topic meta data store
>> which the kafka server is getting hold of after coming back alive?
>>
>> Regards,
>> Virendra
>>
>>

Re: Unable to delete topic - kafka 0.8.0

[Virendra Pratap Singh]

I am 100% sure nothing was running. However I am not sure of consumers.
Would that make any difference?
I thought producer message write request could only cause kafka to
initiate topic creation, not consumer read. Would consumer read request
even succeed if the topic metadata is non-existent in zookeeper.
Since I brought the kafka brokers down, and deleted all logs and zookeeper
metadata, I would assume consumers will sync with that info and fail, as
now the topic is nonexistent. Not sure how consumer can impact in this
situation.

Compared to clean install, the only thing I see is zookeepers not wiped
off and reinstalled/restarted. But I believe that¹s should not be an issue
as I dropped the /brokers znode.

Virendra

On 6/26/14, 9:28 AM, "Guozhang Wang"  wrote:

>Hello Virendra,
>
>Did you have any producer/consumer clients running during the whole
>process?
>
>Guozhang
>
>
>On Wed, Jun 25, 2014 at 11:53 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> I am aware of lack of programmatic way of deleting topics in kafka
>>0.8.0.
>> So using the sledge hammer approach.
>> This is what I am doing:
>>
>> 1. Bring whole of my kafka cluster down.
>> 2. Delete all the content on all the kafka clusters pointed via logs.dir
>> setting.
>> 3. Delete the topic metadata from zookeeper : rmr /brokers (note I am
>>not
>> wiping off the whole zookeeper but the znode /brokers where the kafka
>> broker ids and topic metadata is stored)
>> 4. Restart the kafka cluster again.
>>
>> One would expect that the kafka cluster will come up with no memory of
>>any
>> topic from previous.
>>
>> But guess what, and this is the place where I need help and need to
>> understand, when the kafka cluster comes back, it somehow is able to
>>obtain
>> the info of the previous topics. It promptly goes ahead creating and
>> assigning partitions/replicas to the brokers for the previous topics.
>>Now I
>> am completely at loss to understand where exactly is kafka able to get
>>the
>> info of previous topics when I have wiped it off the zookeeper and also
>> dropped the logs.dir locations across the kafka cluster.
>>
>> An insight is much needed here. Where else is the topic meta data store
>> which the kafka server is getting hold of after coming back alive?
>>
>> Regards,
>> Virendra
>>
>>
>
>
>-- 
>-- Guozhang

pushing delete topic feature out of 0.8

[Jun Rao]

Hi,

We started the work on deleting topics in 0.8 (kafka-330). We realized that
it touches quite a few critical components such as controller, replica
manager, and log, and it will take some time to stabilize this. In order
not to delay the 0.8 release too much, I propose that we push this feature
out of 0.8. Since we don't really support deleting topics in 0.7, this
doesn't reduce the existing features.

Any concerns from people?

Thanks,

Jun

unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

So far, I have been unable to get delete topic to work, with release
candidate 2 for 0.8.2.

It worked ok when I ran it in the debugger locally, on a single node
instance. But when I run it in our staging environment, it is not
successfully even marking the topic for delete, for some reason.

I am setting delete.topic.enable to true (and see the confirmation of this
in the startup logs, e.g.):

INFO [main] utils.VerifiableProperties - Property delete.topic.enable
is overridden to true

I run this command:

java -cp app.jar kafka.admin.TopicCommand --zookeeper
myzkconnect:12345/mynamespace --delete --topic mytopic

log4j:WARN No appenders could be found for logger
(org.I0Itec.zkclient.ZkConnection).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN No appenders could be found for logger
(org.I0Itec.zkclient.ZkEventThread).
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info.
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info.
Topic mytopic is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

I then do a —list which should at least show the topic marked for deletion:

java -cp app.jar kafka.admin.TopicCommand --zookeeper
myzkconnect:12345/mynamespace --list --topic mytopic

log4j:WARN No appenders could be found for logger
(org.I0Itec.zkclient.ZkConnection).
log4j:WARN No appenders could be found for logger
(org.I0Itec.zkclient.ZkEventThread).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info.
mytopic

Note, it doesn’t list it as ‘marked for deletion’. Furthermore, waiting
multiple hours still doesn’t result in the topic being deleted.

The topic has 1 partition, 2 replicas, and no data stored.

In zookeeper, the /admin/deleted_topics/ path is empty.

The zookeeper code looks pretty straightforward, but for some reason is not
writing the deleted_topics path.  We are running zookeeper 3.4.6.

Thoughts?

Jason
​

Delete topic functionality can't use in 0.8.1

[Ryco Xiao]

when I exec the delete command,return information is below:
It mark the kafka-topic.sh not support the delete parameter.
my package is compiled by myself.


​

Delete topic using Admintools is not working

[Sivananda Reddy]

Hi,

As per the jira https://issues.apache.org/jira/browse/KAFKA-1737, I have
set ZKStringSerializer, while creating the ZkClient and still the delete
topic function call is not working:

Version: kafka_2.11-0.8.2.1

*My code looks like this*:
ZkClient zkClient = new ZkClient("localhost:2181", 6000, 6000,
ZKStringSerializer$.MODULE$);
AdminUtils.deleteTopic(zkClient, "testTopic17");

*I am getting the following exception's*:
Exception in thread "main"
org.I0Itec.zkclient.exception.ZkNodeExistsException:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /admin/delete_topics/testTopic17
at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:55)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.createPersistent(ZkClient.java:243)
at kafka.utils.ZkUtils$.createPersistentPath(ZkUtils.scala:306)
at kafka.admin.AdminUtils$.deleteTopic(AdminUtils.scala:159)
at kafka.admin.AdminUtils.deleteTopic(AdminUtils.scala)
at Test.deleteTopic(Test.java:98)
at Test.main(Test.java:76)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException:
KeeperErrorCode = NodeExists for /admin/delete_topics/testTopic17
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:87)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:308)
at org.I0Itec.zkclient.ZkClient$1.call(ZkClient.java:304)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
... 7 more

*The corresponding zookeeper logs*:
[2015-07-16 15:31:14,383] INFO Accepted socket connection from /
127.0.0.1:23363 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2015-07-16 15:31:14,385] INFO Client attempting to establish new session
at /127.0.0.1:23363 (org.apache.zookeeper.server.ZooKeeperServer)
[2015-07-16 15:31:14,391] INFO Established session 0x14e98d2ce98000c with
negotiated timeout 6000 for client /127.0.0.1:23363
(org.apache.zookeeper.server.ZooKeeperServer)
[2015-07-16 15:31:14,510] INFO Got user-level KeeperException when
processing sessionid:0x14e98d2ce98000c type:create cxid:0x1 zxid:0x963
txntype:-1 reqpath:n/a Error Path:/admin/delete_topics/testTopic17
Error:KeeperErrorCode = NodeExists for /admin/delete_topics/testTopic17
(org.apache.zookeeper.server.PrepRequestProcessor)
[2015-07-16 15:31:14,842] WARN caught end of stream exception
(org.apache.zookeeper.server.NIOServerCnxn)
EndOfStreamException: Unable to read additional data from client sessionid
0x14e98d2ce98000c, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:744)
[2015-07-16 15:31:14,843] INFO Closed socket connection for client /
127.0.0.1:23363 which had sessionid 0x14e98d2ce98000c
(org.apache.zookeeper.server.NIOServerCnxn)
[2015-07-16 15:31:21,000] INFO Expiring session 0x14e98d2ce98000c, timeout
of 6000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2015-07-16 15:31:21,001] INFO Processed session termination for sessionid:
0x14e98d2ce98000c (org.apache.zookeeper.server.PrepRequestProcessor)

Could someone please let me know if I am missing anything?

Thank you,
Siva.

Re: can i delete topic like this ?

[Kaufman Ng]

Your zookeeper url doesn't look right.  Port 9092 is kafka broker's default
listening port.  Zookeeper's default is 2181.


On Thu, Jan 12, 2017 at 1:33 PM, Laxmi Narayan NIT DGP  wrote:

> /bin/kafka-topics.sh --zookeeper localhost:9092 --delete --topic topicName
>
>
> I am getting exception saying :
>
> [2017-01-13 00:01:45,101] WARN Client session timed out, have not heard
> from server in 15016ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
> [2017-01-13 00:02:00,902] WARN Client session timed out, have not heard
> from server in 15001ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)
> Exception in thread "main"
> org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
> zookeeper server within timeout: 3
> at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
> at org.I0Itec.zkclient.ZkClient.(ZkClient.java:156)
> at org.I0Itec.zkclient.ZkClient.(ZkClient.java:130)
> at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:76)
> at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
> at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
> at kafka.admin.TopicCommand.main(TopicCommand.scala)
>
>
>
>
>
> **
>
> *Regards,*
> *Laxmi Narayan Patel*
> *MCA NIT Durgapur (2011-2014)*
> *Mob:-9741292048,8345847473*
>



-- 
Kaufman Ng
+1 646 961 8063
Solutions Architect | Confluent | www.confluent.io

Re: can i delete topic like this ?

[Laxmi Narayan NIT DGP]

Hi ,
If my topic is not enabled for deletion any other way to purge message from
topic ?



*Regards,*
*Laxmi Narayan Patel*
*MCA NIT Durgapur (2011-2014)*
*Mob:-9741292048,8345847473*

On Fri, Jan 13, 2017 at 12:16 AM, Kaufman Ng  wrote:

> Your zookeeper url doesn't look right.  Port 9092 is kafka broker's default
> listening port.  Zookeeper's default is 2181.
>
>
> On Thu, Jan 12, 2017 at 1:33 PM, Laxmi Narayan NIT DGP <
> nit.dgp...@gmail.com
> > wrote:
>
> > /bin/kafka-topics.sh --zookeeper localhost:9092 --delete --topic
> topicName
> >
> >
> > I am getting exception saying :
> >
> > [2017-01-13 00:01:45,101] WARN Client session timed out, have not heard
> > from server in 15016ms for sessionid 0x0 (org.apache.zookeeper.
> ClientCnxn)
> > [2017-01-13 00:02:00,902] WARN Client session timed out, have not heard
> > from server in 15001ms for sessionid 0x0 (org.apache.zookeeper.
> ClientCnxn)
> > Exception in thread "main"
> > org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to
> > zookeeper server within timeout: 3
> > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:156)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:130)
> > at kafka.utils.ZkUtils$.createZkClientAndConnection(
> ZkUtils.scala:76)
> > at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
> > at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
> > at kafka.admin.TopicCommand.main(TopicCommand.scala)
> >
> >
> >
> >
> >
> > **
> >
> > *Regards,*
> > *Laxmi Narayan Patel*
> > *MCA NIT Durgapur (2011-2014)*
> > *Mob:-9741292048,8345847473*
> >
>
>
>
> --
> Kaufman Ng
> +1 646 961 8063
> Solutions Architect | Confluent | www.confluent.io
>

RE: can i delete topic like this ?

[Tauzell, Dave]

You can set the retention for the topic to a small time and then wait for Kafka 
to delete the messages before setting it back:

bin/kafka-topics.sh --zookeeper zk.prod.yoursite.com --alter --topic TOPIC_NAME 
--config retention.ms=1000

-Original Message-
From: Laxmi Narayan NIT DGP [mailto:nit.dgp...@gmail.com]
Sent: Thursday, January 12, 2017 12:52 PM
To: users@kafka.apache.org
Subject: Re: can i delete topic like this ?

Hi ,
If my topic is not enabled for deletion any other way to purge message from 
topic ?



*Regards,*
*Laxmi Narayan Patel*
*MCA NIT Durgapur (2011-2014)*
*Mob:-9741292048,8345847473*

On Fri, Jan 13, 2017 at 12:16 AM, Kaufman Ng  wrote:

> Your zookeeper url doesn't look right.  Port 9092 is kafka broker's
> default listening port.  Zookeeper's default is 2181.
>
>
> On Thu, Jan 12, 2017 at 1:33 PM, Laxmi Narayan NIT DGP <
> nit.dgp...@gmail.com
> > wrote:
>
> > /bin/kafka-topics.sh --zookeeper localhost:9092 --delete --topic
> topicName
> >
> >
> > I am getting exception saying :
> >
> > [2017-01-13 00:01:45,101] WARN Client session timed out, have not
> > heard from server in 15016ms for sessionid 0x0 (org.apache.zookeeper.
> ClientCnxn)
> > [2017-01-13 00:02:00,902] WARN Client session timed out, have not
> > heard from server in 15001ms for sessionid 0x0 (org.apache.zookeeper.
> ClientCnxn)
> > Exception in thread "main"
> > org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
> > to zookeeper server within timeout: 3
> > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:156)
> > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:130)
> > at kafka.utils.ZkUtils$.createZkClientAndConnection(
> ZkUtils.scala:76)
> > at kafka.utils.ZkUtils$.apply(ZkUtils.scala:58)
> > at kafka.admin.TopicCommand$.main(TopicCommand.scala:53)
> > at kafka.admin.TopicCommand.main(TopicCommand.scala)
> >
> >
> >
> >
> >
> > **
> >
> > *Regards,*
> > *Laxmi Narayan Patel*
> > *MCA NIT Durgapur (2011-2014)*
> > *Mob:-9741292048,8345847473*
> >
>
>
>
> --
> Kaufman Ng
> +1 646 961 8063
> Solutions Architect | Confluent | www.confluent.io
>
This e-mail and any files transmitted with it are confidential, may contain 
sensitive information, and are intended solely for the use of the individual or 
entity to whom they are addressed. If you have received this e-mail in error, 
please notify the sender by reply e-mail immediately and destroy all copies of 
the e-mail and any attachments.

Re: Can we delete topic in kafka

[Jörg Wagner]

Depending on your version of kafka it may or may not work.

Before 0.8.2 it didn't work afaik and on 0.8.2 it works unrealiably from 
my experience. Can't comment on 0.9 forward.


Cheers

On 11.05.2016 09:48, Snehalata Nagaje wrote:


Hi ,

Can we delete certain topic in kafka?

I have deleted using command

./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181

It says topic marked as deletion, but it does not actually delete topic.

Thanks,
Snehalata



--
Mit freundlichem Gruß

Jörg Wagner

Systemadministrator
Search & Account Security

1&1 Mail & Media Development  & Technology GmbH | Sapporobogen 6-8 | 80637 
München | Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagn...@1und1.de | Web: www.1und1.de

Amtsgericht Montabaur, HRB 5452

Geschäftsführer: Frank Einhellinger, Thomas Ludwig, Jan Oetjen


Member of United Internet

Re: Can we delete topic in kafka

[Jan Omar]

You have to allow topic deletion in server.properties first.

delete.topic.enable = true

Regards

Jan

> On 11 May 2016, at 09:48, Snehalata Nagaje 
>  wrote:
> 
> 
> 
> Hi , 
> 
> Can we delete certain topic in kafka? 
> 
> I have deleted using command 
> 
> ./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181 
> 
> It says topic marked as deletion, but it does not actually delete topic. 
> 
> Thanks, 
> Snehalata 

Re: Can we delete topic in kafka

[Snehalata Nagaje]

I am using 0.9.0.1

- Original Message -
From: "Jörg Wagner" 
To: users@kafka.apache.org
Sent: Wednesday, May 11, 2016 1:20:37 PM
Subject: Re: Can we delete topic in kafka

Depending on your version of kafka it may or may not work.

Before 0.8.2 it didn't work afaik and on 0.8.2 it works unrealiably from 
my experience. Can't comment on 0.9 forward.

Cheers

On 11.05.2016 09:48, Snehalata Nagaje wrote:
>
> Hi ,
>
> Can we delete certain topic in kafka?
>
> I have deleted using command
>
> ./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181
>
> It says topic marked as deletion, but it does not actually delete topic.
>
> Thanks,
> Snehalata
>

-- 
Mit freundlichem Gruß

Jörg Wagner

Systemadministrator
Search & Account Security

1&1 Mail & Media Development  & Technology GmbH | Sapporobogen 6-8 | 80637 
München | Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagn...@1und1.de | Web: www.1und1.de

Amtsgericht Montabaur, HRB 5452

Geschäftsführer: Frank Einhellinger, Thomas Ludwig, Jan Oetjen


Member of United Internet

Re: Can we delete topic in kafka

[Portal Operations]

If it’s marked, he configured it. I recently tried it and it doesn’t mark it 
otherwise.




Am 11.05.16, 09:51 schrieb "Jan Omar" :

>You have to allow topic deletion in server.properties first.
>
>delete.topic.enable = true
>
>Regards
>
>Jan
>
>> On 11 May 2016, at 09:48, Snehalata Nagaje 
>>  wrote:
>> 
>> 
>> 
>> Hi , 
>> 
>> Can we delete certain topic in kafka? 
>> 
>> I have deleted using command 
>> 
>> ./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181 
>> 
>> It says topic marked as deletion, but it does not actually delete topic. 
>> 
>> Thanks, 
>> Snehalata 
>

Re: Can we delete topic in kafka

[Eduardo Costa Alfaia]

Hi,
It’s better creating a script that delete the kafka folder where exist the 
kafka topic and after create it again if need.

BR


Eduardo Costa Alfaia
Ph.D. Student in Telecommunications Engineering
Università degli Studi di Brescia
Tel: +39 3209333018








On 5/11/16, 09:48, "Snehalata Nagaje"  
wrote:

>
>
>Hi , 
>
>Can we delete certain topic in kafka? 
>
>I have deleted using command 
>
>./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181 
>
>It says topic marked as deletion, but it does not actually delete topic. 
>
>Thanks, 
>Snehalata 


-- 

Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: Can we delete topic in kafka

[Jaikiran Pai]

That's actually not the right way to delete topics (or for that matter 
managing a Kafka instance). It can lead to odd/corrupt installation.


-Jaikiran

On Wednesday 11 May 2016 06:27 PM, Eduardo Costa Alfaia wrote:

Hi,
It’s better creating a script that delete the kafka folder where exist the 
kafka topic and after create it again if need.

BR


Eduardo Costa Alfaia
Ph.D. Student in Telecommunications Engineering
Università degli Studi di Brescia
Tel: +39 3209333018








On 5/11/16, 09:48, "Snehalata Nagaje"  
wrote:



Hi ,

Can we delete certain topic in kafka?

I have deleted using command

./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181

It says topic marked as deletion, but it does not actually delete topic.

Thanks,
Snehalata

Re: Can we delete topic in kafka

[Eduardo Costa Alfaia]

I have used:
./kafka-topics.sh --delete --topic unibs  --zookeeper 127.0.0.1:2181
It did work for me:

~/Downloads/kafka_2.11-0.9.0.1/bin$ ./kafka-topics.sh --list --zookeeper 
127.0.0.1:2181
unibs
~/Downloads/kafka_2.11-0.9.0.1/bin$ ./kafka-topics.sh --delete --topic unibs  
--zookeeper 127.0.0.1:2181
Topic unibs is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

~/Downloads/kafka_2.11-0.9.0.1/bin$ ./kafka-topics.sh --list --zookeeper 
127.0.0.1:2181
~/Downloads/kafka_2.11-0.9.0.1/bin$


Did you set the delete.topic.enable to true in the broker file configuration?

BR

Eduardo Costa Alfaia
Ph.D. Student in Telecommunications Engineering
Università degli Studi di Brescia
Tel: +39 3209333018







On 5/11/16, 14:58, "Jaikiran Pai"  wrote:

>That's actually not the right way to delete topics (or for that matter 
>managing a Kafka instance). It can lead to odd/corrupt installation.
>
>-Jaikiran
>
>On Wednesday 11 May 2016 06:27 PM, Eduardo Costa Alfaia wrote:
>> Hi,
>> It’s better creating a script that delete the kafka folder where exist the 
>> kafka topic and after create it again if need.
>>
>> BR
>>
>>
>> Eduardo Costa Alfaia
>> Ph.D. Student in Telecommunications Engineering
>> Università degli Studi di Brescia
>> Tel: +39 3209333018
>>
>>
>>
>>
>>
>>
>>
>>
>> On 5/11/16, 09:48, "Snehalata Nagaje"  
>> wrote:
>>
>>>
>>> Hi ,
>>>
>>> Can we delete certain topic in kafka?
>>>
>>> I have deleted using command
>>>
>>> ./kafka-topics.sh --delete --topic topic_billing --zookeeper localhost:2181
>>>
>>> It says topic marked as deletion, but it does not actually delete topic.
>>>
>>> Thanks,
>>> Snehalata
>>
>


-- 

Informativa sulla Privacy: http://www.unibs.it/node/8155

Re: How to delete Topic in kafka 0.8.1 ?

[François Langelier]

The delete topic isn't working ATM

I think it will be available in the next release
https://issues.apache.org/jira/browse/KAFKA-1397




François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Argentier Fraternité du Piranha <http://fraternitedupiranha.com/> 2012-2014
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior


On 3 June 2014 07:47, rafeeq s  wrote:

> I am using Kafka 0.8.1 and I tried to execute Topic delete command but
> still Topic list showing that specific Topic.
>
> Topic Delete commad which i used:
>
> *bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> my_topic_name*
>
> How can i verify that my_topic is deleted ?
>
> Is there specific steps to delete Topic in Kafka 0.8.1 ?
>
> Thanks in advance!
>

Re: How to delete Topic in kafka 0.8.1 ?

[rafeeq s]

Thanks François,

Then how we can delete topics in kafka 0.8.1 ?

Is there any procedure/steps to delete topic ?

Thanks for your kind response!


On Tue, Jun 3, 2014 at 3:57 PM, François Langelier 
wrote:

> The delete topic isn't working ATM
>
> I think it will be available in the next release
> https://issues.apache.org/jira/browse/KAFKA-1397
>
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Argentier Fraternité du Piranha <http://fraternitedupiranha.com/>
> 2012-2014
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On 3 June 2014 07:47, rafeeq s  wrote:
>
> > I am using Kafka 0.8.1 and I tried to execute Topic delete command but
> > still Topic list showing that specific Topic.
> >
> > Topic Delete commad which i used:
> >
> > *bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> > my_topic_name*
> >
> > How can i verify that my_topic is deleted ?
> >
> > Is there specific steps to delete Topic in Kafka 0.8.1 ?
> >
> > Thanks in advance!
> >
>

Re: How to delete Topic in kafka 0.8.1 ?

[François Langelier]

There is probably some steps, but unfortunately I'm not too sure about them
so I prefer to kept my mouth shut :)

But from what I read in the link I sent you, even if it will be only
available in the next release, there is a patch available and the code has
been commit to trunk
https://issues.apache.org/jira/browse/KAFKA-1397


François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Argentier Fraternité du Piranha <http://fraternitedupiranha.com/> 2012-2014
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior


On 3 June 2014 09:17, rafeeq s  wrote:

> Thanks François,
>
> Then how we can delete topics in kafka 0.8.1 ?
>
> Is there any procedure/steps to delete topic ?
>
> Thanks for your kind response!
>
>
> On Tue, Jun 3, 2014 at 3:57 PM, François Langelier 
> wrote:
>
> > The delete topic isn't working ATM
> >
> > I think it will be available in the next release
> > https://issues.apache.org/jira/browse/KAFKA-1397
> >
> >
> >
> >
> > François Langelier
> > Étudiant en génie Logiciel - École de Technologie Supérieure
> > <http://www.etsmtl.ca/>
> > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > VP-Communication - CS Games <http://csgames.org> 2014
> > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > Argentier Fraternité du Piranha <http://fraternitedupiranha.com/>
> > 2012-2014
> > Comité Organisateur Olympiades ÉTS 2012
> > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> >
> >
> > On 3 June 2014 07:47, rafeeq s  wrote:
> >
> > > I am using Kafka 0.8.1 and I tried to execute Topic delete command but
> > > still Topic list showing that specific Topic.
> > >
> > > Topic Delete commad which i used:
> > >
> > > *bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> > > my_topic_name*
> > >
> > > How can i verify that my_topic is deleted ?
> > >
> > > Is there specific steps to delete Topic in Kafka 0.8.1 ?
> > >
> > > Thanks in advance!
> > >
> >
>

Re: How to delete Topic in kafka 0.8.1 ?

[Guozhang Wang]

For now deleting topics would be a bit tedious and you need to stop the
brokers during the deletion. Here is what you can do:

1. Stop the brokers.
2. Delete the partition folders for this topic on all brokers.
3. Delete the topic-partition znode in ZK under the /brokers path.
4. Restart the brokers.

Guozhang


On Tue, Jun 3, 2014 at 6:32 AM, François Langelier 
wrote:

> There is probably some steps, but unfortunately I'm not too sure about them
> so I prefer to kept my mouth shut :)
>
> But from what I read in the link I sent you, even if it will be only
> available in the next release, there is a patch available and the code has
> been commit to trunk
> https://issues.apache.org/jira/browse/KAFKA-1397
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> <http://www.etsmtl.ca/>
> Capitaine Club Capra <http://capra.etsmtl.ca/>
> VP-Communication - CS Games <http://csgames.org> 2014
> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> Argentier Fraternité du Piranha <http://fraternitedupiranha.com/>
> 2012-2014
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On 3 June 2014 09:17, rafeeq s  wrote:
>
> > Thanks François,
> >
> > Then how we can delete topics in kafka 0.8.1 ?
> >
> > Is there any procedure/steps to delete topic ?
> >
> > Thanks for your kind response!
> >
> >
> > On Tue, Jun 3, 2014 at 3:57 PM, François Langelier <
> f.langel...@gmail.com>
> > wrote:
> >
> > > The delete topic isn't working ATM
> > >
> > > I think it will be available in the next release
> > > https://issues.apache.org/jira/browse/KAFKA-1397
> > >
> > >
> > >
> > >
> > > François Langelier
> > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > <http://www.etsmtl.ca/>
> > > Capitaine Club Capra <http://capra.etsmtl.ca/>
> > > VP-Communication - CS Games <http://csgames.org> 2014
> > > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
> > > Argentier Fraternité du Piranha <http://fraternitedupiranha.com/>
> > > 2012-2014
> > > Comité Organisateur Olympiades ÉTS 2012
> > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > >
> > >
> > > On 3 June 2014 07:47, rafeeq s  wrote:
> > >
> > > > I am using Kafka 0.8.1 and I tried to execute Topic delete command
> but
> > > > still Topic list showing that specific Topic.
> > > >
> > > > Topic Delete commad which i used:
> > > >
> > > > *bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic
> > > > my_topic_name*
> > > >
> > > > How can i verify that my_topic is deleted ?
> > > >
> > > > Is there specific steps to delete Topic in Kafka 0.8.1 ?
> > > >
> > > > Thanks in advance!
> > > >
> > >
> >
>



-- 
-- Guozhang

Re: pushing delete topic feature out of 0.8

[Jason Rosenberg]

Ok,

This is a feature I've been hoping for, so I added an upvote to Kafka-330.
 But I will defer to you in terms of not wanting to delay 0.8 unnecessarily.

Will we still have a backhanded way to remove a topic if need be?
 Ultimately, I'd like  to see the feature where a topic automatically is
created on receipt of a new message, and that topic will then automatically
become deleted after a configurable period of inactivity.  In otherwords,
auto-creation and auto-deletion.

Jason


On Thu, Apr 4, 2013 at 11:35 AM, Jun Rao  wrote:

> Hi,
>
> We started the work on deleting topics in 0.8 (kafka-330). We realized that
> it touches quite a few critical components such as controller, replica
> manager, and log, and it will take some time to stabilize this. In order
> not to delay the 0.8 release too much, I propose that we push this feature
> out of 0.8. Since we don't really support deleting topics in 0.7, this
> doesn't reduce the existing features.
>
> Any concerns from people?
>
> Thanks,
>
> Jun
>

Re: pushing delete topic feature out of 0.8

[David Arthur]

Since you need to learn about the leader for a topic in order to do 
anything, you kind of already have "auto-create" since getting metadata 
for a topic will create it if it doesn't exist.


As for auto-delete, log files will be deleted over time (per the usual 
policy). There will be residual stuff in ZooKeeper, but I don't think it 
will be detrimental to performance (does the broker keep any active 
state for a topic that's not being used? - not sure)


Not to say topic deletion isn't a useful feature, but I think punting 
out of 0.8 is fine.


-David

On 4/4/13 3:01 PM, Jason Rosenberg wrote:

Ok,

This is a feature I've been hoping for, so I added an upvote to Kafka-330.
  But I will defer to you in terms of not wanting to delay 0.8 unnecessarily.

Will we still have a backhanded way to remove a topic if need be?
  Ultimately, I'd like  to see the feature where a topic automatically is
created on receipt of a new message, and that topic will then automatically
become deleted after a configurable period of inactivity.  In otherwords,
auto-creation and auto-deletion.

Jason


On Thu, Apr 4, 2013 at 11:35 AM, Jun Rao  wrote:


Hi,

We started the work on deleting topics in 0.8 (kafka-330). We realized that
it touches quite a few critical components such as controller, replica
manager, and log, and it will take some time to stabilize this. In order
not to delay the 0.8 release too much, I propose that we push this feature
out of 0.8. Since we don't really support deleting topics in 0.7, this
doesn't reduce the existing features.

Any concerns from people?

Thanks,

Jun

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Do you have delete.topic.enable turned on in all brokers?

Thanks,

Jun

On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg  wrote:

> So far, I have been unable to get delete topic to work, with release
> candidate 2 for 0.8.2.
>
> It worked ok when I ran it in the debugger locally, on a single node
> instance. But when I run it in our staging environment, it is not
> successfully even marking the topic for delete, for some reason.
>
> I am setting delete.topic.enable to true (and see the confirmation of this
> in the startup logs, e.g.):
>
> INFO [main] utils.VerifiableProperties - Property delete.topic.enable
> is overridden to true
>
> I run this command:
>
> java -cp app.jar kafka.admin.TopicCommand --zookeeper
> myzkconnect:12345/mynamespace --delete --topic mytopic
>
> log4j:WARN No appenders could be found for logger
> (org.I0Itec.zkclient.ZkConnection).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN No appenders could be found for logger
> (org.I0Itec.zkclient.ZkEventThread).
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info.
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info.
> Topic mytopic is marked for deletion.
> Note: This will have no impact if delete.topic.enable is not set to true.
>
> I then do a —list which should at least show the topic marked for deletion:
>
> java -cp app.jar kafka.admin.TopicCommand --zookeeper
> myzkconnect:12345/mynamespace --list --topic mytopic
>
> log4j:WARN No appenders could be found for logger
> (org.I0Itec.zkclient.ZkConnection).
> log4j:WARN No appenders could be found for logger
> (org.I0Itec.zkclient.ZkEventThread).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> for more info.
> mytopic
>
> Note, it doesn’t list it as ‘marked for deletion’. Furthermore, waiting
> multiple hours still doesn’t result in the topic being deleted.
>
> The topic has 1 partition, 2 replicas, and no data stored.
>
> In zookeeper, the /admin/deleted_topics/ path is empty.
>
> The zookeeper code looks pretty straightforward, but for some reason is not
> writing the deleted_topics path.  We are running zookeeper 3.4.6.
>
> Thoughts?
>
> Jason
> ​
>

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

yes

On Mon, Jan 26, 2015 at 12:18 AM, Jun Rao  wrote:

> Do you have delete.topic.enable turned on in all brokers?
>
> Thanks,
>
> Jun
>
> On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg  wrote:
>
> > So far, I have been unable to get delete topic to work, with release
> > candidate 2 for 0.8.2.
> >
> > It worked ok when I ran it in the debugger locally, on a single node
> > instance. But when I run it in our staging environment, it is not
> > successfully even marking the topic for delete, for some reason.
> >
> > I am setting delete.topic.enable to true (and see the confirmation of
> this
> > in the startup logs, e.g.):
> >
> > INFO [main] utils.VerifiableProperties - Property delete.topic.enable
> > is overridden to true
> >
> > I run this command:
> >
> > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > myzkconnect:12345/mynamespace --delete --topic mytopic
> >
> > log4j:WARN No appenders could be found for logger
> > (org.I0Itec.zkclient.ZkConnection).
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN No appenders could be found for logger
> > (org.I0Itec.zkclient.ZkEventThread).
> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > for more info.
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > for more info.
> > Topic mytopic is marked for deletion.
> > Note: This will have no impact if delete.topic.enable is not set to true.
> >
> > I then do a —list which should at least show the topic marked for
> deletion:
> >
> > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > myzkconnect:12345/mynamespace --list --topic mytopic
> >
> > log4j:WARN No appenders could be found for logger
> > (org.I0Itec.zkclient.ZkConnection).
> > log4j:WARN No appenders could be found for logger
> > (org.I0Itec.zkclient.ZkEventThread).
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN Please initialize the log4j system properly.
> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > for more info.
> > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > for more info.
> > mytopic
> >
> > Note, it doesn’t list it as ‘marked for deletion’. Furthermore, waiting
> > multiple hours still doesn’t result in the topic being deleted.
> >
> > The topic has 1 partition, 2 replicas, and no data stored.
> >
> > In zookeeper, the /admin/deleted_topics/ path is empty.
> >
> > The zookeeper code looks pretty straightforward, but for some reason is
> not
> > writing the deleted_topics path.  We are running zookeeper 3.4.6.
> >
> > Thoughts?
> >
> > Jason
> > ​
> >
>

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Could you do a get on /admin/deleted_topics and see if its cversion changes
after issuing the delete topic command? This will tell us whether the
path /admin/deleted_topics//mytopic
is actually created or not. Anything in the controller log that indicates a
delete topic event is fired?

Thanks,

Jun

On Sun, Jan 25, 2015 at 9:55 PM, Jason Rosenberg  wrote:

> yes
>
> On Mon, Jan 26, 2015 at 12:18 AM, Jun Rao  wrote:
>
> > Do you have delete.topic.enable turned on in all brokers?
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg 
> wrote:
> >
> > > So far, I have been unable to get delete topic to work, with release
> > > candidate 2 for 0.8.2.
> > >
> > > It worked ok when I ran it in the debugger locally, on a single node
> > > instance. But when I run it in our staging environment, it is not
> > > successfully even marking the topic for delete, for some reason.
> > >
> > > I am setting delete.topic.enable to true (and see the confirmation of
> > this
> > > in the startup logs, e.g.):
> > >
> > > INFO [main] utils.VerifiableProperties - Property delete.topic.enable
> > > is overridden to true
> > >
> > > I run this command:
> > >
> > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > myzkconnect:12345/mynamespace --delete --topic mytopic
> > >
> > > log4j:WARN No appenders could be found for logger
> > > (org.I0Itec.zkclient.ZkConnection).
> > > log4j:WARN Please initialize the log4j system properly.
> > > log4j:WARN No appenders could be found for logger
> > > (org.I0Itec.zkclient.ZkEventThread).
> > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > for more info.
> > > log4j:WARN Please initialize the log4j system properly.
> > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > for more info.
> > > Topic mytopic is marked for deletion.
> > > Note: This will have no impact if delete.topic.enable is not set to
> true.
> > >
> > > I then do a —list which should at least show the topic marked for
> > deletion:
> > >
> > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > myzkconnect:12345/mynamespace --list --topic mytopic
> > >
> > > log4j:WARN No appenders could be found for logger
> > > (org.I0Itec.zkclient.ZkConnection).
> > > log4j:WARN No appenders could be found for logger
> > > (org.I0Itec.zkclient.ZkEventThread).
> > > log4j:WARN Please initialize the log4j system properly.
> > > log4j:WARN Please initialize the log4j system properly.
> > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > for more info.
> > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > for more info.
> > > mytopic
> > >
> > > Note, it doesn’t list it as ‘marked for deletion’. Furthermore, waiting
> > > multiple hours still doesn’t result in the topic being deleted.
> > >
> > > The topic has 1 partition, 2 replicas, and no data stored.
> > >
> > > In zookeeper, the /admin/deleted_topics/ path is empty.
> > >
> > > The zookeeper code looks pretty straightforward, but for some reason is
> > not
> > > writing the deleted_topics path.  We are running zookeeper 3.4.6.
> > >
> > > Thoughts?
> > >
> > > Jason
> > > ​
> > >
> >
>

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

:14,533  INFO [kafka-request-handler-0] log.LogManager
- Created log for partition [mytopic,0] in /mypath with properties
{segment.index.bytes -> 10485760, file.delete.delay.ms -> 6,
segment.bytes -> 1073741824, flush.ms -> 9223372036854775807,
delete.retention.ms -> 8640, index.interval.bytes -> 4096,
retention.bytes -> 500, min.insync.replicas -> 1,
cleanup.policy -> delete, unclean.leader.election.enable -> true,
segment.ms -> 60480, max.message.bytes -> 112, flush.messages
-> 9223372036854775807, min.cleanable.dirty.ratio -> 0.5, retention.ms
-> 8640, segment.jitter.ms -> 0}.

On Mon, Jan 26, 2015 at 1:26 AM, Jun Rao  wrote:

Could you do a get on /admin/deleted_topics and see if its cversion changes
> after issuing the delete topic command? This will tell us whether the
> path /admin/deleted_topics//mytopic
> is actually created or not. Anything in the controller log that indicates a
> delete topic event is fired?
>
> Thanks,
>
> Jun
>
> On Sun, Jan 25, 2015 at 9:55 PM, Jason Rosenberg  wrote:
>
> > yes
> >
> > On Mon, Jan 26, 2015 at 12:18 AM, Jun Rao  wrote:
> >
> > > Do you have delete.topic.enable turned on in all brokers?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sun, Jan 25, 2015 at 7:56 PM, Jason Rosenberg 
> > wrote:
> > >
> > > > So far, I have been unable to get delete topic to work, with release
> > > > candidate 2 for 0.8.2.
> > > >
> > > > It worked ok when I ran it in the debugger locally, on a single node
> > > > instance. But when I run it in our staging environment, it is not
> > > > successfully even marking the topic for delete, for some reason.
> > > >
> > > > I am setting delete.topic.enable to true (and see the confirmation of
> > > this
> > > > in the startup logs, e.g.):
> > > >
> > > > INFO [main] utils.VerifiableProperties - Property delete.topic.enable
> > > > is overridden to true
> > > >
> > > > I run this command:
> > > >
> > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > > myzkconnect:12345/mynamespace --delete --topic mytopic
> > > >
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkConnection).
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkEventThread).
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > Topic mytopic is marked for deletion.
> > > > Note: This will have no impact if delete.topic.enable is not set to
> > true.
> > > >
> > > > I then do a —list which should at least show the topic marked for
> > > deletion:
> > > >
> > > > java -cp app.jar kafka.admin.TopicCommand --zookeeper
> > > > myzkconnect:12345/mynamespace --list --topic mytopic
> > > >
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkConnection).
> > > > log4j:WARN No appenders could be found for logger
> > > > (org.I0Itec.zkclient.ZkEventThread).
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN Please initialize the log4j system properly.
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> > > > for more info.
> > > > mytopic
> > > >
> > > > Note, it doesn’t list it as ‘marked for deletion’. Furthermore,
> waiting
> > > > multiple hours still doesn’t result in the topic being deleted.
> > > >
> > > > The topic has 1 partition, 2 replicas, and no data stored.
> > > >
> > > > In zookeeper, the /admin/deleted_topics/ path is empty.
> > > >
> > > > The zookeeper code looks pretty straightforward, but for some reason
> is
> > > not
> > > > writing the deleted_topics path.  We are running zookeeper 3.4.6.
> > > >
> > > > Thoughts?
> > > >
> > > > Jason
> > > > ​
> > > >
> > >
> >
>
​

Re: unable to delete topic with 0.8.2 rc2

[Harsha]

There could be another case where if you have auto.create.topics.enable
to set to true ( its true by default) . Any TopicMetadataRequest can
recreate topics. So if you issued a delete topic command and you have
producers running or consumers? too which is issuing a
TopicMetadataRequest than the topic will be recreated.
-Harsha

On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> cversion did change (incremented by 2) when I issue the delete command.
> 
> From the logs on the conroller broker (also the leader for the topic), it
> looks like the delete proceeds, and then the topic gets recreated
> immediately (highlighted in yellow). It appears maybe it’s due to a
> consumer client app trying to consume the topic. Also, the consumer is
> not
> yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> problem?
> 
> 
> 2015-01-26 07:02:14,281  INFO
> [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> controller.PartitionStateMachine$DeleteTopicsListener -
> [DeleteTopicsListener on 6]: Starting topic deletion for topics
> mytopic
> 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Handling deletion for topics mytopic
> 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Deletion of topic mytopic (re)started
> 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> deletion callback for mytopic
> 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager - [Topic Deletion Manager 6],
> Partition deletion callback for [mytopic,0]
> 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to OfflineReplica for replicas
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> controller.KafkaController - [Controller 6]: New leader and ISR for
> partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> controller.KafkaController - [Controller 6]: New leader and ISR for
> partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to ReplicaDeletionStarted for replicas
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> Removed fetcher for partitions [mytopic,0]
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> Removed fetcher for partitions [mytopic,0]
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> log.OffsetIndex - Deleting index
> /mypath/mytopic-0/.index
> 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7] log.LogManager
> - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> [Topic=mytopic,Partition=0,Replica=6]
> 2015-01-26 07:02:14,314  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Handling deletion for topics mytopic
> 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Deletion for replicas 7 for partition
> [mytopic,0] of topic mytopic in progress
> 2015-01-26 07:02:14,316  INFO [Controller-6-to-broker-7-send-thread]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> [Topic=mytopic,Partition=0,Replica=7]
> 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> controller.TopicDeletionManager$DeleteTopicsThread -
> [delete-topics-thread-6], Handling deletion for topics mytopic
> 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
> controller.ReplicaStateMachine - [Replica state machine on controller
> 6]: Invoking state change to NonExistentReplica for replicas
> [Topic=mytopic,Partition=0,Replica=6],[Topic=mytopic,Partition=0,Replica=7]
> 2015-01-26 07:02:14,318  INFO [delete-topics-thread-6]
&

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Yes, that's the issue. Currently, topics can be auto-created on
TopicMetadataRequest, which can be issued from both the producer and the
consumer. To prevent that, you would need to stop the producer and the
consumer before deleting a topic. We plan to address this issue once we
have a separate request for creating topics.

Thanks,

Jun

On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:

> There could be another case where if you have auto.create.topics.enable
> to set to true ( its true by default) . Any TopicMetadataRequest can
> recreate topics. So if you issued a delete topic command and you have
> producers running or consumers? too which is issuing a
> TopicMetadataRequest than the topic will be recreated.
> -Harsha
>
> On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > cversion did change (incremented by 2) when I issue the delete command.
> >
> > From the logs on the conroller broker (also the leader for the topic), it
> > looks like the delete proceeds, and then the topic gets recreated
> > immediately (highlighted in yellow). It appears maybe it’s due to a
> > consumer client app trying to consume the topic. Also, the consumer is
> > not
> > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > problem?
> >
> >
> > 2015-01-26 07:02:14,281  INFO
> > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > controller.PartitionStateMachine$DeleteTopicsListener -
> > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > mytopic
> > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Handling deletion for topics mytopic
> > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > deletion callback for mytopic
> > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > Partition deletion callback for [mytopic,0]
> > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > controller.ReplicaStateMachine - [Replica state machine on controller
> > 6]: Invoking state change to OfflineReplica for replicas
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > controller.KafkaController - [Controller 6]: New leader and ISR for
> > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > controller.KafkaController - [Controller 6]: New leader and ISR for
> > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > controller.ReplicaStateMachine - [Replica state machine on controller
> > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > Removed fetcher for partitions [mytopic,0]
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > Removed fetcher for partitions [mytopic,0]
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > log.OffsetIndex - Deleting index
> > /mypath/mytopic-0/.index
> > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7] log.LogManager
> > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thread]
> > controller.ReplicaStateMachine - [Replica state machine on controller
> > 6]: Invoking state change to ReplicaDeletionSuccessful for replicas
> > [Topic=mytopic,Partition=0,Replica=6]
> > 2015-01-26 07:02:14,314  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Handling deletion for topics mytopic
> > 2015-01-26 07:02:14,316  INFO [delete-topics-thread-6]
> > controller.TopicDeletionManager$DeleteTopicsThread -
> > [delete-topics-thread-6], Deletion for replicas 7 for partition
> > [mytopic,0] of topic mytopic in progress
> > 2015-01-26 0

Re: unable to delete topic with 0.8.2 rc2

[Harsha]

Jun,
  I made an attempt at fixing that issue as part of this JIRA
  https://issues.apache.org/jira/browse/KAFKA-1507 . 
As Jay pointed out there should be admin api if there is more info on
this api I am interested in adding/fixing this issue.
Thanks,
Harsha

On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> Yes, that's the issue. Currently, topics can be auto-created on
> TopicMetadataRequest, which can be issued from both the producer and the
> consumer. To prevent that, you would need to stop the producer and the
> consumer before deleting a topic. We plan to address this issue once we
> have a separate request for creating topics.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> 
> > There could be another case where if you have auto.create.topics.enable
> > to set to true ( its true by default) . Any TopicMetadataRequest can
> > recreate topics. So if you issued a delete topic command and you have
> > producers running or consumers? too which is issuing a
> > TopicMetadataRequest than the topic will be recreated.
> > -Harsha
> >
> > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > cversion did change (incremented by 2) when I issue the delete command.
> > >
> > > From the logs on the conroller broker (also the leader for the topic), it
> > > looks like the delete proceeds, and then the topic gets recreated
> > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > consumer client app trying to consume the topic. Also, the consumer is
> > > not
> > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > problem?
> > >
> > >
> > > 2015-01-26 07:02:14,281  INFO
> > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > mytopic
> > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > deletion callback for mytopic
> > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > Partition deletion callback for [mytopic,0]
> > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > 6]: Invoking state change to OfflineReplica for replicas
> > >
> > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > >
> > [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > Removed fetcher for partitions [mytopic,0]
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > Removed fetcher for partitions [mytopic,0]
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > log.OffsetIndex - Deleting index
> > > /mypath/mytopic-0/.index
> > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7] log.LogManager
> > > - Deleted log for partition [mytopic,0] in /mypath/mytopic-0.
> > > 2015-01-26 07:02:14,314  INFO [Controller-6-to-broker-6-send-thr

Re: unable to delete topic with 0.8.2 rc2

[Ari Flink]

unsubscribe

On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:

> Jun,
>   I made an attempt at fixing that issue as part of this JIRA
>   https://issues.apache.org/jira/browse/KAFKA-1507 .
> As Jay pointed out there should be admin api if there is more info on
> this api I am interested in adding/fixing this issue.
> Thanks,
> Harsha
>
> On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > Yes, that's the issue. Currently, topics can be auto-created on
> > TopicMetadataRequest, which can be issued from both the producer and the
> > consumer. To prevent that, you would need to stop the producer and the
> > consumer before deleting a topic. We plan to address this issue once we
> > have a separate request for creating topics.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> >
> > > There could be another case where if you have auto.create.topics.enable
> > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > recreate topics. So if you issued a delete topic command and you have
> > > producers running or consumers? too which is issuing a
> > > TopicMetadataRequest than the topic will be recreated.
> > > -Harsha
> > >
> > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > cversion did change (incremented by 2) when I issue the delete
> command.
> > > >
> > > > From the logs on the conroller broker (also the leader for the
> topic), it
> > > > looks like the delete proceeds, and then the topic gets recreated
> > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > consumer client app trying to consume the topic. Also, the consumer
> is
> > > > not
> > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > > problem?
> > > >
> > > >
> > > > 2015-01-26 07:02:14,281  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > mytopic
> > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > deletion callback for mytopic
> > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > Partition deletion callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to OfflineReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6]
> > > > Removed fetcher for partitions [mytopic,0]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-7]
> > > > server.ReplicaFetcherManager - [ReplicaFetcherManager on broker 6

Re: unable to delete topic with 0.8.2 rc2

[Guozhang Wang]

It then seems to me that delete-topic will not actually work "smoothly"
until create topic request is added since it is too much to require people
to turn off their clients while deleting topics. In this case shall we make
it clear in the release docs of in 0.8.2, or even still mark it as
not-supported?

Guozhang

On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:

> Jun,
>   I made an attempt at fixing that issue as part of this JIRA
>   https://issues.apache.org/jira/browse/KAFKA-1507 .
> As Jay pointed out there should be admin api if there is more info on
> this api I am interested in adding/fixing this issue.
> Thanks,
> Harsha
>
> On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > Yes, that's the issue. Currently, topics can be auto-created on
> > TopicMetadataRequest, which can be issued from both the producer and the
> > consumer. To prevent that, you would need to stop the producer and the
> > consumer before deleting a topic. We plan to address this issue once we
> > have a separate request for creating topics.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> >
> > > There could be another case where if you have auto.create.topics.enable
> > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > recreate topics. So if you issued a delete topic command and you have
> > > producers running or consumers? too which is issuing a
> > > TopicMetadataRequest than the topic will be recreated.
> > > -Harsha
> > >
> > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > cversion did change (incremented by 2) when I issue the delete
> command.
> > > >
> > > > From the logs on the conroller broker (also the leader for the
> topic), it
> > > > looks like the delete proceeds, and then the topic gets recreated
> > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > consumer client app trying to consume the topic. Also, the consumer
> is
> > > > not
> > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > > problem?
> > > >
> > > >
> > > > 2015-01-26 07:02:14,281  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > mytopic
> > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > deletion callback for mytopic
> > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > Partition deletion callback for [mytopic,0]
> > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to OfflineReplica for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > partition [mytopic,0] is {"leader":-1,"leader_epoch":2,"isr":[]}
> > > > 2015-01-26 07:02:14,313  INFO [delete-topics-thread-6]
> > > > controller.ReplicaStateMachine - [Replica state machine on controller
> > > > 6]: Invoking state change to ReplicaDeletionStarted for replicas
> > > >
> > >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > 2015-01-26 07:02:14,313  INFO [kafka-request-handler-5]
> 

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

So, this is rather disappointing, especially since topic deletion is really
the primary feature in 0.8.2 I'm interested in.  The topic I was trying to
delete above had no data for many months.  The consumer which is triggering
recreation of that topic has been restarted several times since that topic
stopped receiving data.

In our case, we have a large number of topics, that no longer receive
messages (many have not received messages in over 12 months).  However, we
have consumers that use a regex which matches multiple topics, some of
which are these empty zombie topics.  Thus, we have a chicken-and-egg
problem:

1.  Consumer uses a regex to discover matching topics.
2.  Consumer starts consuming 'topic.foo.which.is.empty'.  This results in
regular meta-data requests for data for that topic.
3.  'topic.foo.which.is.empty' is deleted.
4.  Consumer encounters an error trying to fetch this topic, so issues a
meta-data request to find the leader for this topic.
5.  Broker recreates the topic in response to this.

We can stop consumers for maintenance and do a batch delete, but it is
problematic, because there are realtime dependencies on those consumers
being up and running.

I expect this will necessarily result in my voting no for 0.8.2 RC2 (if I
have a vote :)).

Just read up on KAFKA-1507, and left my 2 cents:

"I think relegating topic creation to an admin client would be very
limitiing. It's extremely useful to have a self-service system where new
applications can just create a new topic on demand (with reasonable
defaults), without the need for an admin to come in and prepare topics
ahead of a code release (leave that to dba's managing transactional
databases!).

I do like the idea of an automatic create topic request from a producer, in
response to a topic not found exception, rather than auto-creating topics
from meta-data requests (which happens asynchronously and causes the
initial meta data request to fail usually!). Consumers should never create
a topic, I should think."

On Mon, Jan 26, 2015 at 11:14 AM, Harsha  wrote:

> Jun,
>   I made an attempt at fixing that issue as part of this JIRA
>   https://issues.apache.org/jira/browse/KAFKA-1507 .
> As Jay pointed out there should be admin api if there is more info on
> this api I am interested in adding/fixing this issue.
> Thanks,
> Harsha
>
> On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > Yes, that's the issue. Currently, topics can be auto-created on
> > TopicMetadataRequest, which can be issued from both the producer and the
> > consumer. To prevent that, you would need to stop the producer and the
> > consumer before deleting a topic. We plan to address this issue once we
> > have a separate request for creating topics.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> >
> > > There could be another case where if you have auto.create.topics.enable
> > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > recreate topics. So if you issued a delete topic command and you have
> > > producers running or consumers? too which is issuing a
> > > TopicMetadataRequest than the topic will be recreated.
> > > -Harsha
> > >
> > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > cversion did change (incremented by 2) when I issue the delete
> command.
> > > >
> > > > From the logs on the conroller broker (also the leader for the
> topic), it
> > > > looks like the delete proceeds, and then the topic gets recreated
> > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > consumer client app trying to consume the topic. Also, the consumer
> is
> > > > not
> > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of the
> > > > problem?
> > > >
> > > >
> > > > 2015-01-26 07:02:14,281  INFO
> > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > mytopic
> > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > 2015-01-26 07:02:14,286  INFO [delete-to

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

I think this shortcoming should probably delay release of 0.8.2 until
resolved, no?

On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang  wrote:

> It then seems to me that delete-topic will not actually work "smoothly"
> until create topic request is added since it is too much to require people
> to turn off their clients while deleting topics. In this case shall we make
> it clear in the release docs of in 0.8.2, or even still mark it as
> not-supported?
>
> Guozhang
>
> On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
>
> > Jun,
> >   I made an attempt at fixing that issue as part of this JIRA
> >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > As Jay pointed out there should be admin api if there is more info on
> > this api I am interested in adding/fixing this issue.
> > Thanks,
> > Harsha
> >
> > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > Yes, that's the issue. Currently, topics can be auto-created on
> > > TopicMetadataRequest, which can be issued from both the producer and
> the
> > > consumer. To prevent that, you would need to stop the producer and the
> > > consumer before deleting a topic. We plan to address this issue once we
> > > have a separate request for creating topics.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > >
> > > > There could be another case where if you have
> auto.create.topics.enable
> > > > to set to true ( its true by default) . Any TopicMetadataRequest can
> > > > recreate topics. So if you issued a delete topic command and you have
> > > > producers running or consumers? too which is issuing a
> > > > TopicMetadataRequest than the topic will be recreated.
> > > > -Harsha
> > > >
> > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > cversion did change (incremented by 2) when I issue the delete
> > command.
> > > > >
> > > > > From the logs on the conroller broker (also the leader for the
> > topic), it
> > > > > looks like the delete proceeds, and then the topic gets recreated
> > > > > immediately (highlighted in yellow). It appears maybe it’s due to a
> > > > > consumer client app trying to consume the topic. Also, the consumer
> > is
> > > > > not
> > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of
> the
> > > > > problem?
> > > > >
> > > > >
> > > > > 2015-01-26 07:02:14,281  INFO
> > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > mytopic
> > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6], Topic
> > > > > deletion callback for mytopic
> > > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> > > > > Partition deletion callback for [mytopic,0]
> > > > > 2015-01-26 07:02:14,295  INFO [delete-topics-thread-6]
> > > > > controller.ReplicaStateMachine - [Replica state machine on
> controller
> > > > > 6]: Invoking state change to OfflineReplica for replicas
> > > > >
> > > >
> >
> [Topic=mytopic,Partition=0,Replica=7],[Topic=mytopic,Partition=0,Replica=6]
> > > > > 2015-01-26 07:02:14,303  INFO [delete-topics-thread-6]
> > > > > controller.KafkaController - [Controller 6]: New leader and ISR for
> > > > > partition [mytopic,0] is {"leader":6,"leader_epoch":1,"isr":[6]}
> > > > > 2015-01-26 07:02:14,312  INFO [delete-topics-thread-6]
> > > > > controller.KafkaController - [Controller 6]: New leade

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Hi, Jason,

I am not sure that we should delay the 0.8.2 release. The reasons are (1)
There are other features such as the new java producer and Kafka-based
offset management that are potentially useful to people. (2) It may take
some time to completely fix the issue with deleting topic since it involves
wire protocol changes.

What we can do is probably just to document the limitation of delete topic
in 0.8.2.0 in our site.

Thanks,

Jun

On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg  wrote:

> I think this shortcoming should probably delay release of 0.8.2 until
> resolved, no?
>
> On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang  wrote:
>
> > It then seems to me that delete-topic will not actually work "smoothly"
> > until create topic request is added since it is too much to require
> people
> > to turn off their clients while deleting topics. In this case shall we
> make
> > it clear in the release docs of in 0.8.2, or even still mark it as
> > not-supported?
> >
> > Guozhang
> >
> > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> >
> > > Jun,
> > >   I made an attempt at fixing that issue as part of this JIRA
> > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > As Jay pointed out there should be admin api if there is more info on
> > > this api I am interested in adding/fixing this issue.
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > TopicMetadataRequest, which can be issued from both the producer and
> > the
> > > > consumer. To prevent that, you would need to stop the producer and
> the
> > > > consumer before deleting a topic. We plan to address this issue once
> we
> > > > have a separate request for creating topics.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > >
> > > > > There could be another case where if you have
> > auto.create.topics.enable
> > > > > to set to true ( its true by default) . Any TopicMetadataRequest
> can
> > > > > recreate topics. So if you issued a delete topic command and you
> have
> > > > > producers running or consumers? too which is issuing a
> > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > -Harsha
> > > > >
> > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > cversion did change (incremented by 2) when I issue the delete
> > > command.
> > > > > >
> > > > > > From the logs on the conroller broker (also the leader for the
> > > topic), it
> > > > > > looks like the delete proceeds, and then the topic gets recreated
> > > > > > immediately (highlighted in yellow). It appears maybe it’s due
> to a
> > > > > > consumer client app trying to consume the topic. Also, the
> consumer
> > > is
> > > > > > not
> > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of
> > the
> > > > > > problem?
> > > > > >
> > > > > >
> > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > > mytopic
> > > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > [delete-topics-thread-6], Deletion of topic mytopic (re)started
> > > > > > 2015-01-26 07:02:14,286  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],
> Topic
> > > > > > deletion callback for mytopic
> > > > > > 2015-01-26 07:02:14,289  INFO [delete-topics-thread-6]
> > > > > > controller.TopicDeletionManager - [Topic Deletion Manager 6],

Re: unable to delete topic with 0.8.2 rc2

[Joel Koshy]

Hey Jason,

Is it an option for you to do the following:

- Bounce in a config change to the brokers to turn off auto-create
- (Batch)-delete the topic(s)
- Wait long enough for consumers to rebalance (after which they will
  no longer consume the topic(s))
- Bounce in a config change to the brokers to turn on auto-create

Joel

On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> Hi, Jason,
> 
> I am not sure that we should delay the 0.8.2 release. The reasons are (1)
> There are other features such as the new java producer and Kafka-based
> offset management that are potentially useful to people. (2) It may take
> some time to completely fix the issue with deleting topic since it involves
> wire protocol changes.
> 
> What we can do is probably just to document the limitation of delete topic
> in 0.8.2.0 in our site.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg  wrote:
> 
> > I think this shortcoming should probably delay release of 0.8.2 until
> > resolved, no?
> >
> > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang  wrote:
> >
> > > It then seems to me that delete-topic will not actually work "smoothly"
> > > until create topic request is added since it is too much to require
> > people
> > > to turn off their clients while deleting topics. In this case shall we
> > make
> > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > not-supported?
> > >
> > > Guozhang
> > >
> > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > >
> > > > Jun,
> > > >   I made an attempt at fixing that issue as part of this JIRA
> > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > As Jay pointed out there should be admin api if there is more info on
> > > > this api I am interested in adding/fixing this issue.
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > TopicMetadataRequest, which can be issued from both the producer and
> > > the
> > > > > consumer. To prevent that, you would need to stop the producer and
> > the
> > > > > consumer before deleting a topic. We plan to address this issue once
> > we
> > > > > have a separate request for creating topics.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > > >
> > > > > > There could be another case where if you have
> > > auto.create.topics.enable
> > > > > > to set to true ( its true by default) . Any TopicMetadataRequest
> > can
> > > > > > recreate topics. So if you issued a delete topic command and you
> > have
> > > > > > producers running or consumers? too which is issuing a
> > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > -Harsha
> > > > > >
> > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > cversion did change (incremented by 2) when I issue the delete
> > > > command.
> > > > > > >
> > > > > > > From the logs on the conroller broker (also the leader for the
> > > > topic), it
> > > > > > > looks like the delete proceeds, and then the topic gets recreated
> > > > > > > immediately (highlighted in yellow). It appears maybe it’s due
> > to a
> > > > > > > consumer client app trying to consume the topic. Also, the
> > consumer
> > > > is
> > > > > > > not
> > > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part of
> > > the
> > > > > > > problem?
> > > > > > >
> > > > > > >
> > > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > > > mytopic
> > > > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > &g

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

Yeah Joel,

I just thought of that idea too (e.g. temporarily disable auto topic
creation).  I can probably make that work (it may result in unexpected lack
of topic creation, but I can message this as a maintenance downtime, etc.).

So, that was my next question, so a consumer rebalance will trigger a fresh
topic pull from the consumers?  How long is 'long enough' to ensure a
rebalance has occurred everywhere?

Jason

On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy  wrote:

> Hey Jason,
>
> Is it an option for you to do the following:
>
> - Bounce in a config change to the brokers to turn off auto-create
> - (Batch)-delete the topic(s)
> - Wait long enough for consumers to rebalance (after which they will
>   no longer consume the topic(s))
> - Bounce in a config change to the brokers to turn on auto-create
>
> Joel
>
> On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > Hi, Jason,
> >
> > I am not sure that we should delay the 0.8.2 release. The reasons are (1)
> > There are other features such as the new java producer and Kafka-based
> > offset management that are potentially useful to people. (2) It may take
> > some time to completely fix the issue with deleting topic since it
> involves
> > wire protocol changes.
> >
> > What we can do is probably just to document the limitation of delete
> topic
> > in 0.8.2.0 in our site.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> wrote:
> >
> > > I think this shortcoming should probably delay release of 0.8.2 until
> > > resolved, no?
> > >
> > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> wrote:
> > >
> > > > It then seems to me that delete-topic will not actually work
> "smoothly"
> > > > until create topic request is added since it is too much to require
> > > people
> > > > to turn off their clients while deleting topics. In this case shall
> we
> > > make
> > > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > > not-supported?
> > > >
> > > > Guozhang
> > > >
> > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > > >
> > > > > Jun,
> > > > >   I made an attempt at fixing that issue as part of this JIRA
> > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > As Jay pointed out there should be admin api if there is more info
> on
> > > > > this api I am interested in adding/fixing this issue.
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > > TopicMetadataRequest, which can be issued from both the producer
> and
> > > > the
> > > > > > consumer. To prevent that, you would need to stop the producer
> and
> > > the
> > > > > > consumer before deleting a topic. We plan to address this issue
> once
> > > we
> > > > > > have a separate request for creating topics.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > > > >
> > > > > > > There could be another case where if you have
> > > > auto.create.topics.enable
> > > > > > > to set to true ( its true by default) . Any
> TopicMetadataRequest
> > > can
> > > > > > > recreate topics. So if you issued a delete topic command and
> you
> > > have
> > > > > > > producers running or consumers? too which is issuing a
> > > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > > -Harsha
> > > > > > >
> > > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > > cversion did change (incremented by 2) when I issue the
> delete
> > > > > command.
> > > > > > > >
> > > > > > > > From the logs on the conroller broker (also the leader for
> the
> > > > > topic), it
> > > > > > > > looks like the delete proceeds, and then the topic gets
> recreated
> &

Re: unable to delete topic with 0.8.2 rc2

[Jason Rosenberg]

Yeah Jun,

I realize it's not worth blocking things, since it's not trivial to solve.

Thanks for all the work on this.

Jason

On Mon, Jan 26, 2015 at 3:01 PM, Jun Rao  wrote:

> Hi, Jason,
>
> I am not sure that we should delay the 0.8.2 release. The reasons are (1)
> There are other features such as the new java producer and Kafka-based
> offset management that are potentially useful to people. (2) It may take
> some time to completely fix the issue with deleting topic since it involves
> wire protocol changes.
>
> What we can do is probably just to document the limitation of delete topic
> in 0.8.2.0 in our site.
>
> Thanks,
>
> Jun
>
> On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> wrote:
>
> > I think this shortcoming should probably delay release of 0.8.2 until
> > resolved, no?
> >
> > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> wrote:
> >
> > > It then seems to me that delete-topic will not actually work "smoothly"
> > > until create topic request is added since it is too much to require
> > people
> > > to turn off their clients while deleting topics. In this case shall we
> > make
> > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > not-supported?
> > >
> > > Guozhang
> > >
> > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > >
> > > > Jun,
> > > >   I made an attempt at fixing that issue as part of this JIRA
> > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > As Jay pointed out there should be admin api if there is more info on
> > > > this api I am interested in adding/fixing this issue.
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > TopicMetadataRequest, which can be issued from both the producer
> and
> > > the
> > > > > consumer. To prevent that, you would need to stop the producer and
> > the
> > > > > consumer before deleting a topic. We plan to address this issue
> once
> > we
> > > > > have a separate request for creating topics.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha  wrote:
> > > > >
> > > > > > There could be another case where if you have
> > > auto.create.topics.enable
> > > > > > to set to true ( its true by default) . Any TopicMetadataRequest
> > can
> > > > > > recreate topics. So if you issued a delete topic command and you
> > have
> > > > > > producers running or consumers? too which is issuing a
> > > > > > TopicMetadataRequest than the topic will be recreated.
> > > > > > -Harsha
> > > > > >
> > > > > > On Sun, Jan 25, 2015, at 11:26 PM, Jason Rosenberg wrote:
> > > > > > > cversion did change (incremented by 2) when I issue the delete
> > > > command.
> > > > > > >
> > > > > > > From the logs on the conroller broker (also the leader for the
> > > > topic), it
> > > > > > > looks like the delete proceeds, and then the topic gets
> recreated
> > > > > > > immediately (highlighted in yellow). It appears maybe it’s due
> > to a
> > > > > > > consumer client app trying to consume the topic. Also, the
> > consumer
> > > > is
> > > > > > > not
> > > > > > > yet updated to 0.8.2 (it’s using 0.8.1.1), perhaps that’s part
> of
> > > the
> > > > > > > problem?
> > > > > > >
> > > > > > >
> > > > > > > 2015-01-26 07:02:14,281  INFO
> > > > > > > [ZkClient-EventThread-21-myzkserver:12345/mynamespace]
> > > > > > > controller.PartitionStateMachine$DeleteTopicsListener -
> > > > > > > [DeleteTopicsListener on 6]: Starting topic deletion for topics
> > > > > > > mytopic
> > > > > > > 2015-01-26 07:02:14,282  INFO [delete-topics-thread-6]
> > > > > > > controller.TopicDeletionManager$DeleteTopicsThread -
> > > > > > > [delete-topics-thread-6], Handling deletion for topics mytopic
> 

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Actually, I think the deletion of a topic will trigger a rebalance on all
wildcard subscribers. The time to complete the rebalance depends on the #
of topic/partitions.

Thanks,

Jun

On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg  wrote:

> Yeah Joel,
>
> I just thought of that idea too (e.g. temporarily disable auto topic
> creation).  I can probably make that work (it may result in unexpected lack
> of topic creation, but I can message this as a maintenance downtime, etc.).
>
> So, that was my next question, so a consumer rebalance will trigger a fresh
> topic pull from the consumers?  How long is 'long enough' to ensure a
> rebalance has occurred everywhere?
>
> Jason
>
> On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy  wrote:
>
> > Hey Jason,
> >
> > Is it an option for you to do the following:
> >
> > - Bounce in a config change to the brokers to turn off auto-create
> > - (Batch)-delete the topic(s)
> > - Wait long enough for consumers to rebalance (after which they will
> >   no longer consume the topic(s))
> > - Bounce in a config change to the brokers to turn on auto-create
> >
> > Joel
> >
> > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > Hi, Jason,
> > >
> > > I am not sure that we should delay the 0.8.2 release. The reasons are
> (1)
> > > There are other features such as the new java producer and Kafka-based
> > > offset management that are potentially useful to people. (2) It may
> take
> > > some time to completely fix the issue with deleting topic since it
> > involves
> > > wire protocol changes.
> > >
> > > What we can do is probably just to document the limitation of delete
> > topic
> > > in 0.8.2.0 in our site.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> > wrote:
> > >
> > > > I think this shortcoming should probably delay release of 0.8.2 until
> > > > resolved, no?
> > > >
> > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> > wrote:
> > > >
> > > > > It then seems to me that delete-topic will not actually work
> > "smoothly"
> > > > > until create topic request is added since it is too much to require
> > > > people
> > > > > to turn off their clients while deleting topics. In this case shall
> > we
> > > > make
> > > > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > > > not-supported?
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > > > >
> > > > > > Jun,
> > > > > >   I made an attempt at fixing that issue as part of this JIRA
> > > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > > As Jay pointed out there should be admin api if there is more
> info
> > on
> > > > > > this api I am interested in adding/fixing this issue.
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > > > TopicMetadataRequest, which can be issued from both the
> producer
> > and
> > > > > the
> > > > > > > consumer. To prevent that, you would need to stop the producer
> > and
> > > > the
> > > > > > > consumer before deleting a topic. We plan to address this issue
> > once
> > > > we
> > > > > > > have a separate request for creating topics.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 7:21 AM, Harsha 
> wrote:
> > > > > > >
> > > > > > > > There could be another case where if you have
> > > > > auto.create.topics.enable
> > > > > > > > to set to true ( its true by default) . Any
> > TopicMetadataRequest
> > > > can
> > > > > > > > recreate topics. So if you issued a delete topic command and
> > you
> > > > have
> > > > > &

Re: unable to delete topic with 0.8.2 rc2

[Joel Koshy]

I think this should work even on non-wildcard consumers. (i.e., I just
tried it locally and it appears to work fine).

Here is what I did:

- Leave a console consumer up and running, bounce the broker to turn
  off auto-create
- Delete the topic
- The consumer should rebalance and stop consuming that topic
- Bounce the broker to turn on auto-create

The only issue is that while your broker has auto-create off you will
see errors on some producers - which you can announce as a brief
outage to your users.

Joel

On Mon, Jan 26, 2015 at 01:02:57PM -0800, Jun Rao wrote:
> Actually, I think the deletion of a topic will trigger a rebalance on all
> wildcard subscribers. The time to complete the rebalance depends on the #
> of topic/partitions.
> 
> Thanks,
> 
> Jun
> 
> On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg  wrote:
> 
> > Yeah Joel,
> >
> > I just thought of that idea too (e.g. temporarily disable auto topic
> > creation).  I can probably make that work (it may result in unexpected lack
> > of topic creation, but I can message this as a maintenance downtime, etc.).
> >
> > So, that was my next question, so a consumer rebalance will trigger a fresh
> > topic pull from the consumers?  How long is 'long enough' to ensure a
> > rebalance has occurred everywhere?
> >
> > Jason
> >
> > On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy  wrote:
> >
> > > Hey Jason,
> > >
> > > Is it an option for you to do the following:
> > >
> > > - Bounce in a config change to the brokers to turn off auto-create
> > > - (Batch)-delete the topic(s)
> > > - Wait long enough for consumers to rebalance (after which they will
> > >   no longer consume the topic(s))
> > > - Bounce in a config change to the brokers to turn on auto-create
> > >
> > > Joel
> > >
> > > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > > Hi, Jason,
> > > >
> > > > I am not sure that we should delay the 0.8.2 release. The reasons are
> > (1)
> > > > There are other features such as the new java producer and Kafka-based
> > > > offset management that are potentially useful to people. (2) It may
> > take
> > > > some time to completely fix the issue with deleting topic since it
> > > involves
> > > > wire protocol changes.
> > > >
> > > > What we can do is probably just to document the limitation of delete
> > > topic
> > > > in 0.8.2.0 in our site.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg 
> > > wrote:
> > > >
> > > > > I think this shortcoming should probably delay release of 0.8.2 until
> > > > > resolved, no?
> > > > >
> > > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang 
> > > wrote:
> > > > >
> > > > > > It then seems to me that delete-topic will not actually work
> > > "smoothly"
> > > > > > until create topic request is added since it is too much to require
> > > > > people
> > > > > > to turn off their clients while deleting topics. In this case shall
> > > we
> > > > > make
> > > > > > it clear in the release docs of in 0.8.2, or even still mark it as
> > > > > > not-supported?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha  wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >   I made an attempt at fixing that issue as part of this JIRA
> > > > > > >   https://issues.apache.org/jira/browse/KAFKA-1507 .
> > > > > > > As Jay pointed out there should be admin api if there is more
> > info
> > > on
> > > > > > > this api I am interested in adding/fixing this issue.
> > > > > > > Thanks,
> > > > > > > Harsha
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015, at 07:28 AM, Jun Rao wrote:
> > > > > > > > Yes, that's the issue. Currently, topics can be auto-created on
> > > > > > > > TopicMetadataRequest, which can be issued from both the
> > producer
> > > and
> > > > > > the
> > > > > > >

Re: unable to delete topic with 0.8.2 rc2

[Jun Rao]

Joel,

That's probably because console consumer always uses wildcard for
consumption.

Thanks,

Jun

On Mon, Jan 26, 2015 at 1:44 PM, Joel Koshy  wrote:

> I think this should work even on non-wildcard consumers. (i.e., I just
> tried it locally and it appears to work fine).
>
> Here is what I did:
>
> - Leave a console consumer up and running, bounce the broker to turn
>   off auto-create
> - Delete the topic
> - The consumer should rebalance and stop consuming that topic
> - Bounce the broker to turn on auto-create
>
> The only issue is that while your broker has auto-create off you will
> see errors on some producers - which you can announce as a brief
> outage to your users.
>
> Joel
>
> On Mon, Jan 26, 2015 at 01:02:57PM -0800, Jun Rao wrote:
> > Actually, I think the deletion of a topic will trigger a rebalance on all
> > wildcard subscribers. The time to complete the rebalance depends on the #
> > of topic/partitions.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Jan 26, 2015 at 12:26 PM, Jason Rosenberg 
> wrote:
> >
> > > Yeah Joel,
> > >
> > > I just thought of that idea too (e.g. temporarily disable auto topic
> > > creation).  I can probably make that work (it may result in unexpected
> lack
> > > of topic creation, but I can message this as a maintenance downtime,
> etc.).
> > >
> > > So, that was my next question, so a consumer rebalance will trigger a
> fresh
> > > topic pull from the consumers?  How long is 'long enough' to ensure a
> > > rebalance has occurred everywhere?
> > >
> > > Jason
> > >
> > > On Mon, Jan 26, 2015 at 3:07 PM, Joel Koshy 
> wrote:
> > >
> > > > Hey Jason,
> > > >
> > > > Is it an option for you to do the following:
> > > >
> > > > - Bounce in a config change to the brokers to turn off auto-create
> > > > - (Batch)-delete the topic(s)
> > > > - Wait long enough for consumers to rebalance (after which they will
> > > >   no longer consume the topic(s))
> > > > - Bounce in a config change to the brokers to turn on auto-create
> > > >
> > > > Joel
> > > >
> > > > On Mon, Jan 26, 2015 at 12:01:59PM -0800, Jun Rao wrote:
> > > > > Hi, Jason,
> > > > >
> > > > > I am not sure that we should delay the 0.8.2 release. The reasons
> are
> > > (1)
> > > > > There are other features such as the new java producer and
> Kafka-based
> > > > > offset management that are potentially useful to people. (2) It may
> > > take
> > > > > some time to completely fix the issue with deleting topic since it
> > > > involves
> > > > > wire protocol changes.
> > > > >
> > > > > What we can do is probably just to document the limitation of
> delete
> > > > topic
> > > > > in 0.8.2.0 in our site.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Mon, Jan 26, 2015 at 11:05 AM, Jason Rosenberg <
> j...@squareup.com>
> > > > wrote:
> > > > >
> > > > > > I think this shortcoming should probably delay release of 0.8.2
> until
> > > > > > resolved, no?
> > > > > >
> > > > > > On Mon, Jan 26, 2015 at 2:01 PM, Guozhang Wang <
> wangg...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > It then seems to me that delete-topic will not actually work
> > > > "smoothly"
> > > > > > > until create topic request is added since it is too much to
> require
> > > > > > people
> > > > > > > to turn off their clients while deleting topics. In this case
> shall
> > > > we
> > > > > > make
> > > > > > > it clear in the release docs of in 0.8.2, or even still mark
> it as
> > > > > > > not-supported?
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > > On Mon, Jan 26, 2015 at 8:14 AM, Harsha 
> wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >   I made an attempt at fixing that issue as part of this
> JIRA
> > > > > > > >   https://iss