[jira] [Created] (KAFKA-16956) Broker-side ability to subscribe to record delete events

[Luke Chen (Jira)]

Luke Chen created KAFKA-16956:
-

 Summary: Broker-side ability to subscribe to record delete events
 Key: KAFKA-16956
 URL: https://issues.apache.org/jira/browse/KAFKA-16956
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


In some cases it would be useful for systems outside Kafka to have the ability 
to know when Kafka deletes records (tombstoning or retention).

In general the use-case is where there is a desire to link the lifecycle of a 
record in a third party system (database or filesystem etc) to the lifecycle of 
the record in Kafka.

A concrete use-case:  a system using Kafka to distribute video clips + 
metadata.  The binary content is too big to store in Kafka so the publishing 
application caches the content in cloud storage and publishes a record 
containing a S3 url to the video clip.  The desire is to have a mechanism to 
remove the clip from cloud storage at the same time the record is expunged from 
Kafka by retention or tombstoning.  Currently there is no practical way to 
achieve this.
h2. Desired solution

A pluggable broker-side mechanism that is informed as records are being 
compacted away or deleted.  The API would expose the topic from which the 
record is being deleted, the record key, record headers, timestamp and 
(possibly) record value.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Subscribe for discussion on Kafka® development

[Matthias J. Sax]

This is self service. Please follow the instructions from the web page: 
https://kafka.apache.org/contact



-Matthias


On 6/11/24 9:38 AM, Punsak Incham wrote:

I want to subscribe for discussion on Kafka® development.

Best regards,

A black and blue logo Description automatically generated



*Punsak Incham (Mon)*

Data Platform Engineer

Data Modernization
Tel : +66(0)91-790-1302 | Email : pun...@mfec.co.th 
<mailto:pun...@mfec.co.th>


signature_2789072556

DISCLAIMER: This e-mail and any attachments may contain confidential or 
legally privileged information for useof MFEC Public Company Limited only.


If you are not the intended recipient, you are not authorized to copy or 
disclose all or any part of it without theprior written consent of the 
company.


Refer tohttps://www.mfec.co.th/en/privacy-policy/ 
<https://www.mfec.co.th/en/privacy-policy/>

Re: Subscribe for discussion on Kafka(R) development

[Greg Harris]

Hi Punsak,

Please find the instructions for subscribing to this mailing list here:
https://kafka.apache.org/contact and here:
https://www.apache.org/foundation/mailinglists

Thanks,
Greg

On Wed, Jun 12, 2024 at 7:11 AM Punsak Incham 
wrote:

> I want to subscribe for discussion on Kafka® development.
>
>
>
> Best regards,
>
>
>
>[image: A black and blue logo Description automatically generated]
>
> *Punsak Incham (Mon)*
>
> Data Platform Engineer
>
> Data Modernization
> Tel : +66(0)91-790-1302 | Email : pun...@mfec.co.th
>
> [image: signature_2789072556]
>
> DISCLAIMER: This e-mail and any attachments may contain confidential or
> legally privileged information for use of MFEC Public Company Limited
> only.
>
> If you are not the intended recipient, you are not authorized to copy or
> disclose all or any part of it without the prior written consent of the
> company.
>
> Refer to https://www.mfec.co.th/en/privacy-policy/
>
>
>
>
>

Subscribe for discussion on Kafka® development

[Punsak Incham]

I want to subscribe for discussion on Kafka® development.


Best regards,

   [A black and blue logo  Description automatically generated]
Punsak Incham (Mon)
Data Platform Engineer
Data Modernization
Tel : +66(0)91-790-1302 | Email : pun...@mfec.co.th<mailto:pun...@mfec.co.th>
[signature_2789072556]
DISCLAIMER: This e-mail and any attachments may contain confidential or legally 
privileged information for use of MFEC Public Company Limited only.
If you are not the intended recipient, you are not authorized to copy or 
disclose all or any part of it without the prior written consent of the company.
Refer to https://www.mfec.co.th/en/privacy-policy/

Re: Hello world - please subscribe me!

[Matthias J. Sax]

Chris,

Subscription is self-service. Please follow the instruction from the web 
page: https://kafka.apache.org/contact



-Matthias

On 6/3/24 1:59 AM, Chris wrote:

Re: please let me subscribe

[Harry Fallows]

Hi Greg,

Thank you for your help! I've used the correct subscription API now
<https://kafka.apache.org/23/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#subscribe-java.util.Collection->
 (consumer.subscribe(Arrays.asList("dev-subscr...@kafka.apache.org"));).

Apologies for the spam.

Harry Fallows

Software Engineer Idiot

On Thu, 16 May 2024 at 22:28, Greg Harris  wrote:

> Hi Harry,
>
> Thanks for your interest in Apache Kafka. You can see how to subscribe
> to the mailing list(s) here: https://kafka.apache.org/contact
> Please note that the email for subscribing/unsubscribing is different
> from the one used for sending and receiving emails once you're on the
> list.
> Also, subscribing needs a follow-up email to confirm the subscription,
> see here: https://www.apache.org/foundation/mailinglists#subscribing
>
> Hope this helps!
> Greg
>
> On Thu, May 16, 2024 at 5:51 AM Harry Fallows
>  wrote:
> >
> > please
> >
> > Harry Fallows
> >
> > Software Engineer
> >
> > Email: hfall...@thoughtmachine.net
> >
> > Web: www.thoughtmachine.net
> >
> >
> > 7 Herbrand Street | London | WC1N 1EX
> >
> >
> >
> > Data Classification: Internal
> >
> > --
> > Thought Machine Group Limited, a company registered in England & Wales.
> > Registered number: 4277.
> > Registered Office: 5 New Street Square,
> > London EC4A 3TW
> > <
> https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW=gmail=g
> >.
> >
> >
> > The content of this email is confidential and intended for the recipient
> > specified in message only. It is strictly forbidden to share any part of
> > this message with any third party, without a written consent of the
> sender.
> > If you received this message by mistake, please reply to this message and
> > follow with its deletion, so that we can ensure such a mistake does not
> > occur in the future.
>

-- 
Thought Machine Group Limited, a company registered in England & Wales.
Registered number: 4277. 
Registered Office: 5 New Street Square, 
London EC4A 3TW 
<https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW=gmail=g>.


The content of this email is confidential and intended for the recipient 
specified in message only. It is strictly forbidden to share any part of 
this message with any third party, without a written consent of the sender. 
If you received this message by mistake, please reply to this message and 
follow with its deletion, so that we can ensure such a mistake does not 
occur in the future.

Re: please let me subscribe

[Greg Harris]

Hi Harry,

Thanks for your interest in Apache Kafka. You can see how to subscribe
to the mailing list(s) here: https://kafka.apache.org/contact
Please note that the email for subscribing/unsubscribing is different
from the one used for sending and receiving emails once you're on the
list.
Also, subscribing needs a follow-up email to confirm the subscription,
see here: https://www.apache.org/foundation/mailinglists#subscribing

Hope this helps!
Greg

On Thu, May 16, 2024 at 5:51 AM Harry Fallows
 wrote:
>
> please
>
> Harry Fallows
>
> Software Engineer
>
> Email: hfall...@thoughtmachine.net
>
> Web: www.thoughtmachine.net
>
>
> 7 Herbrand Street | London | WC1N 1EX
>
>
>
> Data Classification: Internal
>
> --
> Thought Machine Group Limited, a company registered in England & Wales.
> Registered number: 4277.
> Registered Office: 5 New Street Square,
> London EC4A 3TW
> <https://maps.google.com/?q=5+New+Street+Square,+London+EC4A+3TW=gmail=g>.
>
>
> The content of this email is confidential and intended for the recipient
> specified in message only. It is strictly forbidden to share any part of
> this message with any third party, without a written consent of the sender.
> If you received this message by mistake, please reply to this message and
> follow with its deletion, so that we can ensure such a mistake does not
> occur in the future.

[jira] [Created] (KAFKA-16786) New consumer subscribe should not require the deprecated partition.assignment.strategy

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16786:
--

 Summary: New consumer subscribe should not require the deprecated 
partition.assignment.strategy
 Key: KAFKA-16786
 URL: https://issues.apache.org/jira/browse/KAFKA-16786
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 3.7.0
Reporter: Lianet Magrans


The partition.assignment.strategy config is deprecated with the new consumer 
group protocol KIP-848. With the new protocol, server side assignors are 
supported for now, defined in the property
group.remote.assignor, and with default values selected by the broker, so it's 
not even a required property. 

The new AsyncKafkaConsumer supports the new protocol only, but it currently 
throws an IllegalStateException if a call to subscribe is made and the 
deprecated config partition.assignment.strategy is empty (see 
[throwIfNoAssignorsConfigured|https://github.com/apache/kafka/blob/056d232f4e28bf8e67e00f8ed2c103fdb0f3b78e/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java#L1715]).
 
We should remove the reference to ConsumerPartitionAssignor in the 
AsyncKafkaConsumer, along with it's validation for non-empty on subscribe (only 
use it has)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

please let me subscribe

[Harry Fallows]

please

Harry Fallows

Software Engineer

Email: hfall...@thoughtmachine.net

Web: www.thoughtmachine.net


7 Herbrand Street | London | WC1N 1EX



Data Classification: Internal

-- 
Thought Machine Group Limited, a company registered in England & Wales.
Registered number: 4277. 
Registered Office: 5 New Street Square, 
London EC4A 3TW 
.


The content of this email is confidential and intended for the recipient 
specified in message only. It is strictly forbidden to share any part of 
this message with any third party, without a written consent of the sender. 
If you received this message by mistake, please reply to this message and 
follow with its deletion, so that we can ensure such a mistake does not 
occur in the future.

Re: Subscribe to Developer mailing list

[Bruno Cadonna]

Hi LoÏC,

subscription to the mailing lists is self-service. See details under 
https://kafka.apache.org/contact


Best,
Bruno

On 2/29/24 9:48 AM, Loic Greffier wrote:

Hi @dev@kafka.apache.org <mailto:dev@kafka.apache.org>,

I am working as a Software Engineer at Michelin, and would like to 
subscribe to the Developer mailing list to be able to open a KIP and 
contribute to Apache Kafka.


LoÏC GREFFIER

*/GROUPE MICHELIN – /*/Development Technology Specialist « *DOTI/BS/SMI* »/

//

noun_94615_cc_blue8, Rue de la Grolière | R1-2 | 63100 Clermont-Ferrand 
Cedex 09 | France


loic.greff...@michelin.com <mailto:loic.greff...@michelin.com>

Subscribe to Developer mailing list

[Loic Greffier]

Hi @dev@kafka.apache.org<mailto:dev@kafka.apache.org>,

I am working as a Software Engineer at Michelin, and would like to subscribe to 
the Developer mailing list to be able to open a KIP and contribute to Apache 
Kafka.

LoÏC GREFFIER
GROUPE MICHELIN - Development Technology Specialist « DOTI/BS/SMI »
  [cid:image001.png@01DA6AF3.002B0EF0]
[noun_94615_cc_blue]8, Rue de la Grolière | R1-2 | 63100 Clermont-Ferrand Cedex 
09 | France
[cid:image003.png@01DA6AF3.002B0EF0]  
loic.greff...@michelin.com<mailto:loic.greff...@michelin.com>
[cid:image004.png@01DA6AF3.002B0EF0]

[jira] [Created] (KAFKA-16301) Review fenced member unsubscribe/subscribe callbacks interaction

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-16301:
--

 Summary: Review fenced member unsubscribe/subscribe callbacks 
interaction
 Key: KAFKA-16301
 URL: https://issues.apache.org/jira/browse/KAFKA-16301
 Project: Kafka
  Issue Type: Sub-task
  Components: clients, consumer
Reporter: Lianet Magrans


When a member gets fenced, it triggers the onPartitionsLost callback if any, 
and then rejoins the group. If while the callback completes the member attempts 
to leave the group (ex. unsubscribe), the leave operation detects that the 
member is already removed from the group (fenced), and just aligns the client 
state with the current broker state, and marks the client as UNSUBSCRIBED 
(client side state for not in group). 

This means that the member could attempt to rejoin the group if the user calls 
subscribe, get an assignment, and trigger onPartitionsAssigned, when maybe the 
onPartitionsLost hasn't completed.

This approach keeps the client state machine simple given that it does not need 
to block the new member (it will effectively be a new member because the old 
one got fenced). The new member could rejoin, get an assignment and make 
progress. Downside is that it would potentially allow for overlapped callback 
executions (lost and assign) in the above edge case, which is not the behaviour 
in the old coordinator. Review and validate. Alternative would definitely 
require more complex logic on the client to ensure that we do not allow a new 
member to rejoin until the fenced one completes the callback



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15836) KafkaConsumer subscribe to multiple topics does not respect max.poll.records

[Philip Nee (Jira)]

Philip Nee created KAFKA-15836:
--

 Summary: KafkaConsumer subscribe to multiple topics does not 
respect max.poll.records
 Key: KAFKA-15836
 URL: https://issues.apache.org/jira/browse/KAFKA-15836
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Philip Nee
Assignee: Kirk True


We discovered that when KafkaConsumer subscribes to multiple topics with 
max.poll.record configured.  The max.poll.record is not properly respected for 
all poll() invocation.

 

I was able to reproduce it with the AK example, here is how I ran my tests:

[https://github.com/apache/kafka/pull/14772]

 

1. start zookeeper and kafka server (or kraft mode should be fine too)

2. Run: examples/bin/java-producer-consumer-demo.sh 1000

3. Polled records > 400 will be printed to stdout

 

Here is what the program does:

The produce produces a large number of records to multiple topics.  We 
configure the consumer using a max.poll.record = 400, and subscribed to 
multiple topics.  The consumer poll, and the returned records can sometimes be 
larger than 400.

 

This is an issue in AK 3.6 but 3.5 was fine.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Subscribe to mailing list

[Josep Prat]

Hi Mital,
Thanks for your interest in helping out with Apache Kafka.
You can find the instructions on how to subscribe to the mailing list here:
https://kafka.apache.org/contact.html

|>
*Developer mailing list: A list for discussion on Kafka® development. To
subscribe, send an email to dev-subscr...@kafka.apache.org
. Once subscribed, you can have discussion
on Kafka® development by mailing to dev@kafka.apache.org
. Archives are available here
<https://lists.apache.org/list.html?dev@kafka.apache.org>. To unsubscribe,
send an email to dev-unsubscr...@kafka.apache.org
.*

Best,
———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Sun, Jul 16, 2023, 15:54 Mital Awachat  wrote:

> Hi,
>
> I would like to subscribe to the mailing list.
>
> --
> Regards
> Mital Awachat
>

Subscribe to mailing list

[Mital Awachat]

Hi,

I would like to subscribe to the mailing list.

-- 
Regards
Mital Awachat

Re: Subscribe to Kafka dev mailing list

[Divij Vaidya]

Hello

You sent this email to the wrong email address. Please find instruction on
how to subscribe at https://kafka.apache.org/contact

--
Divij Vaidya



On Sun, Jul 2, 2023 at 10:59 PM 最红 <1559063...@qq.com.invalid> wrote:

> Subscribe to Kafka dev mailing list

Re: Subscribe to Kafka dev mailing list

[Mickael Maison]

Hi,

To subscribe to this list, you need to send an email to
dev-subscr...@kafka.apache.org.
See the instructions on https://kafka.apache.org/contact

Thanks,
Mickael

On Thu, May 4, 2023 at 10:09 AM Hao Zhang  wrote:
>
> Subscribe to Kafka dev mailing list.

Subscribe to Kafka dev mailing list

[Hao Zhang]

Subscribe to Kafka dev mailing list.

subscribe kafka issues

[ChunYa Sun]

subscribe Kafka issues

Re: Please subscribe me for Kafka Developer mailing list

[Mickael Maison]

Hi Amit,

Subscription to the mailing lists is self served. See the steps listed
in https://kafka.apache.org/contact:
> To subscribe, send an email to dev-subscr...@kafka.apache.org. Once 
> subscribed, you can have discussion on Kafka® development by mailing to 
> dev@kafka.apache.org.

Thanks,
Mickael

On Mon, Aug 1, 2022 at 4:10 PM Amit Ramswaroop Baheti
 wrote:
>
> Hi,
>
> I will taking over the power support for Apache Kafka upstream. Kindly add me 
> to the developer subscription list.
>
> Thanks,
> Amit Baheti

Please subscribe me for Kafka Developer mailing list

[Amit Ramswaroop Baheti]

Hi,

I will taking over the power support for Apache Kafka upstream. Kindly add me 
to the developer subscription list.

Thanks,
Amit Baheti

Re: Subscribe to Kafka dev mailing list

[Tom Bentley]

Hi,

To subscribe you need to send email to dev-subscr...@kafka.apache.org, not
dev@

Full instructions: https://kafka.apache.org/contact

Kind regards,

Tom

On Tue, 29 Mar 2022 at 01:29, 那一叶零落 <1664637...@qq.com.invalid> wrote:

> Subscribe to Kafka dev mailing list

Subscribe to Kafka dev mailing list

[??????????]

Subscribe to Kafka dev mailing list

Re: Subscribe to Kafka dev mailing list

[Matthias J. Sax]

If you want to subscribe, please follow the instruction on the web-page:
https://kafka.apache.org/contact

-Matthias

On 7/16/21 7:03 PM, 我 wrote:
> Subscribe to Kafka dev mailing list
> 

Subscribe to Kafka dev mailing list

[我]

Subscribe to Kafka dev mailing list

subscribe to kafka dev

[姜家俊]

subscribe to kafka dev mailing list

Subscribe to kafka dev mailing list

[????]

Subscribe to Kafka dev mailing list

Re: Request to subscribe to kafka mailing list

[Guozhang Wang]

Hi SaiTejia,

You can add yourself to the mailing list, it's self service:


https://kafka.apache.org/contact


Guozhang

On Sat, Aug 29, 2020 at 8:35 AM SaiTeja Ramisetty 
wrote:

> Regards,
> SaiTeja - Data Engineer
>


-- 
-- Guozhang

Request to subscribe to kafka mailing list

[SaiTeja Ramisetty]

Regards,
SaiTeja - Data Engineer

Re: KIP idea: Separate publish request from the subscribe request

[Ming Liu]

Hi Guozhang,
 Yes, the goal is to reduce latency of produce latency from the
consumer fetch requests. So the best separate in theory would be:
1. Socket server 1:  All produce requests and all follower fetch
requests  (ACK=ALL)
2. Socket server 2:  All other requests (metadata,
commit/fetch-offsets, and consumer fetch requests).
   Just want to make sure the idea is reasonable before working on a more
detailed KIP design.

   We tried increasing num.network.threads (to 48), but the produce latency
still remains very high (we have > 5000 fanout ratio).

Thanks!
Ming

On 2020/08/24 18:49:27, Guozhang Wang  wrote:
> Hello Ming,
>
> Thanks for bringing this to the community. Just to clarify your proposal,
> are you suggesting to use a separate port for fetch requests from all
other
> requests including produce, but also e.g. metadata, commit/fetch-offsets,
> and other inter-broker requests? If yes that would mean the consumer
> metadata request should then be handled differently from producer metadata
> request so that they can return different port values, and also on the
> server side we'd need two socket servers as well. Maybe we can get a more
> concrete design on how this would work end-to-end for others to review.
>
> Also I'm wondering if you have tried increasing the num.network.threads
and
> see if that helps with the large fanout multiplexing issue other than
> separating them to different ports?
>
>
> Guozhang
>
> On Thu, Aug 20, 2020 at 7:03 PM Ming Liu  wrote:
>
> > Hi Kafka community,
> >I like to surface a KIP idea, which is to separate publish request
from
> > the subscribe request using different ports.
> >
> >The context: We have some workload with over 5000 subscribers, the
> > latency on publish latency can be as high as 3000 ms second. After
> > investigation, we found the reason is mainly because there are too many
> > connections on socketserver and the multiplexing slows down the publish
> > latency.
> >
> >The proposal is somewhat similar to KIP-291: Separating controller
> > connections and requests from the data plane
> >
> >I like to check with experts here whether this is a viable idea to
> > continue pursuing or not?
> >
> > Thanks!
> > Ming
> >
>
>
> --
> -- Guozhang
>

Re: KIP idea: Separate publish request from the subscribe request

[Guozhang Wang]

Hello Ming,

Thanks for bringing this to the community. Just to clarify your proposal,
are you suggesting to use a separate port for fetch requests from all other
requests including produce, but also e.g. metadata, commit/fetch-offsets,
and other inter-broker requests? If yes that would mean the consumer
metadata request should then be handled differently from producer metadata
request so that they can return different port values, and also on the
server side we'd need two socket servers as well. Maybe we can get a more
concrete design on how this would work end-to-end for others to review.

Also I'm wondering if you have tried increasing the num.network.threads and
see if that helps with the large fanout multiplexing issue other than
separating them to different ports?


Guozhang

On Thu, Aug 20, 2020 at 7:03 PM Ming Liu  wrote:

> Hi Kafka community,
>I like to surface a KIP idea, which is to separate publish request from
> the subscribe request using different ports.
>
>The context: We have some workload with over 5000 subscribers, the
> latency on publish latency can be as high as 3000 ms second. After
> investigation, we found the reason is mainly because there are too many
> connections on socketserver and the multiplexing slows down the publish
> latency.
>
>The proposal is somewhat similar to KIP-291: Separating controller
> connections and requests from the data plane
>
>I like to check with experts here whether this is a viable idea to
> continue pursuing or not?
>
> Thanks!
> Ming
>


-- 
-- Guozhang

KIP idea: Separate publish request from the subscribe request

[Ming Liu]

Hi Kafka community,
   I like to surface a KIP idea, which is to separate publish request from
the subscribe request using different ports.

   The context: We have some workload with over 5000 subscribers, the
latency on publish latency can be as high as 3000 ms second. After
investigation, we found the reason is mainly because there are too many
connections on socketserver and the multiplexing slows down the publish
latency.

   The proposal is somewhat similar to KIP-291: Separating controller
connections and requests from the data plane

   I like to check with experts here whether this is a viable idea to
continue pursuing or not?

Thanks!
Ming

Re: subscribe confirm

[Matthias J. Sax]

Roc,

if you want to subscribe to the mailing list, please follow the
instruction from the web page: https://kafka.apache.org/contact

-Matthias

On 6/6/20 9:32 AM, Roc Marshal wrote:
> flin...@126.com
> 



signature.asc
Description: OpenPGP digital signature

subscribe confirm

[Roc Marshal]

flin...@126.com

Re: Please I Want to Subscribe

[Ricardo Ferreira]

Thanks for the heads up!

Yeah, I saw the right email but ended up sending the subscribe request 
for the wrong one =)


Everything is fixed now.

Thanks,

-- Ricardo

On 6/4/20 12:40 PM, Guozhang Wang wrote:

Hello Ricardo,

Here you go for the guidelines, it is self-service:
https://kafka.apache.org/contact


Guozhang

On Wed, Jun 3, 2020 at 1:09 PM Ricardo Ferreira 
wrote:


Please I Want to Subscribe

Re: Please I Want to Subscribe

[Guozhang Wang]

Hello Ricardo,

Here you go for the guidelines, it is self-service:
https://kafka.apache.org/contact


Guozhang

On Wed, Jun 3, 2020 at 1:09 PM Ricardo Ferreira 
wrote:

> Please I Want to Subscribe
>
>

-- 
-- Guozhang

Please I Want to Subscribe

[Ricardo Ferreira]

Please I Want to Subscribe

Subscribe to Kafka dev mailing list

[108414055]

发自我的iPhone

subscribe kafka dev list

[Shubham Bhindwal]

subscription to dev

subscribe kafka dev mail

[Walker Xia]

subscribe kafka dev mail

Re: Need to subscribe to dev group

[Sriram Ganesh]

Thanks, I'll check it out.

On Sun, 20 Oct 2019, 12:16 Matthias J. Sax,  wrote:

> It's self-service: https://kafka.apache.org/contact
>
> Also check out:
>  - https://kafka.apache.org/contributing
>  -
> https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes
>
>
> -Matthias
>
> On 10/17/19 1:59 AM, Sriram Ganesh wrote:
> > Hi,
> >
> > I was using kafka for last 3 years. I wanna contribute to kafka. Please
> add
> > me in the "dev@kafka.apache.org"
> >
> > Thanks,
> >
>
>

Re: Need to subscribe to dev group

[Matthias J. Sax]

It's self-service: https://kafka.apache.org/contact

Also check out:
 - https://kafka.apache.org/contributing
 -
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes


-Matthias

On 10/17/19 1:59 AM, Sriram Ganesh wrote:
> Hi,
> 
> I was using kafka for last 3 years. I wanna contribute to kafka. Please add
> me in the "dev@kafka.apache.org"
> 
> Thanks,
> 



signature.asc
Description: OpenPGP digital signature

Need to subscribe to dev group

[Sriram Ganesh]

Hi,

I was using kafka for last 3 years. I wanna contribute to kafka. Please add
me in the "dev@kafka.apache.org"

Thanks,

-- 
*Sriram G*
*Tech*

[jira] [Created] (KAFKA-9043) Problem committing offsets when using consumer subscribe and assign method on different topics with same group.id

[Michael Arndt (Jira)]

Michael Arndt created KAFKA-9043:


 Summary: Problem committing offsets when using consumer subscribe 
and assign method on different topics with same group.id
 Key: KAFKA-9043
 URL: https://issues.apache.org/jira/browse/KAFKA-9043
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 2.3.0
 Environment: not relevant
Reporter: Michael Arndt


I am using a consumer with group.id "myGroup" on an *_empty_* topic "election" 
with 24 partitions in subscribe mode. The poll loop is a noop and is for 
rebalancing only. On new assigments (only the partitions are relevant) I start 
two consumers per assigned partition on other topics ("foo" and "bar") using 
assignments on a specific partition/topic.

Summary:

consumer1: election-0, election-1 (subscribe)
consumer2: foo-0 (assign)
consumer3: bar-0 (assign)
consumer4: foo-1 (assign)
consumer5: bar-1 (assign)

The reason for this setup is a join on time and a certain key in the topics foo 
and bar, where the content in one partition is complete for the key.

However, consumer1 never commits since the topic is empty and just for 
rebalancing issues that will guarantee that I will always read from the same 
partitions from foo and bar in this service instance.

When trying to commit consumer2-4 (they are using the same group-id as 
consumer1) the ClientCoordinator reports an error telling the consumer is not 
responsible for this topic and partition. This is an error raised at the server.

When using different group-ids for consumer1 and 2-4 the commit works fine.

This behaviour is somehow reasonable since the server only knows about the 
subscription, but since assignment and subscription run on different topics 
it's at least a lack in documentation.

As this will also happen when using assign and subscribe from different clients 
on differnt topis one would not be able to use a common group-id for 
identification issues (e.g. team related; useful when working with ACLs in an 
environment where on kafka serves many teams and you want to limit the number 
of group ids managed in the ACLs).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: mixing assign and subscribe for a group

[Ryanne Dolan]

Shailesh, the users list may be a better place for this question.

Assign() and subscribe() cannot be used on the same Consumer. Moreover, an
assign()'d Consumer is not really a member of a group.

You should use AdminClient.listConsumerGroupOffsets() for this purpose.

Ryanne

On Thu, Aug 22, 2019 at 9:35 AM Ranjan, Shailesh (Contractor - CB - Front
Office L, Group IT)  wrote:

> Classification: Public
>
> Hi All,
>
> We have a kafka cluster where we are using consumer.subscribe() for a
> group id. We also are trying to write a simple monitoring tool running as a
> separate instance on a different host than the consumers for checking the
> consumer offsets and for that we have used below code which as I
> understand is mixing assign and subscribe here.
>
> I wanted to understand what problems/issues will happen if I have the
> below code where I have assigned a partition in to a new instance (which
> would have been assigned to a consumer, what happens to that consumers
> ownership of the partition?) belonging to the same group but not calling
> poll() but just getting the current position.
>
> Code snippet:
>
> try (KafkaConsumer consumer =
> createConsumer(consumerGroup)) {
>TopicPartition topicPartition = new TopicPartition(topic,
> Integer.parseInt(partition));
>consumer.assign(Collections.singletonList(topicPartition));
>return consumer.position(topicPartition);
> } catch (Exception ex) {
>log.error("Error occurres when try to get consumer group position");
>return -1;
> }
>
>
> Thanks,
> Shailesh
> Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ.
> Registered in Scotland no. SC95000. Telephone: 0131 225 4555.
>
> Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN.
> Registered in England and Wales no. 2065. Telephone 0207626 1500.
>
> Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ.
> Registered in Scotland no. SC327000. Telephone: 03457 801 801.
>
> Lloyds Bank Corporate Markets plc. Registered office: 25 Gresham Street,
> London EC2V 7HN. Registered in England and Wales no. 10399850.
>
> Scottish Widows Schroder Personal Wealth Limited. Registered Office: 25
> Gresham Street, London EC2V 7HN. Registered in England and Wales no.
> 11722983.
>
> Lloyds Bank plc, Bank of Scotland plc and Lloyds Bank Corporate Markets
> plc are authorised by the Prudential Regulation Authority and regulated by
> the Financial Conduct Authority and Prudential Regulation Authority.
>
> Scottish Widows Schroder Personal Wealth Limited is authorised and
> regulated by the Financial Conduct Authority.
>
> Lloyds Bank Corporate Markets Wertpapierhandelsbank GmbH is a wholly-owned
> subsidiary of Lloyds Bank Corporate Markets plc. Lloyds Bank Corporate
> Markets Wertpapierhandelsbank GmbH has its registered office at
> Thurn-und-Taxis Platz 6, 60313 Frankfurt, Germany. The company is
> registered with the Amtsgericht Frankfurt am Main, HRB 111650. Lloyds Bank
> Corporate Markets Wertpapierhandelsbank GmbH is supervised by the
> Bundesanstalt für Finanzdienstleistungsaufsicht.
>
> Halifax is a division of Bank of Scotland plc.
>
> HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in
> Scotland no. SC218813.
>
>
>
> This e-mail (including any attachments) is private and confidential and
> may contain privileged material. If you have received this e-mail in error,
> please notify the sender and delete it (including any attachments)
> immediately. You must not copy, distribute, disclose or use any of the
> information in it or any attachments. Telephone calls may be monitored or
> recorded.
>

mixing assign and subscribe for a group

[Ranjan, Shailesh (Contractor - CB - Front Office L, Group IT)]

Classification: Public

Hi All,

We have a kafka cluster where we are using consumer.subscribe() for a group id. 
We also are trying to write a simple monitoring tool running as a separate 
instance on a different host than the consumers for checking the consumer 
offsets and for that we have used below code which as I  understand is mixing 
assign and subscribe here.

I wanted to understand what problems/issues will happen if I have the below 
code where I have assigned a partition in to a new instance (which would have 
been assigned to a consumer, what happens to that consumers ownership of the 
partition?) belonging to the same group but not calling poll() but just getting 
the current position.

Code snippet:

try (KafkaConsumer consumer = createConsumer(consumerGroup)) {
   TopicPartition topicPartition = new TopicPartition(topic, 
Integer.parseInt(partition));
   consumer.assign(Collections.singletonList(topicPartition));
   return consumer.position(topicPartition);
} catch (Exception ex) {
   log.error("Error occurres when try to get consumer group position");
   return -1;
}


Thanks,
Shailesh
Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. 
Registered in Scotland no. SC95000. Telephone: 0131 225 4555.

Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. 
Registered in England and Wales no. 2065. Telephone 0207626 1500.

Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. 
Registered in Scotland no. SC327000. Telephone: 03457 801 801.

Lloyds Bank Corporate Markets plc. Registered office: 25 Gresham Street, London 
EC2V 7HN. Registered in England and Wales no. 10399850.

Scottish Widows Schroder Personal Wealth Limited. Registered Office: 25 Gresham 
Street, London EC2V 7HN. Registered in England and Wales no. 11722983.

Lloyds Bank plc, Bank of Scotland plc and Lloyds Bank Corporate Markets plc are 
authorised by the Prudential Regulation Authority and regulated by the 
Financial Conduct Authority and Prudential Regulation Authority.

Scottish Widows Schroder Personal Wealth Limited is authorised and regulated by 
the Financial Conduct Authority.

Lloyds Bank Corporate Markets Wertpapierhandelsbank GmbH is a wholly-owned 
subsidiary of Lloyds Bank Corporate Markets plc. Lloyds Bank Corporate Markets 
Wertpapierhandelsbank GmbH has its registered office at Thurn-und-Taxis Platz 
6, 60313 Frankfurt, Germany. The company is registered with the Amtsgericht 
Frankfurt am Main, HRB 111650. Lloyds Bank Corporate Markets 
Wertpapierhandelsbank GmbH is supervised by the Bundesanstalt für 
Finanzdienstleistungsaufsicht.

Halifax is a division of Bank of Scotland plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in 
Scotland no. SC218813.



This e-mail (including any attachments) is private and confidential and may 
contain privileged material. If you have received this e-mail in error, please 
notify the sender and delete it (including any attachments) immediately. You 
must not copy, distribute, disclose or use any of the information in it or any 
attachments. Telephone calls may be monitored or recorded.

[CSOT]subscribe

[佘迎松]

subscribe

[jira] [Created] (KAFKA-8420) Graceful handling when consumer switches from subscribe to manual assign

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-8420:


 Summary: Graceful handling when consumer switches from subscribe 
to manual assign
 Key: KAFKA-8420
 URL: https://issues.apache.org/jira/browse/KAFKA-8420
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Reporter: Guozhang Wang


Today if a consumer switches between subscribe (and hence relies on group 
rebalance to get assignment) and manual assign, it may cause unnecessary 
rebalances. For example:

1. consumer.subscribe();
2. consumer.poll(); // join-group request sent, returns empty because poll 
timeout
3. consumer.unsubscribe();
4. consumer.assign(..);
5. consumer.poll(); // sync-group request received, and the assigned 
partitions does not match the current subscription-state. In this case it will 
tries to re-join which is not necessary.

In the worst case (i.e. leader keep sending incompatible assignment), this 
would case the consumer to fall into endless re-joins.

Although it is not a very common usage scenario, it still worth being better 
handled than the status-quo.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: confirm subscribe to dev@kafka.apache.org

[Amol Badgujar]

Please add amol.badgu...@gmail.com to dev mailing list for Apache Kafka.

On Mon, Dec 24, 2018 at 7:06 PM  wrote:

> Hi! This is the ezmlm program. I'm managing the
> dev@kafka.apache.org mailing list.
>
> I'm working for my owner, who can be reached
> at dev-ow...@kafka.apache.org.
>
> To confirm that you would like
>
>amol.badgu...@gmail.com
>
> added to the dev mailing list, please send
> a short reply to this address:
>
>dev-sc.1545658570.pphchgpmmpcfaaknmgpn-amol.badgujar=
> gmail@kafka.apache.org
>
> Usually, this happens when you just hit the "reply" button.
> If this does not work, simply copy the address and paste it into
> the "To:" field of a new message.
>
> or click here:
> mailto:dev-sc.1545658570.pphchgpmmpcfaaknmgpn-amol.badgujar=
> gmail@kafka.apache.org
>
> This confirmation serves two purposes. First, it verifies that I am able
> to get mail through to you. Second, it protects you in case someone
> forges a subscription request in your name.
>
> Please note that ALL Apache dev- and user- mailing lists are publicly
> archived.  Do familiarize yourself with Apache's public archive policy at
>
> http://www.apache.org/foundation/public-archives.html
>
> prior to subscribing and posting messages to dev@kafka.apache.org.
> If you're not sure whether or not the policy applies to this mailing list,
> assume it does unless the list name contains the word "private" in it.
>
> Some mail programs are broken and cannot handle long addresses. If you
> cannot reply to this request, instead send a message to
>  and put the
> entire address listed above into the "Subject:" line.
>
>
> --- Administrative commands for the dev list ---
>
> I can handle administrative requests automatically. Please
> do not send them to the list address! Instead, send
> your message to the correct command address:
>
> To subscribe to the list, send a message to:
>
>
> To remove your address from the list, send a message to:
>
>
> Send mail to the following for info and FAQ for this list:
>
>
>
> Similar addresses exist for the digest list:
>
>
>
> To get messages 123 through 145 (a maximum of 100 per request), mail:
>
>
> To get an index with subject and author for messages 123-456 , mail:
>
>
> They are always returned as sets of 100, max 2000 per request,
> so you'll actually get 100-499.
>
> To receive all messages with the same subject as message 12345,
> send a short message to:
>
>
> The messages should contain one line or word of text to avoid being
> treated as sp@m, but I will ignore their content.
> Only the ADDRESS you send to is important.
>
> You can start a subscription for an alternate address,
> for example "john@host.domain", just add a hyphen and your
> address (with '=' instead of '@') after the command word:
> 
>
> To stop subscription for this address, mail:
> 
>
> In both cases, I'll send a confirmation message to that address. When
> you receive it, simply reply to it to complete your subscription.
>
> If despite following these instructions, you do not get the
> desired results, please contact my owner at
> dev-ow...@kafka.apache.org. Please be patient, my owner is a
> lot slower than I am ;-)
>
> --- Enclosed is a copy of the request I received.
>
> Return-Path: 
> Received: (qmail 34821 invoked by uid 99); 24 Dec 2018 13:36:10 -
> Received: from pnap-us-west-generic-nat.apache.org (HELO
> spamd4-us-west.apache.org) (209.188.14.142)
> by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 24 Dec 2018 13:36:10
> +
> Received: from localhost (localhost [127.0.0.1])
> by spamd4-us-west.apache.org (ASF Mail Server at
> spamd4-us-west.apache.org) with ESMTP id C595CC0791
> for ; Mon, 24 Dec 2018 13:36:09
> + (UTC)
> X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org
> X-Spam-Flag: NO
> X-Spam-Score: 1.798
> X-Spam-Level: *
> X-Spam-Status: No, score=1.798 tagged_above=-999 required=6.31
> tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
> DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2,
> RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled
> Authentication-Results: spamd4-us-west.apache.org (amavisd-new);
> dkim=pass (2048-bit key) header.d=gmail.com
> Received: from mx1-lw-eu.apache.org ([10.40.0.8])
> by localhost (spamd4-us-west.apache.org [10.40.0.11])
> (amavisd-new, port 10024)
> with ESMTP id 2VwOUOdML7LX for ;
> Mon, 24 Dec 2018 13:36:08 + (UTC)
> Received: from mail-lj1-f178.google.com (

Re: Need to subscribe to mail list and get access to contribute to jira tickets

[Matthias J. Sax]

Mailing list subscription is self-service: https://kafka.apache.org/contact

On 11/20/18 8:39 AM, Manikumar wrote:
> Hi,
> 
> I have given JIRA permissions for "kaushik srinivas" JIRA username.
> 
> On Tue, Nov 20, 2018 at 3:29 PM KAUSHIK SRINIVAS <
> kaushiksrinivas...@gmail.com> wrote:
> 
>> Hi,
>>
>> Need subscription to kafka mailing list.
>>
>> Also need to assign jira tickets to myself. Have worked on few pull
>> requests and need to submit the code.
>>
>> Need support in getting the required permissions to assign the kafka jira
>> ticket to myself.
>>
>> Thanks & Regards,
>> kaushik
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: Need to subscribe to mail list and get access to contribute to jira tickets

[Manikumar]

Hi,

I have given JIRA permissions for "kaushik srinivas" JIRA username.

On Tue, Nov 20, 2018 at 3:29 PM KAUSHIK SRINIVAS <
kaushiksrinivas...@gmail.com> wrote:

> Hi,
>
> Need subscription to kafka mailing list.
>
> Also need to assign jira tickets to myself. Have worked on few pull
> requests and need to submit the code.
>
> Need support in getting the required permissions to assign the kafka jira
> ticket to myself.
>
> Thanks & Regards,
> kaushik
>

Need to subscribe to mail list and get access to contribute to jira tickets

[KAUSHIK SRINIVAS]

Hi,

Need subscription to kafka mailing list.

Also need to assign jira tickets to myself. Have worked on few pull
requests and need to submit the code.

Need support in getting the required permissions to assign the kafka jira
ticket to myself.

Thanks & Regards,
kaushik

confirm subscribe to dev@kafka.apache.org

[黃旭威]

JIRA ID: doniel0...@gmail.com
Cwiki ID: DANIEL HUANG

[jira] [Resolved] (KAFKA-5789) Deleted topic is recreated when consumer subscribe the deleted one

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5789.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-7320/KIP-361

> Deleted topic is recreated when consumer subscribe the deleted one
> --
>
> Key: KAFKA-5789
> URL: https://issues.apache.org/jira/browse/KAFKA-5789
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.11.0.0
>Reporter: Takafumi Saito
>Priority: Major
>  Labels: needs-kip
>
> When setting auto.create.topic.enbale true in broker, some deleted topics 
> will be re-created.
> Because when consumer that subscribe deleted topic is exist, broker will 
> create topic having same name.
> It is not necessary that consumers trigger new topic creation , so auto topic 
> creation in consumer should be disabled.
> I attatch the log outputted in our broker.
> This show that a topic (topic_1) was deleted at 12:02:24,672, but same topic 
> was created shortly thereafter:
> {code:java}
> [2017-08-22 12:02:24,666] INFO [ReplicaFetcherManager on broker 1] Removed 
> fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
> [2017-08-22 12:02:24,666] INFO [ReplicaFetcherManager on broker 1] Removed 
> fetcher for partitions  (kafka.server.ReplicaFetcherManager)
> [2017-08-22 12:02:24,667] INFO [ReplicaFetcherManager on broker 1] Removed 
> fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
> [2017-08-22 12:02:24,672] INFO Log for partition topic_1-0 is renamed to 
> /data/topic_1-0.ad490e8326704ae6a6fd9f6399c29614-delete and is scheduled for 
> deletion (kafka.log.LogManager)
> [2017-08-22 12:02:24,736] INFO Loading producer state from offset 0 for 
> partition topic_1-0 with message format version 2 (kafka.log.Log)
> [2017-08-22 12:02:24,736] INFO Completed load of log topic_1-0 with 1 log 
> segments, log start offset 0 and log end offset 0 in 1 ms (kafka.log.Log)
> [2017-08-22 12:02:24,737] INFO Created log for partition [topic_1,0] in /data 
> with properties {compression.type -> producer, message.format.version -> 
> 0.11.0-IV2, file.delete.delay.ms -> 6,
> max.message.bytes -> 112, min.compaction.lag.ms -> 0, 
> message.timestamp.type -> CreateTime, min.insync.replicas -> 1, 
> segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 
> 0.5, i
> ndex.interval.bytes -> 4096, unclean.leader.election.enable -> false, 
> retention.bytes -> -1, delete.retention.ms -> 8640, cleanup.policy -> 
> [delete], flush.ms -> 9223372036854775807, segment.ms -> 60
> 480, segment.bytes -> 1073741824, retention.ms -> 8640, 
> message.timestamp.difference.max.ms -> 9223372036854775807, 
> segment.index.bytes -> 10485760, flush.messages -> 9223372036854775807}. 
> (kafka
> .log.LogManager)
> [2017-08-22 12:02:24,738] INFO [ReplicaFetcherManager on broker 1] Removed 
> fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
> [2017-08-22 12:02:24,738] INFO [ReplicaFetcherManager on broker 1] Added 
> fetcher for partitions List([topic_1-0, initOffset 0 to broker 
> BrokerEndPoint(2,sbx-patriot-kafka02.amb-patriot.incvb.io,
> 9092)] ) (kafka.server.ReplicaFetcherManager)
> [2017-08-22 12:02:25,200] INFO [ReplicaFetcherThread-0-2]: Based on 
> follower's leader epoch, leader replied with an offset 0 >= the follower's 
> log end offset 0 in topic_1-0. No truncation needed
> . (kafka.server.ReplicaFetcherThread)
> [2017-08-22 12:02:25,200] INFO Truncating topic_1-0 to 0 has no effect as the 
> largest offset in the log is -1. (kafka.log.Log)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: subscribe

[Ted Yu]

Please see instructions here:

http://kafka.apache.org/contact

On Sat, Aug 18, 2018 at 8:18 AM Aegeaner 
wrote:

>
>
>

Re: subscribe mail list

[Matthias J. Sax]

To subscribe, please follow instructions here:
https://kafka.apache.org/contact


On 5/24/18 8:16 PM,  wrote:
> subscribe mail list
> 



signature.asc
Description: OpenPGP digital signature

subscribe mail list

[????????????]

subscribe mail list

Re: please subscribe me to dev list

[Matthias J. Sax]

It's self-service: https://kafka.apache.org/contact

-Matthias


On 5/7/18 5:13 PM, Ravi Chinoy wrote:



signature.asc
Description: OpenPGP digital signature

please subscribe me to dev list

[Ravi Chinoy]

-- 
Regards
Ravi Chinoy
Senior Scalability Engineer
JSMN Inc.
http://www.jsmninc.com/ 
Phone: (415) 230 9971

try to subscribe

[赖志滨]

Re: Please subscribe me in your updates

[Tom Bentley]

As detailed at https://kafka.apache.org/contact, to subscribe, send an
email to dev-subscr...@kafka.apache.org < dev-subscr...@kafka.apache.org>.



On 22 October 2017 at 15:15, Veeramani S <veeraman...@gmail.com> wrote:

>
>

Please subscribe me in your updates

[Veeramani S]

Can you please subscribe me in this project

[Nikhil Deore]

Hi,

I want to learn and contribute to this project,
Please subscribe me in.

Thanks,
Nikhil

[jira] [Created] (KAFKA-5848) KafkaConsumer should validate topics/TopicPartitions on subscribe/assign

[Matthias J. Sax (JIRA)]

Matthias J. Sax created KAFKA-5848:
--

 Summary: KafkaConsumer should validate topics/TopicPartitions on 
subscribe/assign
 Key: KAFKA-5848
 URL: https://issues.apache.org/jira/browse/KAFKA-5848
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.11.0.0
Reporter: Matthias J. Sax
Priority: Minor


Currently, {{KafkaConsumer}} checks if the provided topics on {{subscribe()}} 
and {{TopicPartition}} on {{assign()}} don't contain topic names that are 
{{null}} or an empty string. 

However, it could do some more validation:
 - check if invalid topic characters are in the string (this might be feasible 
for {Patterns}}, too?)
 - check if provided partition numbers are valid (ie, not negative and maybe 
not larger than the available partitions?)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5846) Use singleton NoOpConsumerRebalanceListener in subscribe() call where listener is not specified

[Ted Yu (JIRA)]

Ted Yu created KAFKA-5846:
-

 Summary: Use singleton NoOpConsumerRebalanceListener in 
subscribe() call where listener is not specified
 Key: KAFKA-5846
 URL: https://issues.apache.org/jira/browse/KAFKA-5846
 Project: Kafka
  Issue Type: Task
Reporter: Ted Yu
Priority: Minor


Currently KafkaConsumer creates instance of NoOpConsumerRebalanceListener for 
each subscribe() call where ConsumerRebalanceListener is not specified:
{code}
public void subscribe(Pattern pattern) {
subscribe(pattern, new NoOpConsumerRebalanceListener());
{code}
We can create a singleton NoOpConsumerRebalanceListener to be used in such 
scenarios.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KAFKA-5789) Deleted topic is recreated when consumer subscribe the deleted one

[Takafumi Saito (JIRA)]

Takafumi Saito created KAFKA-5789:
-

 Summary: Deleted topic is recreated when consumer subscribe the 
deleted one
 Key: KAFKA-5789
 URL: https://issues.apache.org/jira/browse/KAFKA-5789
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.11.0.0
Reporter: Takafumi Saito


When setting auto.create.topic.enbale true in broker, some deleted topics will 
be re-created.
Because when consumer that subscribe deleted topic is exist, broker will create 
topic having same name.
It is not necessary that consumers trigger new topic creation , so auto topic 
creation in consumer should be disabled.

I attatch the log outputted in our broker.
This show that a topic (topic_1) was deleted at 12:02:24,672, but same topic 
was created shortly thereafter:

{code:java}
[2017-08-22 12:02:24,666] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,666] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions  (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,667] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,672] INFO Log for partition topic_1-0 is renamed to 
/data/topic_1-0.ad490e8326704ae6a6fd9f6399c29614-delete and is scheduled for 
deletion (kafka.log.LogManager)
[2017-08-22 12:02:24,736] INFO Loading producer state from offset 0 for 
partition topic_1-0 with message format version 2 (kafka.log.Log)
[2017-08-22 12:02:24,736] INFO Completed load of log topic_1-0 with 1 log 
segments, log start offset 0 and log end offset 0 in 1 ms (kafka.log.Log)
[2017-08-22 12:02:24,737] INFO Created log for partition [topic_1,0] in /data 
with properties {compression.type -> producer, message.format.version -> 
0.11.0-IV2, file.delete.delay.ms -> 6,
max.message.bytes -> 112, min.compaction.lag.ms -> 0, 
message.timestamp.type -> CreateTime, min.insync.replicas -> 1, 
segment.jitter.ms -> 0, preallocate -> false, min.cleanable.dirty.ratio -> 0.5, 
i
ndex.interval.bytes -> 4096, unclean.leader.election.enable -> false, 
retention.bytes -> -1, delete.retention.ms -> 8640, cleanup.policy -> 
[delete], flush.ms -> 9223372036854775807, segment.ms -> 60
480, segment.bytes -> 1073741824, retention.ms -> 8640, 
message.timestamp.difference.max.ms -> 9223372036854775807, segment.index.bytes 
-> 10485760, flush.messages -> 9223372036854775807}. (kafka
.log.LogManager)
[2017-08-22 12:02:24,738] INFO [ReplicaFetcherManager on broker 1] Removed 
fetcher for partitions topic_1-0 (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:24,738] INFO [ReplicaFetcherManager on broker 1] Added 
fetcher for partitions List([topic_1-0, initOffset 0 to broker 
BrokerEndPoint(2,sbx-patriot-kafka02.amb-patriot.incvb.io,
9092)] ) (kafka.server.ReplicaFetcherManager)
[2017-08-22 12:02:25,200] INFO [ReplicaFetcherThread-0-2]: Based on follower's 
leader epoch, leader replied with an offset 0 >= the follower's log end offset 
0 in topic_1-0. No truncation needed
. (kafka.server.ReplicaFetcherThread)
[2017-08-22 12:02:25,200] INFO Truncating topic_1-0 to 0 has no effect as the 
largest offset in the log is -1. (kafka.log.Log)
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KAFKA-5537) Subscribe Earliest is not working as in 0.10.2.1

[Michael Andre Pearce (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Andre Pearce resolved KAFKA-5537.
-
Resolution: Won't Fix

> Subscribe Earliest is not working as in 0.10.2.1
> 
>
> Key: KAFKA-5537
> URL: https://issues.apache.org/jira/browse/KAFKA-5537
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Michael Andre Pearce
>Priority: Critical
> Attachments: kafka_0.10.2.1.log, kafka_0.11.0.0.log, KafkaSub.java, 
> KafkaSubLatest.java
>
>
> We have seen issue with subscription where auto offset when set to earliest 
> (and also latest) does not behave the same as in 0.10.2.1 release.
> We have managed to create a repeatable test for this, which passes when 
> pointing to 0.10.2.1 broker.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Consumer assign vs subscribe influence on __consumer_offset

[Jason Gustafson]

Hey Paolo,

When you use the subscribe() API, you are using Kafka for group
coordination. All members with the same group.id will balance the
partitions of subscribed topics between themselves. When you use assign(),
no such balancing takes place but you can still using Kafka to store
offsets. However, we have a safety check in place such that if there is an
active consumer group (i.e. one using group coordination with the
subscribe() API), then offset commits from consumers that are not part of
that group are rejected. So in your situation, if C1 and C2 are active at
the same time, only C2 will be able to commit offsets. If you look in C1's
log, you should see some errors from the auto offset commits. I believe
this explains the three cases mentioned above.

-Jason

On Thu, Jul 27, 2017 at 3:59 AM, Paolo Patierno <ppatie...@live.com> wrote:

> Hi devs,
>
> I have one simple topic named "test" with only one partition. then ...
>
> I have a consumer C1 using assign() for having assigned partition 0 of
> topic "test" and this consumer has group.id = "mygroup"
>
> I have another consumer C2 using subscribe() for having assigned
> partitions from topic "test" (so it will have assigned partition 0) and
> this consumer has group.id = "mygroup" (the same as above)
>
>
> USE CASE 1
>
> Both consumers have auto commit enabled
>
> Reading messages from the __consumer_offset I always see only ONE message
> of this type
>
>
> [mygroup,test,0]::[OffsetMetadata[7,NO_METADATA],CommitTime
> 1501149394800,ExpirationTime 1501235794800]
>
>
> So even if both consumers receives messages from "test" and have auto
> commit enabled, only one offset commit message is there. Is that normal ?
>
>
> USE CASE 2
>
> Consumer C1 has auto commit enabled
>
> Consumer C2 has auto commit disabled and it doesn't execute any offset
> commit manually
>
> Both consumers get messages from "test" but, there are no offset commit
> messages into __consumer_offset. Is that normal ?
>
> I expect that the C1 with auto commit enabled sends the offset commit
> message but it doesn't happen. It seems that the C2 settings about auto
> commit disabled influences the C1 settings on that.
>
>
> USE CASE 3 (opposite of 2)
>
> Consumer C1 has auto commit disabled
>
> Consumer C2 has auto commit enabled
>
> Both consumers get messages from "test" and I can see offset commit
> messages into __consumer_offset (from C2).
>
>
> [mygroup,test,0]::[OffsetMetadata[7,NO_METADATA],CommitTime
> 1501149394800,ExpirationTime 1501235794800]
>
>
> In this use case, the auto commit disabled by C1 doesn't influence the
> enabled from C2. It happened in the use case 2 (in the opposite direction).
>
>
> Conclusion ... it's possible that there is something that I didn't
> understand about this interaction between consumers in the same group but
> asking for partitions in two different way (assign vs subscribe).
>
>
> Thanks,
>
> Paolo
>
>
> Paolo Patierno
> Senior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoT
> Microsoft Azure Advisor
>
> Twitter : @ppatierno<http://twitter.com/ppatierno>
> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
> Blog : DevExperience<http://paolopatierno.wordpress.com/>
>

Consumer assign vs subscribe influence on __consumer_offset

[Paolo Patierno]

Hi devs,

I have one simple topic named "test" with only one partition. then ...

I have a consumer C1 using assign() for having assigned partition 0 of topic 
"test" and this consumer has group.id = "mygroup"

I have another consumer C2 using subscribe() for having assigned partitions 
from topic "test" (so it will have assigned partition 0) and this consumer has 
group.id = "mygroup" (the same as above)


USE CASE 1

Both consumers have auto commit enabled

Reading messages from the __consumer_offset I always see only ONE message of 
this type


[mygroup,test,0]::[OffsetMetadata[7,NO_METADATA],CommitTime 
1501149394800,ExpirationTime 1501235794800]


So even if both consumers receives messages from "test" and have auto commit 
enabled, only one offset commit message is there. Is that normal ?


USE CASE 2

Consumer C1 has auto commit enabled

Consumer C2 has auto commit disabled and it doesn't execute any offset commit 
manually

Both consumers get messages from "test" but, there are no offset commit 
messages into __consumer_offset. Is that normal ?

I expect that the C1 with auto commit enabled sends the offset commit message 
but it doesn't happen. It seems that the C2 settings about auto commit disabled 
influences the C1 settings on that.


USE CASE 3 (opposite of 2)

Consumer C1 has auto commit disabled

Consumer C2 has auto commit enabled

Both consumers get messages from "test" and I can see offset commit messages 
into __consumer_offset (from C2).


[mygroup,test,0]::[OffsetMetadata[7,NO_METADATA],CommitTime 
1501149394800,ExpirationTime 1501235794800]


In this use case, the auto commit disabled by C1 doesn't influence the enabled 
from C2. It happened in the use case 2 (in the opposite direction).


Conclusion ... it's possible that there is something that I didn't understand 
about this interaction between consumers in the same group but asking for 
partitions in two different way (assign vs subscribe).


Thanks,

Paolo


Paolo Patierno
Senior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoT
Microsoft Azure Advisor

Twitter : @ppatierno<http://twitter.com/ppatierno>
Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno>
Blog : DevExperience<http://paolopatierno.wordpress.com/>

[jira] [Created] (KAFKA-5537) Subscribe Earliest is not working as in 0.10.2.1

[Michael Andre Pearce (JIRA)]

Michael Andre Pearce created KAFKA-5537:
---

 Summary: Subscribe Earliest is not working as in 0.10.2.1
 Key: KAFKA-5537
 URL: https://issues.apache.org/jira/browse/KAFKA-5537
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.11.0.0
Reporter: Michael Andre Pearce
Priority: Critical


We have seen issue with subscription where auto offset when set to earliest 
does not behave the same as in 0.10.2.1 release.

We have managed to create a repeatable test for this, which passes when 
pointing to 0.10.2.1 broker.


@Test
public void testSubscribeEarliest(){

Properties properties = new Properties();
properties.setProperty(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, 
bootstrap);
properties.setProperty(ConsumerConfig.GROUP_ID_CONFIG, "test-group");
properties.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, 
"earliest");
KafkaProducer<byte[], byte[]> kafkaProducer = new 
KafkaProducer<>(properties, new ByteArraySerializer(), new 
ByteArraySerializer());
kafkaProducer.send(new ProducerRecord<>("topic", "hello".getBytes()));


KafkaConsumer<byte[], byte[]> kafkaConsumer =
new KafkaConsumer<>(properties, new ByteArrayDeserializer(), new 
ByteArrayDeserializer());
kafkaConsumer.subscribe(Collections.singletonList("topic"));
ConsumerRecords<byte[], byte[]> consumerRecords = 
kafkaConsumer.poll(1000);

assertEquals(1, consumerRecords.count());
}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Please subscribe me

[Dimple Patel]

Please subscribe me

Re: Subscribe to mailing list

[Matthias J. Sax]

It's self service:

http://kafka.apache.org/contact


-Matthias

On 4/19/17 7:55 AM, Arunkumar wrote:
> Hi There
> I would like to subscribe to this mailing list and know more about kafka. 
> Please add me to the list. Thanks in advance
> 
> Thanks
> Arunkumar Pichaimuthu, PMP
> 



signature.asc
Description: OpenPGP digital signature

Subscribe to mailing list

[Arunkumar]

Hi There
I would like to subscribe to this mailing list and know more about kafka. 
Please add me to the list. Thanks in advance

Thanks
Arunkumar Pichaimuthu, PMP

Subscribe request

[Arun Mathew]

[subscribe Request]

[Ashish Singh]

Hi Team,

I would like to subscribe to Kafka dev group

Thanks,
Ashish Singh

Re: Subscribe to mailing list

[Dhwani Katagade]

Hi Abhishek,

Please follow the instructions here https://kafka.apache.org/contact. 
For your purpose the users list would be more appropriate I feel.


-dhwani

On Wednesday 11 January 2017 03:15 PM, Barot, Abhishek wrote:

Hi Team,

I would like to subscribe for the mailing list ( 
dev@kafka.apache.org<mailto:dev@kafka.apache.org> ). I would like to post few 
queries we've around setting up KAFKA cluster in our production
Environment with respect to ideal configuration for 6 node cluster across 3 
data centers.

Thanks,
Abhishek Barot

--
This message w/attachments (message) is intended solely for the use of the 
intended recipient(s) and may contain information that is privileged, 
confidential or proprietary.  If you are not an intended recipient, please 
notify the sender, and then please delete and destroy all copies and 
attachments, and be advised that any review or dissemination of, or the taking 
of any action in reliance on, the information contained in or attached to this 
message is prohibited.
Unless specifically indicated, this message is not an offer to sell or a 
solicitation of any investment products or other financial product or service, 
an official confirmation of any transaction, or an official statement of 
Sender.  Subject to applicable law, Sender may intercept, monitor, review and 
retain e-communications (EC) traveling through its networks/systems and may 
produce any such EC to regulators, law enforcement, in litigation and as 
required by law.
The laws of the country of each sender/recipient may impact the handling of EC, 
and EC may be archived, supervised and produced in countries other than the 
country in which you are located. This message cannot be guaranteed to be 
secure or free of errors or viruses.  Attachments that are part of this EC may 
have additional important disclosures and disclaimers, which you should read.   
By messaging with Sender you consent to the foregoing.




DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Subscribe to mailing list

[Barot, Abhishek]

Hi Team,

I would like to subscribe for the mailing list ( 
dev@kafka.apache.org<mailto:dev@kafka.apache.org> ). I would like to post few 
queries we've around setting up KAFKA cluster in our production
Environment with respect to ideal configuration for 6 node cluster across 3 
data centers.

Thanks,
Abhishek Barot

--
This message w/attachments (message) is intended solely for the use of the 
intended recipient(s) and may contain information that is privileged, 
confidential or proprietary.  If you are not an intended recipient, please 
notify the sender, and then please delete and destroy all copies and 
attachments, and be advised that any review or dissemination of, or the taking 
of any action in reliance on, the information contained in or attached to this 
message is prohibited. 
Unless specifically indicated, this message is not an offer to sell or a 
solicitation of any investment products or other financial product or service, 
an official confirmation of any transaction, or an official statement of 
Sender.  Subject to applicable law, Sender may intercept, monitor, review and 
retain e-communications (EC) traveling through its networks/systems and may 
produce any such EC to regulators, law enforcement, in litigation and as 
required by law. 
The laws of the country of each sender/recipient may impact the handling of EC, 
and EC may be archived, supervised and produced in countries other than the 
country in which you are located. This message cannot be guaranteed to be 
secure or free of errors or viruses.  Attachments that are part of this EC may 
have additional important disclosures and disclaimers, which you should read.   
By messaging with Sender you consent to the foregoing.

[jira] [Updated] (KAFKA-4533) subscribe() then poll() on new topic is very slow when subscribed to many topics

[Sergey Alaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Alaev updated KAFKA-4533:

Attachment: sgs.log.tar.gz

org.apache.kafka.* TRACE logs

> subscribe() then poll() on new topic is very slow when subscribed to many 
> topics
> 
>
> Key: KAFKA-4533
> URL: https://issues.apache.org/jira/browse/KAFKA-4533
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Sergey Alaev
> Attachments: sgs.log.tar.gz
>
>
> Given following case:
> consumer.subscribe(my_new_topic, [249 existing topics])
> publisher.send(my_new_topic, key, value)
> poll(10) until data from my_new_topic arrives
> I see data from `my_new_topic` only after approx. 90 seconds.
> If I subscribe only to my_new_topic, I get results within seconds.
> Logs contain lots of lines like this:
> 19:28:07.972 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Resetting offset for 
> partition demo.com_recipient-2-0 to earliest offset.
> 19:28:08.247 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Fetched {timestamp=-1, 
> offset=0} for partition demo.com_recipient-2-0
> Probably you should do that in batch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4533) subscribe() then poll() on new topic is very slow when subscribed to many topics

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15767243#comment-15767243
 ] 

Ismael Juma commented on KAFKA-4533:


Thanks for the report. The requests are done in batches. We'd need more 
information to understand the issue. Maybe you can attach more of the log?

> subscribe() then poll() on new topic is very slow when subscribed to many 
> topics
> 
>
> Key: KAFKA-4533
> URL: https://issues.apache.org/jira/browse/KAFKA-4533
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Sergey Alaev
>
> Given following case:
> consumer.subscribe(my_new_topic, [249 existing topics])
> publisher.send(my_new_topic, key, value)
> poll(10) until data from my_new_topic arrives
> I see data from `my_new_topic` only after approx. 90 seconds.
> If I subscribe only to my_new_topic, I get results within seconds.
> Logs contain lots of lines like this:
> 19:28:07.972 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Resetting offset for 
> partition demo.com_recipient-2-0 to earliest offset.
> 19:28:08.247 [kafka-thread] DEBUG 
> org.apache.kafka.clients.consumer.internals.Fetcher - Fetched {timestamp=-1, 
> offset=0} for partition demo.com_recipient-2-0
> Probably you should do that in batch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Dev list subscribe

[Rajini Sivaram]

Sorry, that was a mail sent by mistake.

On Tue, Dec 13, 2016 at 5:39 PM, Guozhang Wang <wangg...@gmail.com> wrote:

> Rajini,
>
> It's self-service :)
>
> https://kafka.apache.org/contact
>
> Guozhang
>
> On Tue, Dec 13, 2016 at 5:38 AM, Rajini Sivaram <rsiva...@pivotal.io>
> wrote:
>
> > Please subscribe me to the Kafka dev list.
> >
> >
> > Thank you,
> >
> > Rajini
> >
>
>
>
> --
> -- Guozhang
>

Re: Dev list subscribe

[Guozhang Wang]

Rajini,

It's self-service :)

https://kafka.apache.org/contact

Guozhang

On Tue, Dec 13, 2016 at 5:38 AM, Rajini Sivaram <rsiva...@pivotal.io> wrote:

> Please subscribe me to the Kafka dev list.
>
>
> Thank you,
>
> Rajini
>



-- 
-- Guozhang

[jira] [Created] (KAFKA-4533) subscribe() then poll() on new topic is very slow when subscribed to many topics

[Sergey Alaev (JIRA)]

Sergey Alaev created KAFKA-4533:
---

 Summary: subscribe() then poll() on new topic is very slow when 
subscribed to many topics
 Key: KAFKA-4533
 URL: https://issues.apache.org/jira/browse/KAFKA-4533
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.1.0
Reporter: Sergey Alaev


Given following case:

consumer.subscribe(my_new_topic, [249 existing topics])
publisher.send(my_new_topic, key, value)
poll(10) until data from my_new_topic arrives

I see data from `my_new_topic` only after approx. 90 seconds.

If I subscribe only to my_new_topic, I get results within seconds.

Logs contain lots of lines like this:

19:28:07.972 [kafka-thread] DEBUG 
org.apache.kafka.clients.consumer.internals.Fetcher - Resetting offset for 
partition demo.com_recipient-2-0 to earliest offset.
19:28:08.247 [kafka-thread] DEBUG 
org.apache.kafka.clients.consumer.internals.Fetcher - Fetched {timestamp=-1, 
offset=0} for partition demo.com_recipient-2-0

Probably you should do that in batch.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Dev list subscribe

[Rajini Sivaram]

Please subscribe me to the Kafka dev list.


Thank you,

Rajini

[jira] [Commented] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15724068#comment-15724068
 ] 

Vahid Hashemian commented on KAFKA-4471:


[~hachikuji] Thanks for the clarification.
[~salaev] I'll close this JIRA based on [~hachikuji]'s explanation. Please 
advise if you disagree.

> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>Assignee: Vahid Hashemian
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed partitions
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused partitions will unpause.
> We are using 3-node Kafka cluster, server version = client version = 
> 0.10.0.0., one partition per topic
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian resolved KAFKA-4471.

Resolution: Not A Bug

> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>Assignee: Vahid Hashemian
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed partitions
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused partitions will unpause.
> We are using 3-node Kafka cluster, server version = client version = 
> 0.10.0.0., one partition per topic
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710135#comment-15710135
 ] 

Jason Gustafson commented on KAFKA-4471:


[~salaev] This is actually intentional. We do not preserve the pause state 
across rebalances, so you have to repause the partitions in the 
{{onPartitionsAssigned}} callback. Since we don't have a way to preserve the 
pause state when a partition is moved to another consumer, applications 
generally have to add checks to verify the state of the partition after the 
rebalance completes. Otherwise some of the assigned partitions may be paused if 
they had been before, while others are not (if they were assigned for the first 
time). Since applications need to handle this anyway, it seemed simpler to 
start all partitions in a clean state. Similarly, we do not preserve the 
transient (uncommitted) position of the consumer for a partition: after 
rebalancing, each consumer looks up the last committed offset and sets its 
position accordingly. 

> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>Assignee: Vahid Hashemian
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed partitions
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused partitions will unpause.
> We are using 3-node Kafka cluster, server version = client version = 
> 0.10.0.0., one partition per topic
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709355#comment-15709355
 ] 

Vahid Hashemian commented on KAFKA-4471:


[~salaev] Thanks for reporting the issue and providing the steps to reproduce 
it. I'll take a look.

> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>Assignee: Vahid Hashemian
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed partitions
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused partitions will unpause.
> We are using 3-node Kafka cluster, server version = client version = 
> 0.10.0.0., one partition per topic
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian reassigned KAFKA-4471:
--

Assignee: Vahid Hashemian

> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>Assignee: Vahid Hashemian
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed partitions
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused partitions will unpause.
> We are using 3-node Kafka cluster, server version = client version = 
> 0.10.0.0., one partition per topic
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Sergey Alaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Alaev updated KAFKA-4471:

Description: 
How to reproduce:
1. initialize KafkaConsumer and subscribe to some topics by calling `void 
subscribe(Collection topics)`
2. pause some subscribed partitions
3. call `void subscribe(Collection topics)` with one more topic
4. paused partitions will unpause.

We are using 3-node Kafka cluster, server version = client version = 0.10.0.0., 
one partition per topic

Note:
There was another problem with 0.9.x - that client did not purge unsubscribed 
queues if they are paused. Probably that got 'fixed'.

  was:
How to reproduce:
1. initialize KafkaConsumer and subscribe to some topics by calling `void 
subscribe(Collection topics)`
2. pause some subscribed queues
3. call `void subscribe(Collection topics)` with one more topic
4. paused queues will unpause.

We are using 3-node Kafka cluster, server version = client version = 0.10.0.0.

Note:
There was another problem with 0.9.x - that client did not purge unsubscribed 
queues if they are paused. Probably that got 'fixed'.


> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed partitions
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused partitions will unpause.
> We are using 3-node Kafka cluster, server version = client version = 
> 0.10.0.0., one partition per topic
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Sergey Alaev (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708867#comment-15708867
 ] 

Sergey Alaev commented on KAFKA-4471:
-

We are using single KafkaConsumer instance to manage many topics, one partition 
per topic since 0.9.x. Care is taken to work with KafkaConsumer from single 
thread yet there are many bugs we had to work around. For example, call to 
pause() can be ignored in some circumstances. If you want, I can create more 
jira issues.

> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed queues
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused queues will unpause.
> We are using 3-node Kafka cluster, server version = client version = 0.10.0.0.
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4471) KafkaConsumer unpauses partitions after subscribe()

[Sergey Alaev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Alaev updated KAFKA-4471:

Summary: KafkaConsumer unpauses partitions after subscribe()  (was: 
KafkaConsumer unpauses queues after subscribe())

> KafkaConsumer unpauses partitions after subscribe()
> ---
>
> Key: KAFKA-4471
> URL: https://issues.apache.org/jira/browse/KAFKA-4471
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Sergey Alaev
>
> How to reproduce:
> 1. initialize KafkaConsumer and subscribe to some topics by calling `void 
> subscribe(Collection topics)`
> 2. pause some subscribed queues
> 3. call `void subscribe(Collection topics)` with one more topic
> 4. paused queues will unpause.
> We are using 3-node Kafka cluster, server version = client version = 0.10.0.0.
> Note:
> There was another problem with 0.9.x - that client did not purge unsubscribed 
> queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4471) KafkaConsumer unpauses queues after subscribe()

[Sergey Alaev (JIRA)]

Sergey Alaev created KAFKA-4471:
---

 Summary: KafkaConsumer unpauses queues after subscribe()
 Key: KAFKA-4471
 URL: https://issues.apache.org/jira/browse/KAFKA-4471
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.0.0
Reporter: Sergey Alaev


How to reproduce:
1. initialize KafkaConsumer and subscribe to some topics by calling `void 
subscribe(Collection topics)`
2. pause some subscribed queues
3. call `void subscribe(Collection topics)` with one more topic
4. paused queues will unpause.

We are using 3-node Kafka cluster, server version = client version = 0.10.0.0.

Note:
There was another problem with 0.9.x - that client did not purge unsubscribed 
queues if they are paused. Probably that got 'fixed'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Subscribe to Kafka

[Guozhang Wang]

Husayn,

It is a self-service:

https://kafka.apache.org/contact

cheers,

Guozhang

On Fri, Nov 11, 2016 at 10:37 AM, Husayn Campbell 
wrote:

> Newbie developer to kafka.
>



-- 
-- Guozhang

Re: [Subscribe]

[Guozhang Wang]

Hi Ashish,

It's self-service: https://kafka.apache.org/contact


Guozhang


On Fri, Nov 11, 2016 at 4:20 PM, Ashish Singh <ashishle...@gmail.com> wrote:

> Please subscribe me to this group
>



-- 
-- Guozhang

[Subscribe]

[Ashish Singh]

Please subscribe me to this group

Subscribe to Kafka

[Husayn Campbell]

Newbie developer to kafka.

Re: Subscribe me

[Guozhang Wang]

Hi Kiran,

You can follow the instructions here to subscribe:
http://kafka.apache.org/contact.html

Guozhang

On Fri, Sep 9, 2016 at 6:57 AM, Kiran Potladurthi <
kiran.potladur...@gmail.com> wrote:

> Please subscribe me for this mailing list.
>
> Regards
> Kiran
>



-- 
-- Guozhang

Subscribe me

[Kiran Potladurthi]

Please subscribe me for this mailing list.

Regards
Kiran

[jira] [Resolved] (KAFKA-3905) Check for null in KafkaConsumer#{subscribe, assign}

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3905.

   Resolution: Fixed
 Reviewer: Ismael Juma
Fix Version/s: 0.10.1.0

> Check for null in KafkaConsumer#{subscribe, assign}
> ---
>
> Key: KAFKA-3905
> URL: https://issues.apache.org/jira/browse/KAFKA-3905
> Project: Kafka
>  Issue Type: Wish
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Xing Huang
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Currently, KafkaConsumer's subscribe methods accept Collection as 
> topics to be subscribed, but a Collection may have null as its element. For 
> example
> {code}
> String topic = null;
> Collection topics = Arrays.asList(topic);
> consumer.subscribe(topics)
> {code}
> When this happens, consumer will throw a puzzling NullPointerException:
> {code}
>   at org.apache.kafka.common.utils.Utils.utf8Length(Utils.java:245)
>   at org.apache.kafka.common.protocol.types.Type$6.sizeOf(Type.java:248)
>   at 
> org.apache.kafka.common.protocol.types.ArrayOf.sizeOf(ArrayOf.java:85)
>   at org.apache.kafka.common.protocol.types.Schema.sizeOf(Schema.java:89)
>   at org.apache.kafka.common.protocol.types.Struct.sizeOf(Struct.java:244)
>   at 
> org.apache.kafka.common.requests.RequestSend.serialize(RequestSend.java:35)
>   at 
> org.apache.kafka.common.requests.RequestSend.(RequestSend.java:29)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.request(NetworkClient.java:616)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:639)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:552)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:258)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:970)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:934)
> {code}
> Maybe it's better to remove null when doing subscription.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3905) Check for null in KafkaConsumer#{subscribe, assign}

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3905:
---
Summary: Check for null in KafkaConsumer#{subscribe, assign}  (was: remove 
null from subscribed topics  in KafkaConsumer#subscribe)

> Check for null in KafkaConsumer#{subscribe, assign}
> ---
>
> Key: KAFKA-3905
> URL: https://issues.apache.org/jira/browse/KAFKA-3905
> Project: Kafka
>  Issue Type: Wish
>  Components: clients
>Affects Versions: 0.10.0.0
>Reporter: Xing Huang
>Assignee: Rekha Joshi
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> Currently, KafkaConsumer's subscribe methods accept Collection as 
> topics to be subscribed, but a Collection may have null as its element. For 
> example
> {code}
> String topic = null;
> Collection topics = Arrays.asList(topic);
> consumer.subscribe(topics)
> {code}
> When this happens, consumer will throw a puzzling NullPointerException:
> {code}
>   at org.apache.kafka.common.utils.Utils.utf8Length(Utils.java:245)
>   at org.apache.kafka.common.protocol.types.Type$6.sizeOf(Type.java:248)
>   at 
> org.apache.kafka.common.protocol.types.ArrayOf.sizeOf(ArrayOf.java:85)
>   at org.apache.kafka.common.protocol.types.Schema.sizeOf(Schema.java:89)
>   at org.apache.kafka.common.protocol.types.Struct.sizeOf(Struct.java:244)
>   at 
> org.apache.kafka.common.requests.RequestSend.serialize(RequestSend.java:35)
>   at 
> org.apache.kafka.common.requests.RequestSend.(RequestSend.java:29)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.request(NetworkClient.java:616)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:639)
>   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:552)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:258)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:179)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:970)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:934)
> {code}
> Maybe it's better to remove null when doing subscription.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)