date:20200226

[jira] [Resolved] (KAFKA-9372) Add producer config to make topicExpiry configurable

2020-02-26 Thread Jiao Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiao Zhang resolved KAFKA-9372.
---
Resolution: Duplicate

let me close it as this issue could be covered by 
https://issues.apache.org/jira/browse/KAFKA-8904

> Add producer config to make topicExpiry configurable
> 
>
> Key: KAFKA-9372
> URL: https://issues.apache.org/jira/browse/KAFKA-9372
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 1.1.0
>Reporter: Jiao Zhang
>Assignee: Brian Byrne
>Priority: Minor
>
> Sometimes we got error "org.apache.kafka.common.errors.TimeoutException: 
> Failed to update metadata after 1000 ms" on producer side. We did the 
> investigation and found
>  # our producer produced messages in really low rate, the interval is more 
> than 10 minutes
>  # by default, producer would expire topics after TOPIC_EXPIRY_MS, after 
> topic expired if no data produce before next metadata update (automatically 
> triggered by metadata.max.age.ms) partitions entry for the topic would 
> disappear from the Metadata cache As a result, almost for every time's 
> produce, producer need fetch metadata which could possibly end with timeout.
> To solve this, we propose to add a new config metadata.topic.expiry for 
> producer to make topicExpiry configurable. Topic expiry is good only when 
> producer is long-lived and is used for producing variable counts of topics. 
> But in the case that producers are bounded to single or few fixed topics, 
> there is no need to expire topics at all.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9616) Add new metrics to get total response time with throttle time subtracted

2020-02-26 Thread Jiao Zhang (Jira)

Jiao Zhang created KAFKA-9616:
-

Summary: Add new metrics to get total response time with throttle
time subtracted
Key: KAFKA-9616
URL: https://issues.apache.org/jira/browse/KAFKA-9616
Project: Kafka
Issue Type: Improvement
Components: core
Affects Versions: 1.1.0
Reporter: Jiao Zhang

We are using these RequestMetrics for our cluster monitoring
[https://github.com/apache/kafka/blob/fb5bd9eb7cdfdae8ed1ea8f68e9be5687f610b28/core/src/main/scala/kafka/network/RequestChannel.scala#L364]

and config our AlertManager to fire alerts if 99th value of 'TotalTimeMs'
exceeds the threshold value. This alert is very important as it really notifies
cluster administrators the bad situation for example when one server is bailed
out from cluster or lost leadership.

But we suffer from false alerts sometimes. This is the case. We set quota like
'producer_byte_rate' for some clients, so when requests from these clients are
throttled, 'ThrottleTimeMs' is long and sometimes due to throttle 'TotalTimeMs'
exceeds the threshold value and alert is triggered. As a result we have to put
some time to check details for false alerts either.

So this ticket proposes to add a new metrics 'ProcessTimeMs', the value of
which is total response time with throttle time subtracted. This metrics is
more accurate and could help us only notice the really unexpected situation.

Btw, we tried to achieve this by using PromQL against existing metrics, like
Total - Throttle. But it does not work as it seems these two metrics are
inconsistent in time. So better to expose a new metrics from broker side.

--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] New committer: Konstantine Karantasis

2020-02-26 Thread Satish Duggana

Congrats Konstantine!!

On Thu, Feb 27, 2020 at 12:35 PM Tom Bentley  wrote:
>
> Congratulations!
>
> On Thu, Feb 27, 2020 at 6:43 AM David Jacot  wrote:
>
> > Congrats!
> >
> > Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian 
> > a écrit :
> >
> > > Congratulations Konstantine!
> > >
> > > Regards,
> > > --Vahid
> > >
> > > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen 
> > > wrote:
> > >
> > > > Congrats Konstantine！
> > > >
> > > > On Wed, Feb 26, 2020 at 6:32 PM Manikumar 
> > > > wrote:
> > > >
> > > > > Congrats Konstantine!
> > > >
> > > >
> > > > > On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax 
> > > > wrote:
> > > > >
> > > > > > -BEGIN PGP SIGNED MESSAGE-
> > > > > > Hash: SHA512
> > > > > >
> > > > > > Congrats!
> > > > > >
> > > > > > On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > > > > > > Congrats Konstantine!
> > > > > > >
> > > > > > > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck 
> > > > > > > wrote:
> > > > > > >>
> > > > > > >> Congratulations Konstantine! Well deserved.
> > > > > > >>
> > > > > > >> -Bill
> > > > > > >>
> > > > > > >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> > > > > > >>  wrote:
> > > > > > >>
> > > > > > >>> The PMC for Apache Kafka has invited Konstantine Karantasis as
> > > > > > >>> a committer and we are pleased to announce that he has
> > > > > > >>> accepted!
> > > > > > >>>
> > > > > > >>> Konstantine has contributed 56 patches and helped to review
> > > > > > >>> even more. His recent work includes a major overhaul of the
> > > > > > >>> Connect task management system in order to support incremental
> > > > > > >>> rebalancing. In addition to code contributions, Konstantine
> > > > > > >>> helps the community in many other ways including talks at
> > > > > > >>> meetups and at Kafka Summit and answering questions on
> > > > > > >>> stackoverflow. He consistently shows good judgement in design
> > > > > > >>> and a careful attention to details when it comes to code.
> > > > > > >>>
> > > > > > >>> Thanks for all the contributions and looking forward to more!
> > > > > > >>>
> > > > > > >>> Jason, on behalf of the Apache Kafka PMC
> > > > > > >>>
> > > > > > -BEGIN PGP SIGNATURE-
> > > > > >
> > > > > > iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> > > > > > /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> > > > > > NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> > > > > > XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> > > > > > nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> > > > > > RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> > > > > > mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> > > > > > zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> > > > > > /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> > > > > > 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> > > > > > jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bRy/QXP
> > > > > > 5EZzI9J+ngJG8MYr08UcWQMZt0ytwBVTX/+FVC8Rx5r0D0fqizo=
> > > > > > =68IH
> > > > > > -END PGP SIGNATURE-
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Thanks!
> > > --Vahid
> > >
> >

Re: [ANNOUNCE] New committer: Konstantine Karantasis

2020-02-26 Thread Tom Bentley

Congratulations!

On Thu, Feb 27, 2020 at 6:43 AM David Jacot  wrote:

> Congrats!
>
> Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian 
> a écrit :
>
> > Congratulations Konstantine!
> >
> > Regards,
> > --Vahid
> >
> > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen 
> > wrote:
> >
> > > Congrats Konstantine！
> > >
> > > On Wed, Feb 26, 2020 at 6:32 PM Manikumar 
> > > wrote:
> > >
> > > > Congrats Konstantine!
> > >
> > >
> > > > On Thu, Feb 27, 2020 at 7:46 AM Matthias J. Sax 
> > > wrote:
> > > >
> > > > > -BEGIN PGP SIGNED MESSAGE-
> > > > > Hash: SHA512
> > > > >
> > > > > Congrats!
> > > > >
> > > > > On 2/27/20 2:21 AM, Jeremy Custenborder wrote:
> > > > > > Congrats Konstantine!
> > > > > >
> > > > > > On Wed, Feb 26, 2020 at 2:39 PM Bill Bejeck 
> > > > > > wrote:
> > > > > >>
> > > > > >> Congratulations Konstantine! Well deserved.
> > > > > >>
> > > > > >> -Bill
> > > > > >>
> > > > > >> On Wed, Feb 26, 2020 at 5:37 PM Jason Gustafson
> > > > > >>  wrote:
> > > > > >>
> > > > > >>> The PMC for Apache Kafka has invited Konstantine Karantasis as
> > > > > >>> a committer and we are pleased to announce that he has
> > > > > >>> accepted!
> > > > > >>>
> > > > > >>> Konstantine has contributed 56 patches and helped to review
> > > > > >>> even more. His recent work includes a major overhaul of the
> > > > > >>> Connect task management system in order to support incremental
> > > > > >>> rebalancing. In addition to code contributions, Konstantine
> > > > > >>> helps the community in many other ways including talks at
> > > > > >>> meetups and at Kafka Summit and answering questions on
> > > > > >>> stackoverflow. He consistently shows good judgement in design
> > > > > >>> and a careful attention to details when it comes to code.
> > > > > >>>
> > > > > >>> Thanks for all the contributions and looking forward to more!
> > > > > >>>
> > > > > >>> Jason, on behalf of the Apache Kafka PMC
> > > > > >>>
> > > > > -BEGIN PGP SIGNATURE-
> > > > >
> > > > > iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5XJl0ACgkQO4miYXKq
> > > > > /OjEERAAp08DioD903r9aqGCJ3oHbkUi2ZwEr0yeu1veFD3rGFflhni7sm4J87/Z
> > > > > NArUdpHQ7+99YATtm+HY8gHN+eQeVY1+NV4lX77vPNLEONx09aIxbcSnc4Ih7JnX
> > > > > XxcHHBeR6N9EJkMQ82nOUk5cQaDiIdkOvbOv8+NoOcZC6RMy9PKsFgwd8NTMKL8l
> > > > > nqG8KwnV6WYNOJ05BszFSpTwJBJBg8zhZlRcmAF7iRD0cLzM4BReOmEVCcoas7ar
> > > > > RK38okIAALDRkC9JNYpQ/s0si4V+OwP4igp0MAjM+Y2NVLhC6kK6uqNzfeD21M6U
> > > > > mWm7nE9Tbh/K+8hgZbqfprN6vw6+NfU8dwPD0iaEOfisXwbavCfDeonwSWK0BoHo
> > > > > zEeHRGEx7e2FHWp8KyC6XgfFWmkWJP6tCWiTtCFEScxSTzZUC+cG+a5PF1n6hIHo
> > > > > /CH3Oml2ZGDxoEl1zt8Hs5AgKW8X4PQCsfA4LWqA4GgR6PPFPn6g8mX3/AR3wkyn
> > > > > 8Dmlh3k8ZtsW8wX26IYBywm/yyjbnlSzRVSgAHAbpaIqe5PoMG+5fdc1tnO/2Tuf
> > > > > jd1BjbgAD6u5BksmIGBeZXADblQ/qqfp5Q+WRTJSYLItf8HMNAZoqJFp0bRy/QXP
> > > > > 5EZzI9J+ngJG8MYr08UcWQMZt0ytwBVTX/+FVC8Rx5r0D0fqizo=
> > > > > =68IH
> > > > > -END PGP SIGNATURE-
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Thanks!
> > --Vahid
> >
>

58 matches

Mail list logo