Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

It seems like we all agree at this point (please correct me if wrong!) that we should NOT change the existing repartitioning behavior, ie we should allow Streams to continue to determine when and where to repartition -- *unless* explicitly informed to by the use of a .through or the new

[jira] [Created] (KAFKA-9194) Missing documentation for replicaMaxWaitTimeMs config value

Tomasz Szlek created KAFKA-9194: --- Summary: Missing documentation for replicaMaxWaitTimeMs config value Key: KAFKA-9194 URL: https://issues.apache.org/jira/browse/KAFKA-9194 Project: Kafka

Jenkins build is back to normal : kafka-trunk-jdk8 #4041

See

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Side remark: If the user specifies `repartition()` on both side of the join, we can actually throw the execption earlier, ie, when we build the topology. Current, we can do this check only after Kafka Streams was started, within `StreamPartitionAssignor#assign()` -- we still need to keep this

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

+1 (binding) On 11/14/19 3:48 PM, Guozhang Wang wrote: > +1 (binding), thanks for the KIP! > > Guozhang > > On Fri, Nov 15, 2019 at 4:38 AM Navinder Brar > wrote: > >> Hello all, >> >> I'd like to propose a vote for serving interactive queries during >> Rebalancing, as it is a big deal for

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Thanks. I buy the argument about the lag for active tasks. Nit: The KIP briefly mentions the deprecation of `metadataFoKey()` methods -- those should be listed as `@deprecated` next to the newly added methods to point this change out more visibly. Nit: in the code example, why do we loop over

Re: Potential Bug in 2.3 version (leading to deletion of state directories)

Hi John, Can you please point us to code where Thread-2 will be able to recreate the state directory once cleaner is done ? Also, we see that in https://issues.apache.org/jira/browse/KAFKA-6122, retries around locks is removed. Please let us know why retry mechanism is removed? Also can you

Standby Tasks stays in “created” hash map in AssignedTasks

We are using kakfa streams version 1.1.0 We made some changes to kafka streams code. We are observing following sequence of events in our production environment. We want to understand if following sequence of events is possible in 1.1.0 version also. time T0 StreamThread-1 : got assigned 0_1,

Re: Potential Bug in 2.3 version (leading to deletion of state directories)

Hi John, i see in https://github.com/apache/kafka/pull/3653 there is discussion around swallowing of LockException and retry not being there. but dguy replied saying that "The retry doesn't happen in this block of code. It will happen the next time the runLoop executes." but state of thread is

Re: Potential Bug in 2.3 version (leading to deletion of state directories)

Hi John, i see in https://github.com/apache/kafka/pull/3653 there is discussion around swallowing of LockException and retry not being there. but dguy replied saying that "The retry doesn't happen in this block of code. It will happen the next time the runLoop executes." but state of thread is

Re: Subject: [VOTE] 2.2.2 RC2

+1 (binding) Thanks Randall. Verified signatures and tests. On Fri, Oct 25, 2019 at 7:10 AM Randall Hauch wrote: > > Hello all, we identified around three dozen bug fixes, including an update > of a third party dependency, and wanted to release a patch release for the > Apache Kafka 2.2.0

Re: Potential Bug in 2.3 version (leading to deletion of state directories)

Hi John, Thanks for the response. Yeah, by "marked for deletion" I meant the unlocking of the store(by which in a way it is marked for deletion). From what I have seen the standby task gets stuck in Created state and doesn't move to Running and is not able to recreate the directory. Also, the

[jira] [Resolved] (KAFKA-3096) Leader is not set to -1 when it is shutdown if followers are down

[ https://issues.apache.org/jira/browse/KAFKA-3096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3096. Resolution: Fixed > Leader is not set to -1 when it is shutdown if followers are down >

Build failed in Jenkins: kafka-trunk-jdk11 #956

See Changes: [bbejeck] fix typo in processor-api developer guide docs (#7689) [bbejeck] MINOR: Add method `hasMetrics()` to class `Sensor` (#7692) [ismael] MINOR: Clarify max.request.size and

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

10/20: I think I'm aligned with John's replies as well. Guozhang On Fri, Nov 15, 2019 at 1:45 AM Vinoth Chandar wrote: > >during restoring state the active might have some lag > > Great catch, yes.. we cannot assume lag = 0 for active. Lets report active > lag as well then. If active is too

Re: [VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

+1 (binding), thanks for the KIP! Guozhang On Fri, Nov 15, 2019 at 4:38 AM Navinder Brar wrote: > Hello all, > > I'd like to propose a vote for serving interactive queries during > Rebalancing, as it is a big deal for applications looking for high > availability. With this change, users will

[jira] [Created] (KAFKA-9193) org.apache.kafka.common.utils.Timer should use monotonic clock

Lucas Bradstreet created KAFKA-9193: --- Summary: org.apache.kafka.common.utils.Timer should use monotonic clock Key: KAFKA-9193 URL: https://issues.apache.org/jira/browse/KAFKA-9193 Project: Kafka

Re: Potential Bug in 2.3 version (leading to deletion of state directories)

Hey Navinder, I think what's happening is a little different. Let's see if my worldview also explains your experiences. There is no such thing as "mark for deletion". When a thread loses a task, it simply releases its lock on the directory. If no one else on the instance claims that lock within

Build failed in Jenkins: kafka-trunk-jdk8 #4040

See Changes: [manikumar] MINOR: fix flaky ConsumerBounceTest.testClose [manikumar] KAFKA-9046: Use top-level worker configs for connector admin clients -- [...truncated

Potential Bug in 2.3 version (leading to deletion of state directories)

Hi, We are facing a peculiar situation in the 2.3 version of Kafka Streams. First of all, I want to clarify if it is possible that a Stream Thread (say Stream Thread-1) which had got an assignment for a standby task (say 0_0) can change to Stream Thread-2 on the same host post rebalancing. The

[jira] [Created] (KAFKA-9192) NullPointerException if field in schema not present in value

Mark Tinsley created KAFKA-9192: --- Summary: NullPointerException if field in schema not present in value Key: KAFKA-9192 URL: https://issues.apache.org/jira/browse/KAFKA-9192 Project: Kafka

[VOTE] KIP-535: Allow state stores to serve stale reads during rebalance

Hello all, I'd like to propose a vote for serving interactive queries during Rebalancing, as it is a big deal for applications looking for high availability. With this change, users will have control over the tradeoff between consistency and availability during serving. The full KIP is

[jira] [Created] (KAFKA-9191) Kafka throughput suffers substantially when scaling topic partitions with small messages

Chris Pettitt created KAFKA-9191: Summary: Kafka throughput suffers substantially when scaling topic partitions with small messages Key: KAFKA-9191 URL: https://issues.apache.org/jira/browse/KAFKA-9191

Build failed in Jenkins: kafka-trunk-jdk11 #955

See Changes: [manikumar] MINOR: fix flaky ConsumerBounceTest.testClose [manikumar] KAFKA-9046: Use top-level worker configs for connector admin clients -- [...truncated

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

Hi Manikumar, It looks like the header for KIP-440 is accurate ("KIP-440: Extend Connect Converter to support headers") but the content appears to correspond to KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") instead. Could we double-check and make sure that the summary for

Preliminary blog post about the Apache Kafka 2.4.0 release

Hi all, I've prepared a preliminary blog post about the upcoming Apache Kafka 2.4.0 release. Please take a look and let me know if you want to add/modify details. Thanks to all who contributed to this blog post. https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1 Thanks,

[VOTE] 2.4.0 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 2.4.0. There is work in progress for couple blockers PRs. I am publishing RC0 to avoid further delays in testing the release. This release includes many new features, including: - Allow

[jira] [Resolved] (KAFKA-9044) Brokers occasionally (randomly?) dropping out of clusters

[ https://issues.apache.org/jira/browse/KAFKA-9044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bukowinski resolved KAFKA-9044. - Resolution: Not A Bug Issue traced to external cause. > Brokers occasionally

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

>during restoring state the active might have some lag Great catch, yes.. we cannot assume lag = 0 for active. Lets report active lag as well then. If active is too laggy, the app can then deem the store partition unavailable (based on what the application is willing to tolerate). @matthias do

[jira] [Created] (KAFKA-9190) Server leaves connections with expired authentication sessions open

Jason Gustafson created KAFKA-9190: -- Summary: Server leaves connections with expired authentication sessions open Key: KAFKA-9190 URL: https://issues.apache.org/jira/browse/KAFKA-9190 Project:

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

I agree with Vinoth and John on having "allLocalStoreOffsetLags()", all actives don't have 0 lag, as during restoring state the active might have some lag and one of the features of this KIP is to provide an option to query from active (which might be in restoring state).  I will update the KIP

[jira] [Created] (KAFKA-9189) Shutdown is blocked if connection to Zookeeper is lost

Boris Granveaud created KAFKA-9189: -- Summary: Shutdown is blocked if connection to Zookeeper is lost Key: KAFKA-9189 URL: https://issues.apache.org/jira/browse/KAFKA-9189 Project: Kafka

[jira] [Created] (KAFKA-9188) Flaky Test SslAdminClientIntegrationTest.testSynchronousAuthorizerAclUpdatesBlockRequestThreads

Bill Bejeck created KAFKA-9188: -- Summary: Flaky Test SslAdminClientIntegrationTest.testSynchronousAuthorizerAclUpdatesBlockRequestThreads Key: KAFKA-9188 URL: https://issues.apache.org/jira/browse/KAFKA-9188

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks, all, I can get behind just totally leaving out reparation-via-groupBy. If we only introduce `repartition()` for now, we're making the minimal change to gain the desired capability. Plus, since we agree that `repartition()` should never be optimizable, it's a future-compatible proposal.

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Hi all, Thanks for the "reset", Vinoth. It brings some needed clarity to the discussion. 10. My 2 cents: we might as well include the lags for the active copies as well. This is a more context-free API. If we only include standbys, this choice won't make sense to users unless they understand

Build failed in Jenkins: kafka-2.4-jdk8 #74

See Changes: [manikumar] MINOR: fix flaky ConsumerBounceTest.testClose [manikumar] KAFKA-9046: Use top-level worker configs for connector admin clients -- [...truncated

[jira] [Created] (KAFKA-9187) kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl

Bill Bejeck created KAFKA-9187: -- Summary: kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl Key: KAFKA-9187 URL: https://issues.apache.org/jira/browse/KAFKA-9187

Build failed in Jenkins: kafka-trunk-jdk8 #4039

See Changes: [github] KAFKA-9169: fix standby checkpoint initialization (#7681) -- [...truncated 5.56 MB...] org.apache.kafka.streams.scala.kstream.MaterializedTest >

Jenkins build is back to normal : kafka-2.3-jdk8 #135

See

[jira] [Created] (KAFKA-9186) Kafka Connect floods logs with probably bogus error messages from DelegatingClassLoader

Piotr Szczepanik created KAFKA-9186: --- Summary: Kafka Connect floods logs with probably bogus error messages from DelegatingClassLoader Key: KAFKA-9186 URL: https://issues.apache.org/jira/browse/KAFKA-9186

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

10. I considered that. Had to pick one or the other. Can just return standby too and rename method to may be “allLocalStandbyOffsetLags()” to have it explicit. (Standby should implicitly convey that we are talking about stores) 20. What I meant was, we are returning HostInfo instead of

[jira] [Resolved] (KAFKA-9046) Connect worker configs require undocumented 'admin.' prefix to configure DLQ for connectors

[ https://issues.apache.org/jira/browse/KAFKA-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9046. -- Fix Version/s: 2.4.0 2.3.2 Resolution: Fixed Issue resolved by pull request

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi Randall and Konstantine, Thanks for the explanation. Since it block users upgrading from older Connect versions to 2.4 release, we will consider the JIRA as blocker for 2.4.0. Thanks On Wed, Nov 13, 2019 at 11:58 PM Konstantine Karantasis < konstant...@confluent.io> wrote: > Hi Manikumar,