Re: Invalid topology building: Processor has no access to StateStore

Hello Florian, In PAPI, you need to associate the state store with the processor node via the Topology#addStateStore(StoreBuilder); note that StateStore#init() is supposed to be triggered by the Streams library only but not by the users: it is called when the library is constructing the specified

[jira] [Created] (KAFKA-6185) java.lang.OutOfMemoryError memory leak on 1.0.0 with 0.11.0.1 on disk and converting to 0.9 clients

Brett Rann created KAFKA-6185: - Summary: java.lang.OutOfMemoryError memory leak on 1.0.0 with 0.11.0.1 on disk and converting to 0.9 clients Key: KAFKA-6185 URL: https://issues.apache.org/jira/browse/KAFKA-6185

[jira] [Created] (KAFKA-6184) report a metric of the lag between the consumer offset and the start offset of the log

Jun Rao created KAFKA-6184: -- Summary: report a metric of the lag between the consumer offset and the start offset of the log Key: KAFKA-6184 URL: https://issues.apache.org/jira/browse/KAFKA-6184 Project:

Re: [ANNOUNCE] New committer: Onur Karaman

Congratulations Onur! On Tue, 7 Nov 2017 at 21:44 Jun Rao wrote: > Affan, > > All known problems in the controller are described in the doc linked from > https://issues.apache.org/jira/browse/KAFKA-5027. > > Thanks, > > Jun > > On Mon, Nov 6, 2017 at 11:00 PM, Affan Syed

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Any other feedback from folks on KIP-211? A prime benefit of this KIP is that it removes the need for the consumer to commit offsets for partitions where the offset hasn't changed. Right now, if the consumer doesn't commit those offsets, they will be deleted, so the consumer keeps blindly

[GitHub] kafka pull request #4190: MINOR: Fix Javadoc Issues

GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/4190 MINOR: Fix Javadoc Issues This PR mainly fixes some broken links and invalid references in the clients Javadoc You can merge this pull request into a Git repository by running: $ git

Re: [VOTE] KIP-201: Rationalising policy interfaces

Okay makes sense thanks! As you said maybe in the future (or now), it's worth starting a server java dependency jar that's not called "client". Probably a debate for another day ( Tom, crossing fingers to see more votes on this! Good stuff On 7/11/17, 9:51 pm, "Ismael Juma"

[GitHub] kafka pull request #4189: KAFKA-6146: minimize the number of triggers enqueu...

GitHub user onurkaraman opened a pull request: https://github.com/apache/kafka/pull/4189 KAFKA-6146: minimize the number of triggers enqueuing PreferredReplicaLeaderElection events We currently enqueue a PreferredReplicaLeaderElection controller event in

Invalid topology building: Processor has no access to StateStore

Hi, Currently it's possible to register a store from a Processor instance via the method StateStore#init(ProcessorContext, StateStore). However, when this method is used the registered store seems to never be add to the the variable ProcessorNode#stateStores which is leading to Invalid topology

[jira] [Resolved] (KAFKA-6110) Warning when running the broker on Windows

[ https://issues.apache.org/jira/browse/KAFKA-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian resolved KAFKA-6110. Resolution: Duplicate Fix Version/s: 1.1.0 > Warning when running the broker on

[GitHub] kafka pull request #1701: MINOR: increase usability of shell-scripts

Github user igalic closed the pull request at: https://github.com/apache/kafka/pull/1701 ---

[jira] [Created] (KAFKA-6183) Broker should send OffsetCommitResponse only after it has written offset to cache

Dong Lin created KAFKA-6183: --- Summary: Broker should send OffsetCommitResponse only after it has written offset to cache Key: KAFKA-6183 URL: https://issues.apache.org/jira/browse/KAFKA-6183 Project: Kafka

[GitHub] kafka pull request #4188: KAFKA-6180: AbstractConfig.getList() should never ...

GitHub user lahabana opened a pull request: https://github.com/apache/kafka/pull/4188 KAFKA-6180: AbstractConfig.getList() should never return null Return an empty list instead of null to make handling these easier You can merge this pull request into a Git repository by running:

Re: [DISCUSS] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

Here is what I'm thinking about this trade-off: we want to fail fast when brokers do not yet support the requested API version, with the cost that we need to do this one-time thing with an expensed NetworkClient plus a bit longer starting up latency. Here are a few different options: 1) we ask

Re: kafka-pr-jdk9-scala2.12 keeps failing

https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/2470/ is green. Thanks Ismael. On Tue, Nov 7, 2017 at 3:46 AM, Ismael Juma wrote: > I changed the Jenkins jobs to use Oracle JDK 9 instead of 9.0.1 until > INFRA-15448 is fixed. > > Ismael > > On Mon, Nov 6, 2017 at 6:25

Re: [ANNOUNCE] New committer: Onur Karaman

Affan, All known problems in the controller are described in the doc linked from https://issues.apache.org/jira/browse/KAFKA-5027. Thanks, Jun On Mon, Nov 6, 2017 at 11:00 PM, Affan Syed wrote: > Congrats Onur, > > Can you also share the document where all known problems are

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats Onur, Can you also share the document where all known problems are listed; I am assuming these bugs are still valid for the current stable release. Affan - Affan On Mon, Nov 6, 2017 at 10:24 PM, Jun Rao wrote: > Hi, everyone, > > The PMC of Apache Kafka is pleased

[jira] [Created] (KAFKA-6182) Automatic co-partitioning of topics via automatic intermediate topic with matching partitions

Antony Stubbs created KAFKA-6182: Summary: Automatic co-partitioning of topics via automatic intermediate topic with matching partitions Key: KAFKA-6182 URL: https://issues.apache.org/jira/browse/KAFKA-6182

[jira] [Created] (KAFKA-6181) Examining log messages with {{--deep-iteration}} should show superset of fields

Yeva Byzek created KAFKA-6181: - Summary: Examining log messages with {{--deep-iteration}} should show superset of fields Key: KAFKA-6181 URL: https://issues.apache.org/jira/browse/KAFKA-6181 Project:

Re: [VOTE] KIP-204 : adding records deletion operation to the new Admin Client API

Because Guozhang and Colin are doing a great job on reviewing the related PR and we are really close to have it in a decent/final shape, what do the other committers think about this KIP ? We are stuck at 2 binding votes (and something like 5 non binding). We need last binding vote for having

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

Hi Jan, Thanks for your comments. I agree that the implementation should not introduce new "bugs" or "known issues" in future. I think we can either i) just drop RecordContext argument for join methods or ii) introduce binary aggregation logic for RecordContexts for two-input-stream-operators.

Jenkins build is back to normal : kafka-trunk-jdk8 #2194

See

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

On 07.11.2017 12:59, Jan Filipiak wrote: On 07.11.2017 11:20, Matthias J. Sax wrote: About implementation if we do the KIP as proposed: I agree with Guozhang that we would need to use the currently processed record's metadata in the context. This does leak some implementation details, but I

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

On 07.11.2017 11:20, Matthias J. Sax wrote: About implementation if we do the KIP as proposed: I agree with Guozhang that we would need to use the currently processed record's metadata in the context. This does leak some implementation details, but I personally don't see a big issue here (at

wiki delete comments permission?

Hi Devs, getting greedy after getting my earlier write access request for the Kafka Wiki granted (https://www.mail-archive.com/dev@kafka.apache.org/msg16718.html), would it be OK if I (mazzag) also had the Wiki "delete comments" right? I'd like to get rid of some of the spam and no longer

Re: kafka-pr-jdk9-scala2.12 keeps failing

I changed the Jenkins jobs to use Oracle JDK 9 instead of 9.0.1 until INFRA-15448 is fixed. Ismael On Mon, Nov 6, 2017 at 6:25 PM, Ismael Juma wrote: > Thanks! > > Ismael > > On Mon, Nov 6, 2017 at 3:48 AM, Ted Yu wrote: > >> Logged

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats Onur ! Well deserved ! Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Azure & IoT Microsoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog :

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats Onur! On Tue, Nov 7, 2017 at 9:17 AM, Tom Bentley wrote: > Well done Onur. > > On 7 November 2017 at 06:52, Jorge Esteban Quilcate Otoya < > quilcate.jo...@gmail.com> wrote: > > > Congratulations Onur!! > > On Tue, 7 Nov 2017 at 06:30, Jaikiran Pai

[jira] [Created] (KAFKA-6180) AbstractConfig.getList() should never return null

Charly Molter created KAFKA-6180: Summary: AbstractConfig.getList() should never return null Key: KAFKA-6180 URL: https://issues.apache.org/jira/browse/KAFKA-6180 Project: Kafka Issue Type:

Build failed in Jenkins: kafka-trunk-jdk7 #2952

See Changes: [ismael] MINOR: Eliminate unnecessary Topic(And)Partition allocations in -- [...truncated 1.40 MB...] org.apache.kafka.common.utils.UtilsTest > testAbs

Re: [VOTE] KIP-201: Rationalising policy interfaces

The idea is that you only depend on a Java jar. The core jar includes the Scala version in the name and you should not care about that when implementing a Java interface. Ismael On Tue, Nov 7, 2017 at 10:37 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Thanks ! > > How about a

Re: [VOTE] KIP-201: Rationalising policy interfaces

Thanks ! How about a java folder package in the core then ? It's not a separate jar and it's still java? Nonetheless I agree these are details. I just got really confused when trying to write my policy and would hope that confusion is not shared by others because it's a "client " class although

Re: [VOTE] KIP-159: Introducing Rich functions to Streams

About implementation if we do the KIP as proposed: I agree with Guozhang that we would need to use the currently processed record's metadata in the context. This does leak some implementation details, but I personally don't see a big issue here (at the same time, I am also fine to remove the

Re: [DISCUSS] KIP-222 - Add "describe consumer group" to KafkaAdminClient

Hi Tom, 1. You're right. I've updated the KIP accordingly. 2. Yes, I have add it to keep consistency, but I'd like to know what others think about this too. Cheers, Jorge. El mar., 7 nov. 2017 a las 9:29, Tom Bentley () escribió: > Hi again Jorge, > > A couple of minor

[GitHub] kafka pull request #4187: MINOR: Handle error metrics removal during shutdow...

GitHub user rajinisivaram opened a pull request: https://github.com/apache/kafka/pull/4187 MINOR: Handle error metrics removal during shutdown You can merge this pull request into a Git repository by running: $ git pull https://github.com/rajinisivaram/kafka

Re: [DISCUSS] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

I would prefer to keep the current check. We could even improve it, and do the check to more than one brokers (even all provided bootstrap.servers) or some random servers after we got all meta data about the cluster. -Matthias On 11/7/17 1:01 AM, Guozhang Wang wrote: > Hello folks, > > One

Re: [VOTE] KIP-201: Rationalising policy interfaces

The location of the policies is fine. Note that the package _does not_ include clients in the name. If we ever have enough server side only code to merit a separate JAR, we can do that and it's mostly compatible (users would only have to update their build dependency). Generally, all public APIs

Jenkins build is back to normal : kafka-trunk-jdk9 #175

See

[GitHub] kafka pull request #4152: MINOR: Eliminate unnecessary Topic(And)Partition a...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4152 ---

Re: [VOTE] KIP-201: Rationalising policy interfaces

Hi Tom, Regarding the java / scala compilation, I believe this is fine (the compiler will know), but any reason why you don't want the policy to be implemented using Scala ? (like the Authorizer) It's usually not best practice to mix in scala / java code. Thanks! Stephane Kind regards, Stephane

Re: [VOTE] KIP-201: Rationalising policy interfaces

Hi Stephane, The vote on this KIP is on-going. I think it would be OK to make minor changes, but Edoardo and Mickael would have to to not disagree with them. The packages have not been brought up as a problem before now. I don't know the reason they're in the client's package, but I agree that

Re: [VOTE] KIP-201: Rationalising policy interfaces

Hi Tom, What's the status of this? I was about to create a KIP to implement a SimpleCreateTopicPolicy (and Alter, etc...) These policies would have some most basic parameter to check for replication factor and min insync replicas (mostly) so that end users can leverage them out of the box. This

Re: [DISCUSS] KIP-222 - Add "describe consumer group" to KafkaAdminClient

Hi again Jorge, A couple of minor points: 1. ConsumerGroupDescription has the member `name`, but everywhere else that I've seen the term "group id" is used, so perhaps calling it "id" or "groupId" would be more consistent. 2. I think you've added ConsumerGroupListing for consistency with

Re: [ANNOUNCE] New committer: Onur Karaman

Well done Onur. On 7 November 2017 at 06:52, Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Congratulations Onur!! > On Tue, 7 Nov 2017 at 06:30, Jaikiran Pai > wrote: > > > Congratulations Onur! > > > > -Jaikiran > > > > > > On 06/11/17 10:54 PM,

[VOTE] KIP-179: Change ReassignPartitionsCommand to use AdminClient

Hi, I would like to start a vote on KIP-179 which would add an AdminClient API for partition reassignment and interbroker replication throttling. https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+ReassignPartitionsCommand+to+use+AdminClient Thanks, Tom