Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Hi Kafka Team, I'd like to bring this thread back at the top of the email stack to get a chance to see this KIP merge in the next minor/major release. Thanks. Le ven. 18 janv. 2019 à 01:20, Florian Hussonnois a écrit : > Hey folks, > > This KIP has start since a while but has n

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

> >>>>>> the > > >>>>>>>>> operation? It doesn't really work. > > >>>>>>>>> > > >>>>>>>>> Adding config objects to the DSL was an attempt to avoid > overload > > >>>

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

`Named`, which would be used in our > >>> adding overload functions. The main reason is to have consistency in > >>> naming." And I think I'm on the same page with John with his more > >> detailed > >>> proposal. > >>> > >>> 3. As

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

(the PR has been rebased) Thanks, Le ven. 22 sept. 2017 à 09:36, Florian Hussonnois a écrit : > Hi team, > > Are there any more votes ? Thanks > > Le 12 sept. 2017 20:18, "Gwen Shapira" a écrit : > >> Thanks for clarifying. >> >> +1 again :) >> &

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

;>> | > > > >>> > > > >> > > > > > > - > > > >>> Repartition Topic | X | |

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

and > > >> processor name independent from each other, it's still unclear to me, > > >> why we cannot use the `Materialized` parameter to specify the > processor > > >> name: > > >> > > >>> // only set the node name > >

[VOTE] KIP-307: Allow to define custom processor names with KStreams DSL

Hi folks, I would like to initiate a vote for the following KIP : https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL Note, there is still some minor discussions regarding the implementation. Thanks -- Florian HUSSONNOIS

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

e `Materialized` to name the processor for this > case, too. Can you elaborate on the motivation? > > > -Matthias > > On 1/11/19 3:39 PM, Florian Hussonnois wrote: > > Hi Guozhang, > > > > I have updated the PR as well as the KIP. I should add more unit tests to > &g

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

ould be not-often >>> used functions for them anyways. And by letting existing control classes to >>> extend Named, we can have a unified method name for static constructor as >>> well. >>> >>> >>> >>> Guozhang >>> >>

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

> > > > About the inconsistency in method naming. I agree, that `as` is very > > generic and maybe not the best choice. > > > > I think it might be helpful, to have a table overview in the KIP, that > > list all existing static/non-static methods that allow to

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

ks! > > > > I hope you find this helpful, rather than frustrating. I'm sorry I didn't > > get a chance to comment sooner. > > > > Thanks for the KIP, I think it will be much nicer to be able to name the > > processor nodes. > > > > -John > > > > On Tue, Nov 27, 2018 at 6:34 PM G

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

store names. > > * For 3, I'd vote for adding overloaded functions with Named. > > * For 4, if users really want to name the processor she can call > > aggregate() instead, so I think it is okay to skip this case. > > > > > > Guozhang > > > > &g

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

ey, value); > > } > > > > @Override > > public String name() { > > return name; > > } > > }; > > } > > } > > > > Then, here's how it would look to use i

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

sor names: > > this will in turn change the internal state store names, as well as > > internal topic names as well, making the new application topology to be > > incompatible with the ones. One rationale I had about this KIP is that > > aligned this effort, moving forward we can

Re: [DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

ined. > > > > > > But instead of adding the "Processed" parameter to a large percentage > of > > > the methods, which would result in overloaded methods (which we removed > > > quite a bit with KIP-182) what do you think of adding a method > >

[DISCUSS] KIP-307: Allow to define custom processor names with KStreams DSL

Hi, I would like to start a new discussion on following KIP : https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL This is still a draft. Looking forward for your feedback. -- Florian HUSSONNOIS

[jira] [Created] (KAFKA-6958) Allow to define custom processor names with KStreams DSL

Florian Hussonnois created KAFKA-6958: - Summary: Allow to define custom processor names with KStreams DSL Key: KAFKA-6958 URL: https://issues.apache.org/jira/browse/KAFKA-6958 Project: Kafka

[jira] [Created] (KAFKA-6957) Add getter to AbstractStream class to make internalTopologyBuilder accessible outside of package

Florian Hussonnois created KAFKA-6957: - Summary: Add getter to AbstractStream class to make internalTopologyBuilder accessible outside of package Key: KAFKA-6957 URL: https://issues.apache.org/jira/browse

Invalid topology building: Processor has no access to StateStore

building exception when accessing store with the method ProcessorContext#getStateStore Is it the desired behavior or a bug ? -- Florian HUSSONNOIS

Re: Before creating KIP : Kafka Connect / Add a configuration provider class

ker), > > correct? > > > > Would all provider implementations be custom implementations, or are > there > > some provider implementations that are general enough for Connect to > > include them? > > > > Best regards, > > > > Randall > > > > On F

Before creating KIP : Kafka Connect / Add a configuration provider class

for all deployed connectors, to provide default configs or to provide sensitive configs like user/password. I've met these requirements on different projects. Do you think, this feature merits a new KIP ? Thanks, -- Florian HUSSONNOIS

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

ld only be read during startup or reconfiguration of a task." > > > is incorrect? because the API extension allows reading offset data at > any > > > point in the lifecycle, right? > > > > > > On Fri, Sep 8, 2017 at 5:18 AM Florian Hussonnois < > fh

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

Hi Shapira, We only expose the OffsetStorageReader to connector which relies on KafkaOffsetBackingStore. The store continuesly consumes offsets from kafka so I think we can't have stale data. Le 8 sept. 2017 06:13, "Randall Hauch" a écrit : > The KIP and PR expose the

Re: [DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

that now we have a `SourceConnectorContext` and `SinkConnectorContext` to which we can add more methods if needed, and they are very similar to `SourceTaskContext` and `SinkTaskContext`. Thoughts? On Wed, Apr 5, 2017 at 3:59 PM, Florian Hussonnois <fhussonn...@gmail.com> wrote: > Hi All, > >

Kafka Connect suggestion before creating new KIP

a configuration management server (for example Consul). Do you think this two minor suggestions can deserved KIPs ? Thanks very much. Florian. -- Florian HUSSONNOIS

Re: [VOTE] KIP 130: Expose states of active tasks to KafkaStreams public API

, 18 May 2017 at 23:46 Matthias J. Sax <matth...@confluent.io> > >> wrote: > >> > >>> +1 > >>> > >>> On 5/18/17 8:26 AM, Bill Bejeck wrote: > >>>> +1 > >>>> > >>>> On Thu, May 18, 2017 at 6:54 AM

Re: [VOTE] KIP 130: Expose states of active tasks to KafkaStreams public API

m> > wrote: > > > > > +1 (non-binding) > > > > > > Thanks > > > Eno > > > > > > > On 21 Apr 2017, at 05:58, Guozhang Wang <wangg...@gmail.com> wrote: > > > > > > > > +1. Thanks a lot for the KIP! >

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

gt;>> subtopology to a task. > >>> > >>> IMHO, adding a simple variable to `Subtopoloy` that provide the id > >>> should be sufficient. We can simply document in the JavaDocs how > >>> Subtopology and TaskMetadata can be linked to each other.

Re: [DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

Hi All, Is there any feedback regarding that KIP ? https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector Thanks, 2017-03-14 22:51 GMT+01:00 Florian Hussonnois <fhussonn...@gmail.com>: > Hi Matthias, > > Sorry I didn't

[VOTE] KIP 130: Expose states of active tasks to KafkaStreams public API

Hi All, I would like to start the vote for the KIP-130 : https://cwiki.apache.org/confluence/display/KAFKA/KIP+130%3A+Expose+states+of+active+tasks+to+KafkaStreams+public+API Thanks, -- Florian HUSSONNOIS

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

the new class ThreadMetadata. Thanks, 2017-03-27 23:40 GMT+02:00 Florian Hussonnois <fhussonn...@gmail.com>: > Hi Guozhang, Matthias, > > It's a great idea to add sub topologies descriptions. This would help > developers to better understand topology concept. > > I agree

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

"TopologyDescription" introduced in KIP-120, it will simply describe the >> > whole topology possibly composed of multiple sub-topologies. So it is >> > hard for users to tell which sub-topology is executed under which task >> > on-the-fly. >> > >>

[jira] [Commented] (KAFKA-4905) StreamPartitionAssignor doesn't respect subscriptions to assign partitions.

[ https://issues.apache.org/jira/browse/KAFKA-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927100#comment-15927100 ] Florian Hussonnois commented on KAFKA-4905: --- Hi [~mjsax], I'm sorry if my explanation

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

those > metrics mentioned in my previous email would suffice even for the use case > that you have mentioned. > > > Guozhang > > On Wed, Mar 8, 2017 at 3:18 PM, Florian Hussonnois <fhussonn...@gmail.com> > wrote: > > > Hi Guozhang > > > &

[jira] [Updated] (KAFKA-4905) StreamPartitionAssignor doesn't respect subscriptions to assign partitions.

[ https://issues.apache.org/jira/browse/KAFKA-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florian Hussonnois updated KAFKA-4905: -- Summary: StreamPartitionAssignor doesn't respect subscriptions to assign partitions

[jira] [Created] (KAFKA-4905) StreamPartitionAssignor doesn't respect subscriptions to asisgn partitions.

Florian Hussonnois created KAFKA-4905: - Summary: StreamPartitionAssignor doesn't respect subscriptions to asisgn partitions. Key: KAFKA-4905 URL: https://issues.apache.org/jira/browse/KAFKA-4905

Re: [DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

rovement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion > > Thanks, > > Matthias > > > On 3/7/17 1:24 PM, Florian Hussonnois wrote: > > Hi all, > > > > I've created a new KIP to add access to OffsetStorageReader from > > SourceConnector > > > >

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

t; state stores and partition assignment for every instance of the > > KafkaStreams application. I'm wondering if that is good enough for what > you > > are trying to achieve? If not could it be modified to include the per > > Thread assignment? > > > > Thanks, > > Damian

[DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

Hi all, I've created a new KIP to add access to OffsetStorageReader from SourceConnector https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector Thanks. -- Florian HUSSONNOIS

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

o be > redundant. It should be the same as > TaskState#consumedOffsetsByPartition.keySet() > > Or do I miss something? > > > -Matthias > > On 3/1/17 5:19 AM, Florian Hussonnois wrote: > > Hi Eno, > > > > Yes, but the state() method only returns th

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

:46 GMT+01:00 Eno Thereska <eno.there...@gmail.com>: > Thanks Florian, > > Have you had a chance to look at the new state methods in 0.10.2, e.g., > KafkaStreams.state()? > > Thanks > Eno > > On 1 Mar 2017, at 11:54, Florian Hussonnois <fhussonn...@gmail.com>

[DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

Hi all, I have just created KIP-130 to add a new method to the KafkaStreams API in order to expose the states of threads and active tasks. https://cwiki.apache.org/confluence/display/KAFKA/KIP+130%3A+Expose+states+of+active+tasks+to+KafkaStreams+public+API Thanks, -- Florian HUSSONNOIS

[jira] [Created] (KAFKA-4819) Expose states of active tasks to public API

Florian Hussonnois created KAFKA-4819: - Summary: Expose states of active tasks to public API Key: KAFKA-4819 URL: https://issues.apache.org/jira/browse/KAFKA-4819 Project: Kafka Issue

[jira] [Updated] (KAFKA-4794) Add access to OffsetStorageReader from SourceConnector

[ https://issues.apache.org/jira/browse/KAFKA-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florian Hussonnois updated KAFKA-4794: -- Affects Version/s: 0.10.2.0 > Add access to OffsetStorageReader from SourceConnec

[jira] [Created] (KAFKA-4794) Add access to OffsetStorageReader from SourceConnector

Florian Hussonnois created KAFKA-4794: - Summary: Add access to OffsetStorageReader from SourceConnector Key: KAFKA-4794 URL: https://issues.apache.org/jira/browse/KAFKA-4794 Project: Kafka

Re: Kafka Connect / Access to OffsetStorageReader from SourceConnector

ing to use offsets > to be able to tell when a task has finished a file? > > Thanks, > Jason > > On Fri, Feb 17, 2017 at 4:45 AM, Florian Hussonnois <fhussonn...@gmail.com > > > wrote: > > > Hi Kafka Team, > > > > I'm developping a connector whic

Kafka Connect / Access to OffsetStorageReader from SourceConnector

contribute to it. Thanks, -- Florian HUSSONNOIS

Re: KStreams / add support for sink processor with dynamic topics

> > As we got this request multiple times already, it might be worth > > adding it IMHO. Not sure what the opinion of other is? We should make > > sure that the feature gets accepted before you put a lot of effort in > > it. :) > > > > > > - -Matthias >

KStreams / add support for sink processor with dynamic topics

. If you think that feature would be usefull I can create a jira and contribute to it. Also, do I need to create a new KIP as this requires changes on a public API ? Thanks, -- Florian HUSSONNOIS

Re: PartitionAssignor / Sort members per subscription time before assigning partitions

Kafka Streams is a little more clever in > how partitions are assigned. It uses a custom assignor which takes into > account the consumer's host information. > > Thanks, > Jason > > On Thu, Sep 1, 2016 at 9:00 AM, Florian Hussonnois <fhussonn...@gmail.com> > wrote: &g

Re: Question regarding Producer and Duplicates

e whole batch is accepted but the ack was > not delivered in time, and hence it was re-tried. > > > Guozhang > > > On Tue, Aug 30, 2016 at 2:45 AM, Florian Hussonnois <fhussonn...@gmail.com > > > wrote: > > > Hi all, > > > > I am using kafka_2.11-0.10.0.1, my

[jira] [Commented] (KAFKA-3340) Add support for rebalance and adding concurrently records with MockConsumer

[ https://issues.apache.org/jira/browse/KAFKA-3340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459658#comment-15459658 ] Florian Hussonnois commented on KAFKA-3340: --- I have updated a pull request for this JIRA

PartitionAssignor / Sort members per subscription time before assigning partitions

Hi Kafka Team, I would like to have your opinion before creating a new JIRA. I'm working with the Java Consumer API. The current partition assignors use the consumer ids to sort members before assigning partitions. This works pretty well as long as all consumers are started into the same JVM

[jira] [Commented] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

[ https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454817#comment-15454817 ] Florian Hussonnois commented on KAFKA-4106: --- Thank you very much, sorry to have created

[jira] [Created] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

Florian Hussonnois created KAFKA-4106: - Summary: Consumer / add configure method to PartitionAssignor interface Key: KAFKA-4106 URL: https://issues.apache.org/jira/browse/KAFKA-4106 Project

Question regarding Producer and Duplicates

Hi all, I am using kafka_2.11-0.10.0.1, my understanding is that the producer API batches records per partition to send efficient requests. We can configure batch.size to increase the throughtput. However, in case of failure all records within the batch failed ? If that is true, does that mean

[jira] [Created] (KAFKA-3922) Add a copy-constructor to AbstractStream

Florian Hussonnois created KAFKA-3922: - Summary: Add a copy-constructor to AbstractStream Key: KAFKA-3922 URL: https://issues.apache.org/jira/browse/KAFKA-3922 Project: Kafka Issue Type

Complexe Event Processing on top of KafkaStreams

Stream DSL. Also, I would like to know if you will plan to add an "external" module in order to add contributions without impacting the kafkastreams APIs? Please feel free to give me your feedback about my API. Thanks for you time and the amazing work you are doing on Kafka. Florian. --

[jira] [Commented] (KAFKA-3737) Closing connection during produce request should be log with WARN level.

[ https://issues.apache.org/jira/browse/KAFKA-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15329482#comment-15329482 ] Florian Hussonnois commented on KAFKA-3737: --- Hi Gwen, My concern is more about

[jira] [Created] (KAFKA-3737) Closing connection during produce request should be log with WARN level.

Florian Hussonnois created KAFKA-3737: - Summary: Closing connection during produce request should be log with WARN level. Key: KAFKA-3737 URL: https://issues.apache.org/jira/browse/KAFKA-3737

[DISCUSS] KIP-53 Add custom policies for reconnect attempts to NetworkdClient

an implementation : https://github.com/apache/kafka/pull/1179 Thanks -- Florian HUSSONNOIS

[jira] [Updated] (KAFKA-3496) Add reconnect attemps policies for client

[ https://issues.apache.org/jira/browse/KAFKA-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Florian Hussonnois updated KAFKA-3496: -- Summary: Add reconnect attemps policies for client (was: Add policies to reconnection

[jira] [Created] (KAFKA-3496) Add policies to reconnection

Florian Hussonnois created KAFKA-3496: - Summary: Add policies to reconnection Key: KAFKA-3496 URL: https://issues.apache.org/jira/browse/KAFKA-3496 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-3340) Add support for rebalance and adding concurrently records with MockConsumer

Florian Hussonnois created KAFKA-3340: - Summary: Add support for rebalance and adding concurrently records with MockConsumer Key: KAFKA-3340 URL: https://issues.apache.org/jira/browse/KAFKA-3340

[jira] [Created] (KAFKA-2998) New Consumer should not retry indefinitely if no broker is available

Florian Hussonnois created KAFKA-2998: - Summary: New Consumer should not retry indefinitely if no broker is available Key: KAFKA-2998 URL: https://issues.apache.org/jira/browse/KAFKA-2998 Project