Re: Something like a unique key to prevent same record from being inserted twice?
On 2019/04/02 22:43:31, jim.me...@concept-solutions.com wrote: > > > On 2019/04/02 22:25:16, jim.me...@concept-solutions.com > wrote: > > > > > > On 2019/04/02 21:59:21, Hans Jespersen wrote: > > > yes. Idempotent publish uses a unique messageID to discard potential > > > duplicate messages caused by failure conditions when publishing. > > > > > > -hans > > > > > > > On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com > > > > wrote: > > > > > > > > Does Kafka have something that behaves like a unique key so a producer > > > > can’t write the same value to a topic twice? > > > > > > > Hi Hans, > > > > Is there some documentation or an example with source code where I can > > learn more about this feature and how it is implemented? > > > > Thanks, > > Jim > > > > By the way I tried this... > echo "key1:value1" | ~/kafka/bin/kafka-console-producer.sh --broker-list > localhost:9092 --topic TestTopic --property "parse.key=true" --property > "key.separator=:" --property "enable.idempotence=true" > /dev/null > > And... that didn't seem to do the trick - after running that command multiple > times I did receive key1 value1 for as many times as I had run the prior > command. > > Maybe it is the way I am setting the flags... > Recently I saw that someone did this... > bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test > --producer-property enable.idempotence=true --request-required-acks -1 > Also... the reason for my question is that we are going to have two JMS topics with nearly redundant data in them have the UNION written to Kafka for further processing.
Re: Race condition with stream use of Global KTable
Yes, I forgot to show an item on the topology: +---> global-ktable +-+ | | + v topic-1stream +> topic-3 + ^ | | +> other stream +--> topic-2 ++ My use case is a "schema evolution" of the data in topic-2, to produce topic-3 via "stream". In order to perform this schema evolution, I need to pull some attributes from the payloads in topic-1. I can't simply join topic-1 and topic-2 because they do not share keys. The global-ktable allows me to easily look up the values I need from topic-1 using an attribute from the payload of topic-2, and combine those to write to topic-3. Regards, Raman On Tue, Apr 2, 2019 at 6:56 PM Guozhang Wang wrote: > > Hello Raman, > > It seems from your case that `topic-1` is used for both the global ktable > as well as another stream, which then be transformed to a new stream that > will be "joined" somehow with the global ktable. Could you elaborate your > case a bit more on why do you want to use the same source topic for two > entities in your topology? > > > Guozhang > > On Tue, Apr 2, 2019 at 3:41 PM Raman Gupta wrote: > > > I have a topology like this: > > > >+---> global-ktable +-+ > >| | > >+ v > > topic-1stream > >+ ^ > >| | > >+> other stream +--> topic-2 ++ > > > > IOW, a global ktable is built from topic-1. Meanwhile, "other stream" > > transforms topic-1 to topic-2. Finally, "stream" operators on topic-2, > > and as part of its logic, reads data from "global-ktable". > > > > I am worried about the race condition present in "stream" between the > > message showing up on topic-2, and the "get" from "global-ktable". Is > > there a way, other than retrying the `get`, to avoid this race? > > > > Regards, > > Raman > > > > > -- > -- Guozhang
Re: Race condition with stream use of Global KTable
Hello Raman, It seems from your case that `topic-1` is used for both the global ktable as well as another stream, which then be transformed to a new stream that will be "joined" somehow with the global ktable. Could you elaborate your case a bit more on why do you want to use the same source topic for two entities in your topology? Guozhang On Tue, Apr 2, 2019 at 3:41 PM Raman Gupta wrote: > I have a topology like this: > >+---> global-ktable +-+ >| | >+ v > topic-1stream >+ ^ >| | >+> other stream +--> topic-2 ++ > > IOW, a global ktable is built from topic-1. Meanwhile, "other stream" > transforms topic-1 to topic-2. Finally, "stream" operators on topic-2, > and as part of its logic, reads data from "global-ktable". > > I am worried about the race condition present in "stream" between the > message showing up on topic-2, and the "get" from "global-ktable". Is > there a way, other than retrying the `get`, to avoid this race? > > Regards, > Raman > -- -- Guozhang
Re: Something like a unique key to prevent same record from being inserted twice?
On 2019/04/02 22:25:16, jim.me...@concept-solutions.com wrote: > > > On 2019/04/02 21:59:21, Hans Jespersen wrote: > > yes. Idempotent publish uses a unique messageID to discard potential > > duplicate messages caused by failure conditions when publishing. > > > > -hans > > > > > On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com > > > wrote: > > > > > > Does Kafka have something that behaves like a unique key so a producer > > > can’t write the same value to a topic twice? > > > > Hi Hans, > > Is there some documentation or an example with source code where I can > learn more about this feature and how it is implemented? > > Thanks, > Jim > By the way I tried this... echo "key1:value1" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TestTopic --property "parse.key=true" --property "key.separator=:" --property "enable.idempotence=true" > /dev/null And... that didn't seem to do the trick - after running that command multiple times I did receive key1 value1 for as many times as I had run the prior command. Maybe it is the way I am setting the flags... Recently I saw that someone did this... bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer-property enable.idempotence=true --request-required-acks -1
Race condition with stream use of Global KTable
I have a topology like this: +---> global-ktable +-+ | | + v topic-1stream + ^ | | +> other stream +--> topic-2 ++ IOW, a global ktable is built from topic-1. Meanwhile, "other stream" transforms topic-1 to topic-2. Finally, "stream" operators on topic-2, and as part of its logic, reads data from "global-ktable". I am worried about the race condition present in "stream" between the message showing up on topic-2, and the "get" from "global-ktable". Is there a way, other than retrying the `get`, to avoid this race? Regards, Raman
Re: Something like a unique key to prevent same record from being inserted twice?
On 2019/04/02 21:59:21, Hans Jespersen wrote: > yes. Idempotent publish uses a unique messageID to discard potential > duplicate messages caused by failure conditions when publishing. > > -hans > > > On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com > > wrote: > > > > Does Kafka have something that behaves like a unique key so a producer > > can’t write the same value to a topic twice? > Hi Hans, Is there some documentation or an example with source code where I can learn more about this feature and how it is implemented? Thanks, Jim
Re: Something like a unique key to prevent same record from being inserted twice?
yes. Idempotent publish uses a unique messageID to discard potential duplicate messages caused by failure conditions when publishing. -hans > On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com > wrote: > > Does Kafka have something that behaves like a unique key so a producer can’t > write the same value to a topic twice?
Something like a unique key to prevent same record from being inserted twice?
Does Kafka have something that behaves like a unique key so a producer can’t write the same value to a topic twice?
Re: Strange KafkaConsumer IllegalStateException
Further investigation has uncovered a defect when resolving a hostname fails - https://issues.apache.org/jira/browse/KAFKA-8182 Looks like it has been present since support for resolving all DNS IPs was added. On Mon, 1 Apr 2019 at 15:55, Mark Anderson wrote: > Hi list, > > I've a question regarding a stack trace I see with the 2.2.0 consumer > > java.lang.IllegalStateException: No entry found for connection 0| > at > org.apache.kafka.clients.ClusterConnectionStates.nodeState(ClusterConnectionStates.java:339)| > at > org.apache.kafka.clients.ClusterConnectionStates.disconnected(ClusterConnectionStates.java:143)| > > at > org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:926)| > > at > org.apache.kafka.clients.NetworkClient.access$700(NetworkClient.java:67)| > at > org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1090)| > > at > org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:976)| > > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:533)| > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)| > > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)| > > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1256)| > > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1200)| > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1176) > > From looking at the code in NetworkClient.initiateConnect I'm struggling > to see how the nodeConnectionId could have been removed from the connection > states between connecting call at line 917 and the exception being thrown > at line 926. Can anyone shed any light? > > Thanks, > Mark > > >
Re: leader none, with only one replicat end no ISR
Hello Adrien
Could you give more details about your topic configurations (how many
partitions, what is the replication factor ?)
Usually, partition operations are performed by a broker that has been
assigned as the "controller". You should be able to check the controller
broker id with this command
> echo "get /controller" | ./zookeeper-shell.sh ctl1.edeal.online:2181
Make sure the controller is alive and properly connected to the rest of the
cluster, and maybe take a look in the controller log. Also as suggested by
Harper, give a try to reassign the partitions
Best
On Mon, Apr 1, 2019 at 10:47 PM Harper Henn wrote:
> Hi Adrien,
>
> What happens when you try running the preferred replica leader election
> script? Does this restore leadership for those partitions you listed?
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.PreferredReplicaLeaderElectionTool
>
> Harper
>
> On Sat, Mar 30, 2019 at 12:19 AM Adrien Ruffie wrote:
>
> > Hi all,
> >
> > I would like to know if nobody is able to answer about several questions
> > that I asked for a few months, or if I was just banished from the mailing
> > list ...
> >
> > thank a lot & best regards,
> >
> > Adrien
> >
> > -- Forwarded message -
> > De : Adrien Ruffie
> > Date: jeu. 28 mars 2019 à 12:18
> > Subject: leader none, with only one replicat end no ISR
> > To:
> >
> >
> > Hello all,
> >
> > since yesterday several of my topics have the following description:
> >
> >
> > ./kafka-topics.sh --zookeeper ctl1.edeal.online:2181 --describe | grep -P
> > "none"
> > !2032
> > Topic: edeal_cell_dev Partition: 0 Leader: none Replicas: 5 Isr:
> > Topic: edeal_number_dev Partition: 0 Leader: none Replicas: 8 Isr:
> > Topic: edeal_timeup_dev Partition: 0 Leader: none Replicas: 8 Isr:
> >
> >
> >
> > Without leader, only one replicas and no isr ... I tried to delete it by
> > --delete from kafka-topics.sh,
> > but nothing change.
> > After that I tried to do this:
> >
> >
> https://medium.com/@contactsunny/manually-delete-apache-kafka-topics-424c7e016ff3
> >
> > but any against ... the /brokers/topics/edeal_cell{number/timeup}_dev
> keep
> > always, but without partition ...
> >
> > I'd run out of ideas ...
> > could someone please help me?
> >
> > thank a lot.
> >
> > Adrian
> >
>
