(all of them as they act as a cluster) and aggregate all the data
to see the full flow of messages in the system. Thats why the logs may seem
overwelming and you need to look at the logs of all the broker (and perhaps all
the clients as well) to get the full picture.
-hans
> On Mar 28, 2
RAID 5 typically is slower because Kafka is very write heavy load and that
creates a bottleneck because writes to any disk require parity writes on the
other disks.
-hans
> On Mar 28, 2020, at 2:55 PM, Vishal Santoshi
> wrote:
>
> Ny one ? We doing a series of tests to
Very good description with pictures in the book Kafka: The Definitive Guide
https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html
-hans
> On Mar 26, 2020, at 12:00 PM, sunil chaudhari
> wrote:
>
> Again
> A consumer can have one or more
Yes it should be going much faster than that. Something is wrong in your setup.
-hans
> On Mar 26, 2020, at 5:58 PM, Vidhya Sakar wrote:
>
> Hi Team,
>
> The Kafka consumer is reading only 8 records per second.We have implemented
> apache Kafka and confluent connect S3. The
sumers
in a consumer group, each consumer in the group would consume from 3 partitions.
-hans
This is a great blog post that explains how kafka works with advertised
listeners and docker
https://rmoff.net/2018/08/02/kafka-listeners-explained/
-hans
> On Oct 18, 2019, at 5:36 AM, Mich Talebzadeh
> wrote:
>
> I do not understand this.
>
> You have on a phy
Gwen Shapira published a great whitepaper with Reference Architectures for
all Kafka and Confluent components in big and small environements and for
bare metal, VMs, and all 3 major public clouds.
https://www.confluent.io/resources/apache-kafka-confluent-enterprise-reference-architecture/
On
Take a look at the Admin Client API here
https://kafka.apache.org/22/javadoc/index.html?org/apache/kafka/clients/admin/AdminClient.html
-hans
On Mon, Jun 17, 2019 at 4:27 PM shubhmeet kaur
wrote:
> hi,
>
> I wish to updater the replciation factor of already created topic through
&g
of messages.
I would recommend you not use auto commit at all and instead manually commit
offsets immediately after sending each email or batch of emails.
-hans
> On May 24, 2019, at 4:35 AM, ASHOK MACHERLA wrote:
>
> Dear Team
>
>
>
> First of all thanks fo
and balance them ahead of time.
-hans
On Wed, May 15, 2019 at 8:45 AM M. Manna wrote:
> Hello,
>
> I am trying to do some performance testing using Kafka-Consumer-Perf-Test.
> Could somone please help me understand whether my setup is correct?
>
> 1) I would like to ru
Can you just use kafka-console-consumer and just redirect the output into a
file?
-hans
On Mon, May 13, 2019 at 1:55 PM Vinay Jain wrote:
> Hi
>
> The data needs to be transferred to some other system in other network, and
> due to some security reasons, the other systems canno
Your connector sounds a lot like this one
https://github.com/jcustenborder/kafka-connect-spooldir
I do not think you can run such a connector in distributed mode though.
Typically something like this runs in standalone mode to avoid conflicts.
-hans
On Wed, Apr 24, 2019 at 1:08 AM Venkata S
rocksdb state store that comes with Kafka Streams (or as a
UDF in KSQL).
You can alternatively write your consuming apps to implement similar message
pruning functionality themselves and avoid one extra component in the end to
end architecture
-hans
> On Apr 2, 2019, at 7:28 PM, jim
yes. Idempotent publish uses a unique messageID to discard potential duplicate
messages caused by failure conditions when publishing.
-hans
> On Apr 1, 2019, at 9:49 PM, jim.me...@concept-solutions.com
> wrote:
>
> Does Kafka have something that behaves like a unique key s
://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions
“As a rule of thumb, we recommend each broker to have up to 4,000 partitions
and each cluster to have up to 200,000 partitions”
-hans
> On Apr 1, 2019, at 2:02 AM, Alexander Kuterin wrote:
>
> Thanks, Hans!
> We
Doesn’t every one of the 20,000 POS terminals want to get the same price list
messages? If so then there is no need for 20,000 partitions.
-hans
> On Mar 31, 2019, at 7:24 PM, wrote:
>
> Hello!
>
>
>
> I ask for your help in connection with the my recent tas
Thats a 4.5 year old benchmark and it was run with a single broker node and
only 1 producer and 1 consumer all running on a single MacBookPro. Definitely
not the target production environment for Kafka.
-hans
> On Mar 21, 2019, at 11:43 AM, M. Manna wrote:
>
> HI All,
>
>
client and the
brokers.
-hans
> On Mar 19, 2019, at 8:19 AM, James Grant wrote:
>
> Hello,
>
> We would like to expose a Kafka cluster running on one network to clients
> that are running on other networks without having to have full routing
> between the two networks.
as if it were published in realtime.
-hans
> On Mar 15, 2019, at 7:52 AM, Pulkit Manchanda wrote:
>
> Hi All,
>
> I am building a data pipeline to send logs from one data source to the
> other node.
> I am using Kafka Connect standalone for this integration.
> Everything wo
-processing-cookbook/
There is even an example for repartitioning topics using the PARTITIONS
parameter.
CREATE STREAM clickstream_new WITH (PARTITIONS=5) AS SELECT * from
clickstream_raw;
-hans
> On Jan 27, 2019, at 9:24 AM, Ryanne Dolan wrote:
>
> You can use MirrorMaker to copy data betwe
this one.
-hans
> On Jan 21, 2019, at 10:02 AM, Rahul Singh
> wrote:
>
> I am using node-kafka, I have used consumer.commit to commit offsets but
> don't know why when I restart the consumer it consume the committed offsets.
>
> Thanks
>
>> On Mon, Jan 21,
/README.md#consumer
https://github.com/Blizzard/node-rdkafka/blob/master/README.md
-hans
> On Jan 21, 2019, at 5:17 AM, Rahul Singh
> wrote:
>
> I am using in Node with node-kafka module.
>
>> On Mon, Jan 21, 2019 at 6:45 PM M. Manna wrote:
>>
>> Please
for performance but the send() teturns a future so
you can make it appear to be a synchrounous publish easily. Examples are in the
javadoc.
-hans
> On Jul 18, 2018, at 7:45 AM, jingguo yao wrote:
>
> The asynchronous sending of a message returns no error even if the
> Kafka server is
Kafka ACLs are at the topic level, not partition level.
Probably better to make 10 topics of 1 partition each and use topic ACLs to
control access.
-hans
> On Jun 25, 2018, at 9:50 PM, Yash Ganthe wrote:
>
> Hi,
>
> If I have a topic with 10 partitions, I would like
You should just recommit the same offsets sooner than every 24 hours (or
whatever your commit topic retention period is set to). The expiry of offsets
is based on the timestamp of the commits.
-hans
> On Jun 1, 2018, at 1:03 AM, Dinesh Subramanian
> wrote:
>
> Hi,
>
>
Why don’t to just put the metadata in the header and leave the key null so it
defaults to round robin?
-hans
> On May 31, 2018, at 6:54 AM, M. Manna wrote:
>
> Hello,
>
> I can see the this has been set as "KIP required".
>
> https://issues.apache.org/jir
the Kafka offset for the consumer
before the first call to poll()
These are the techniques most people use to get end to end exactly once
processing with no duplicates even in the event of a failure.
-hans
> On May 28, 2018, at 12:17 AM, Karthick Kumar wrote:
>
> Hi,
>
> Fac
).
Conclusion
You will see ordered delivery if your either use a key when you publish or
create a topic with one partition.
-hans
> On May 26, 2018, at 7:59 AM, Raymond Xie <xie3208...@gmail.com> wrote:
>
> Thanks. By default, can you explain me why I received the message in
If you create a topic with one partition they will be in order.
Alternatively if you publish with the same key for every message they will be
in the same order even if your topic has more than 1 partition.
Either way above will work for Kafka.
-hans
> On May 25, 2018, at 8:56 PM, Raymond
Sorry I hit send a bit too soon. I was so focused on the systemd part of
the email and not the Mirror Maker part.
Confluent packages include Mirror Maker but the systemd scripts are setup
to use Confluent Replicator rather than Mirror Maker.
My apologies.
-hans
/**
* Hans Jespersen, Director
The latest Confluent packages now ship with systemd scripts. That is since
Confluent Version 4.1 - which included Apache Kafka 1.1
-hans
/**
* Hans Jespersen, Director Systems Engineering, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Fri, Apr 27, 2018 at 11:15 AM, Andrew Otto &l
Kafka 1.1.0
https://issues.apache.org/jira/browse/KAFKA-6240
<https://issues.apache.org/jira/browse/KAFKA-6240>
which seems in include dynamic reconfiguration of SSL keystores
https://issues.apache.org/jira/browse/KAFKA-6241
<https://issues.apache.org/jira/browse/KAFKA-6241>
--
/**
"If your system is stateless and the transformations are not interdependent"
then I would just look at using Kafka Connect's Single Message Transform
(SMT) feature.
-hans
/**
* Hans Jespersen, Director Systems Engineering, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On T
. Previous the consumer could get stuck and not make progress.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-74%3A+Add+Fetch+Response+Size+Limit+in+Bytes
-hans
> On Feb 25, 2018, at 8:04 AM, adrien ruffie <adriennolar...@hotmail.fr> wrote:
>
> Hi Waleed,
>
> thank for you
The period of time in milliseconds after which we force
a refresh of metadata even if we haven't seen any partition leadership changes
to proactively discover any new brokers or partitions
-hans
> On Feb 4, 2018, at 2:16 PM, Wouter Bancken <wouter.banc...@aca-it.be> wrote:
>
> Hi Ha
of the commit log so the lag cannot be predicted in
advance.
-hans
> On Feb 4, 2018, at 11:51 AM, Wouter Bancken <wouter.banc...@aca-it.be> wrote:
>
> Can anyone clarify if this is a bug in Kafka or the expected behavior?
>
> Best regards,
> Wouter
>
>
> On 30 Janu
with indexes beyond
those in Kafka for faster or more complex interactive queries.
-hans
> On Jan 11, 2018, at 1:33 PM, Manoj Khangaonkar <khangaon...@gmail.com> wrote:
>
> Hi,
>
> If I understood the question correctly , then the better approach is to
> consume eve
Check that your __consumer_offsets topic is also setup with replication factor
of 3 and has In Sync Replicas. Often it gets setup first as a one node cluster
with RF=1 and then when the cluster is expanded to 3 nodes the step to increase
the replication factor of this topic gets missed.
-hans
configuration
properties and without coding. If the built in functions are insufficient you
can write your own SMT functions in Java.
-hans
> On Dec 21, 2017, at 7:19 AM, Bill Bejeck <b...@confluent.io> wrote:
>
> Hi Mads,
>
> Great question and yes your use case her
You can call the REST endpoints in KSQL from any programming language. I
wrote some stuff in node.js to call KSQL this way and it works great. The
results don't even have to go to a Kafka topic as the results of and POST
to /query all stream using HTTP.
-hans
/**
* Hans Jespersen, Principal
I think you are just missing the —execute flag.
-hans
> On Oct 25, 2017, at 1:24 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> I wonder if you have hit KAFKA-5600.
>
> Is it possible that you try out 0.11.0.1 ?
>
> Thanks
>
>> On Wed, Oct 25, 2017 at 1:15 PM
tps://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI(AKAListOffset)
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Wed, Sep 27, 2017 at 10:20 AM, Vignesh <vignesh.v...@gma
scripts in the ./bin
directory rather than just typing “confluent start” as it says in the
quickstart documentation.
-hans
> On Sep 19, 2017, at 8:41 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
> we are using the other components of confluent platform without installing
> the
in 0.11 and above see the CLI command bin//kafka-delete-records.sh
-hans
> On Aug 23, 2017, at 7:28 PM, Rahul Singh <rahulronit1...@gmail.com> wrote:
>
> Hello all,
>
> I am unable to purge the topic data from Kafka. Is there any class to flush
> all topic data.
>
> Thank you
We (Confluent) run Kafka as a SaaS-based cloud offering and we do not see any
reason for this feature so I just don’t understand the motivation for it.
Please explain.
-hans
--
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
Doing that doesn't really make sense in a Kafka cluster because the topic
partitions and their replicas are spread out across many brokers in the
cluster. That's what enables the parallel processing and fault tolerance
features of Kafka.
-hans
> On Aug 22, 2017, at 3:14 AM, Mohit Cha
ytest --from-beginning
91
92
93
94
95
96
97
98
99
100
-hans
> On Aug 18, 2017, at 10:32 AM, Manikumar <manikumar.re...@gmail.com> wrote:
>
> This feature got released in Kafka 0.11.0.0. You can
> use
This is an area that is being worked on. See KIP-107 for details.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-107%3A+Add+purgeDataBefore%28%29+API+in+AdminClient
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-107:+Add+purgeDataBefore()+API+in+AdminClient>
-hans
>
See the producer param called metadata.max.age.ms which is "The period of time
in milliseconds after which we force a refresh of metadata even if we haven't
seen any partition leadership changes to proactively discover any new brokers
or partitions."
-hans
> On Aug 4, 2017, at 5:
“kafka-connect-*”. I quick search will
yield a few “kafka-connect-tcp” connectors like this one
https://github.com/dhanuka84/kafka-connect-tcp
<https://github.com/dhanuka84/kafka-connect-tcp>
-hans
> On Jul 4, 2017, at 10:26 AM, Clay Teahouse <clayteaho...@gmail.com> wrote:
&
() to that
offset, and continue consuming with exactly once semantics.
This is how many of the exactly once Kafka Connect Sink Connectors work today.
-hans
> On Jul 1, 2017, at 11:28 PM, fuyou <fuyou...@gmail.com> wrote:
>
> I read the great blog about kafka Exactly-once Sem
Request quotas was just added to 0.11. Does that help in your use case?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas
-hans
> On Jun 29, 2017, at 12:55 AM, sukumar.np <sukumar...@zohocorp.com> wrote:
>
> Hi Team,
>
>
>
> We
Correct. The use of the word "server" in that sentence is meant as broker (or
KafkaServer as it shows up in the 'jps' command) not as a physical or virtual
machine.
-hans
> On Jun 27, 2017, at 1:22 AM, James <896066...@qq.com> wrote:
>
> Hello,
>At https://kaf
Do you list all three brokers on your consumers bootstrap-server list?
-hans
> On Jun 22, 2017, at 5:15 AM, 夏昀 <kingdomm...@126.com> wrote:
>
> hello:
> I am trying the quickstart of kafka documentation,link is,
> https://kafka.apache.org/quickstart. when I moved
) messages and not the lower level semantics that are that
consuming is actually
reading AND writing (albeit only to the offset topic).
-hans
> On Jun 17, 2017, at 10:59 AM, Viktor Somogyi <viktor.somo...@cloudera.com>
> wrote:
>
> Hi Vahid,
>
> +1 for OffsetFetch
If you are setting acks=0 then you don't care about losing data even when the
cluster is up. The only way to get at-least-once is acks=all.
> On Jun 7, 2017, at 1:12 PM, Ankit Jain <ankitjainc...@gmail.com> wrote:
>
> Thanks hans.
>
> It would work but producer will start
Try adding props.put("max.block.ms", "0");
-hans
> On Jun 7, 2017, at 12:24 PM, Ankit Jain <ankitjainc...@gmail.com> wrote:
>
> Hi,
>
> We want to use the non blocking Kafka producer. The producer thread should
> not block if the Kafka is cluster
and see the message format?
-hans
> On Jun 2, 2017, at 9:10 AM, Mina Aslani <aslanim...@gmail.com> wrote:
>
> Hi Hans,
>
> Thank you for your quick response, appreciate it.
>
> In *kafka-connect* docker, I see below settings in
> *kafka-connect.properties* fil
it is running in.
-hans
> On Jun 2, 2017, at 8:12 AM, Mina Aslani <aslanim...@gmail.com> wrote:
>
> Hi,
>
> I would like to add that I use kafka-connect and schema-registery version `
> 3.2.1-6`.
>
> Best regards,
> Mina
>
> On Fri, Jun 2, 2017 at 10:5
.
-hans
> On Jun 2, 2017, at 7:59 AM, Mina Aslani <aslanim...@gmail.com> wrote:
>
> Hi.
>
> Is there any way that I get the data into a Kafka topic in Json format?
> The source that I ingest the data from have the data in Json format,
> however when I look that data
Target is sometime in June. Apache Kafka releases are every 4 months so
February, June, and October of each year
-hans
> On May 30, 2017, at 3:58 PM, Raghav <raghavas...@gmail.com> wrote:
>
> Hans
>
> When will this version (0.11) be available ?
>
> On Tue, Ma
Probably important to read and understand these enhancements coming in 0.11
https://cwiki.apache.org/confluence/display/KAFKA/KIP-117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924
-hans
> On May 30, 2017, at 10:57 AM, Jerry George <jerr...@gmail.com> wrote:
>
> Thank you Hans and Vahid.
>
> That was definitely of great help. Much appreciated!
>
> Regards,
> Jerry
>
> On Tue, May 30, 2017 at 1:53 PM, Vahid S Hashemian <
>
e list of the offsets for a given inactive consumer.
-hans
> On May 30, 2017, at 8:09 AM, Jerry George <jerr...@gmail.com> wrote:
>
> Hi Abhimanyu,
>
> No, actually waiting for someone with operational experience to reply on
> the list. Thank you for bumping the question though
th zookeeper and kafka brokers to help in debugging further
-hans
--
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
> On May 29, 2017, at 1:33 AM, Bennett, Conrad
> <conrad.benn...@verizonwireless.com.INVALID> wrote:
>
&g
, and this is by
design because then it's up to your app to decide to either throw the messages
away, stop publishing, or store them somewhere outside of Kafka.
The easiest solution would be to run more than one broker so that they are
fault tolerant and will take over for any failed broker nodes.
-hans
The producer is asynchronous (assuming you mean the Java Producer)
https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
-hans
> On May 27, 2017, at 5:15 AM, Abhimanyu Nagrath <abhimanyunagr...@gmail.com>
> wrote:
>
> Hi,
probably upgrade to a newer version but that is a separate discussion.
-hans
> On May 25, 2017, at 11:50 AM, Milind Vaidya <kava...@gmail.com> wrote:
>
> In short it should work regardless as per "During the migration phase, if
> the first message in a segment does not ha
even if it is expired, unless all the older segment has been
expired."
If none of the messages in a segment has a timestamp, last modified time
will be used.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Thu, May 25, 2017
together with log.roll.ms to
avoid frequent log segment roll out.
During the migration phase, if the first message in a segment does not have
a timestamp, the log rolling will still be based on the (current time -
create time of the segment)."
-hans
/**
* Hans Jespersen, Principal Systems Engineer,
The timeindex was added in 0.10 so I think you need to use the new Consumer API
to access this functionality. Specifically you should call offsetsForTimes()
https://kafka.apache.org/0102/javadoc/org/apache/kafka/clients/consumer/Consumer.html#offsetsForTimes(java.util.Map)
-hans
> On May
If you enable auto topic creation that that is exactly what will happen.
There are pros and cons to creating topics with defaults values but if you fell
strongly that is the way that you want Kafka to work it is entire possible to
setup the system to work that way.
-hans
> On May 11, 2
Yes you understand correctly that batch == request
-hans
> On Apr 30, 2017, at 11:58 AM, Petr Novak <oss.mli...@gmail.com> wrote:
>
> Thank you a lot.
>
> How requests in max.in.flight.requests.per.connection relates to batches? 1
> request precisely means 1 batch?
to try and produce the next message.
If you set flight.requests.per.connection > 1 (I think the default is 5) then
you can get a commit log with messages out of order wrt the original published
order (because retries are done in parallel rather then in series)
-hans
> On Apr 30, 2017, a
Replication will not effect the users quota as it is done under a different
replication quota (which you can control separately). The user should still see
a 50 MBps maximum rate enforced into each broker.
-hans
> On Apr 23, 2017, at 11:39 PM, Archie <anubhavnidhi1...@gmail.com&
The kafka-console-producer.sh defaults to acks=1 so just be careful with
using those tools for too much debugging. Your output is helpful though.
https://github.com/apache/kafka/blob/5a2fcdd6d480e9f003cc49a59d5952ba4c515a71/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L185
-hans
On Wed
if
the messages are really duplicates in the Kafka log, or if they are just
seeing the same message reprocessed several times in the consumer due to
some other issue with offset commits.
-hans
On Wed, Apr 19, 2017 at 10:19 AM, Onur Karaman <onurkaraman.apa...@gmail.com
> wrote:
> If this
in the kafka cluster.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Tue, Apr 18, 2017 at 4:10 PM, Shrikant Patel <spa...@pdxinc.com> wrote:
> Hi All,
>
> I am seeing strange behavior between ZK and Kafka. We ha
cation
<https://en.wikipedia.org/wiki/Mutual_authentication> and can you provide more
information about the motivation for your question.
-hans
> On Apr 12, 2017, at 1:50 AM, Sriram Srinivasaraghavan (srirsri2)
> <srirs...@cisco.com> wrote:
>
> Team,
>
> Wou
ing-and-async-message-kafka-in-the-same-execu/43312070#43312070>
-hans
> On Apr 8, 2017, at 8:49 PM, Rams N <99ram...@gmail.com> wrote:
>
> Hi,
> I've an usecase to respond to an API call to the client which should happen
> in sync. But within the api execution, the syste
nd N3
have different names.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Thu, Apr 6, 2017 at 4:26 PM, Tushar Sudhakar Jee <tus...@levyx.com>
wrote:
> Hello Sir/Ma'am,
> I was trying to write a simple case of using kaf
Then you will need even more parallel producers to saturate a 10 GigE network
(if you don't hit you disk I/O limit first)
-hans
> On Apr 1, 2017, at 3:15 PM, Archie <anubhavnidhi1...@gmail.com> wrote:
>
> My replication factor is 1.
>
> Thanks,
> Archie
>
>&g
saturate the network if you added a second producer and
consumer at those rates (if your storage system can keep up to the network
bandwidth).
-hans
> On Apr 1, 2017, at 10:25 AM, Archie <anubhavnidhi1...@gmail.com> wrote:
>
> I have set up my kafka cluster in a network with 9.3 Gbp
They are both true. The Apache text is talking about the compatibility of the
Producer/Consumer API and the Confluent text is talking about the Streams API.
-hans
> On Mar 31, 2017, at 11:46 PM, Roger Vandusen
> <roger.vandu...@ticketmaster.com> wrote:
>
> Read below an
username comes from authenticated clients
client.id can be assigned by any client (no authentication required).
It’s hard to enforce a quota on a client.id when the clients can just change
the code to use a difference client.id, hence the recent enhancement to add
user quotas based on the
n and succeed before any other messages are sent.
props.put(“max.in.flight.requests.per.connection”,1);
-hans
> On Mar 30, 2017, at 7:15 AM, Laxmi Narayan <nit.dgp...@gmail.com> wrote:
>
> Hi ,
> I am using kafka 10.2 and sometime my producer does not sends me any ACK
> and in
is the administrative kafka-topics.sh
tool which talks directly to zookeeper and the Kafka brokers to create or
modify topics in the Kafka cluster. This will work to create a topic before you
start your producer app.
-hans
> On Mar 26, 2017, at 2:00 AM, Laxmi Narayan <nit.dgp...@gmail.com> wrote:
Yes, and yes!
-hans
> On Mar 21, 2017, at 7:45 AM, Ali Akhtar <ali.rac...@gmail.com> wrote:
>
> That would require
>
> - Knowing the current window's id (or some other identifier) to
> differentiate it from other windows
>
> - Being able to process i
While it's not exactly the same as the window start/stop time you can store (in
the state store) the earliest and latest timestamps of any messages in each
window and use that as a good approximation for the window boundary times.
-hans
> On Mar 20, 2017, at 1:00 PM, Ali Akhtar <a
before storage in the Kafka log and
Sink Connector transformations happen afterwards for consumers.
-hans
> On Mar 20, 2017, at 6:52 PM, Matt Magoffin <apache@msqr.us> wrote:
>
> Thanks, Hans.
>
> Signing messages is a good idea. Other than that, is there possibly
work.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Mon, Mar 20, 2017 at 3:54 PM, Matt Magoffin <apache@msqr.us> wrote:
> Hello,
>
> I am new to Kafka and am looking for a way for consumers to be able to
> ide
.
-hans
> On Mar 18, 2017, at 11:36 AM, Sabarish Sasidharan <sabarish@gmail.com>
> wrote:
>
> Hans
>
> What you state would work for aggregations, but not for state machines and
> CEP.
>
> Regards
> Sab
>
>> On 19 Mar 2017 12:01 a.m., "Hans Jes
the
added benefit of getting intermediary result at much lower latency.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Sat, Mar 18, 2017 at 10:29 AM, Ali Akhtar <ali.rac...@gmail.com> wrote:
> Is it possible to have
sorry I mixed up Message A and B wrt the to question but the answer is
still valid.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Sat, Mar 18, 2017 at 11:07 AM, Hans Jespersen <h...@confluent.io> wrote:
> The only wa
.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Fri, Mar 17, 2017 at 1:20 PM, Andrew Pennebaker <
andrew.penneba...@gmail.com> wrote:
> If I understand Kafka correctly, since v0.9 / v0.10, users are often
>
-2-million-writes-second-three-cheap-machines
<https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines>
-hans
> On Mar 15, 2017, at 7:51 AM, Nicolas MOTTE <nicolas.mo...@amadeus.com> wrote:
>
> Ok that makes sense, t
ter+Id>
-hans
> On Mar 14, 2017, at 11:20 AM, Sumit Maheshwari <sumitm.i...@gmail.com> wrote:
>
> Can anyone answer the above query?
>
> On Mon, Mar 13, 2017 at 3:41 PM, Sumit Maheshwari <sumitm.i...@gmail.com>
> wrote:
>
>> Hi,
>>
>&g
the latest value for each key
forever, but removes the older messages with the same key in order to
reduce the total about of messages stored.
How much data do you expect to store in your largest topic over the life of
the cluster?
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent
in 0.10.1
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* h...@confluent.io (650)924-2670
*/
On Tue, Mar 14, 2017 at 10:09 AM, Joe San <codeintheo...@gmail.com> wrote:
> Dear Kafka Users,
>
> What are the arguments against setting the retention plociy on
JMS AVRO schema
that includes both the JMS metadata as well as the JMS message body (which can
be any of the JMS message types).
-hans
> On Mar 14, 2017, at 9:26 AM, Robert Quinlivan <rquinli...@signal.co> wrote:
>
> Did you look at the ConsumerRecord
> <https://kafka.ap
1 - 100 of 160 matches
Mail list logo
