Re: CVEs related to Kafka

Consider purchasing support from Confluent to get this sort of request answered quickly. From: Sahil Sharma D Date: Tuesday, May 9, 2023 at 12:40 PM To: users@kafka.apache.org Subject: [EXTERNAL] RE: CVEs related to Kafka Gentle reminder-2 ! -Original Message- From: Sahil Sharma D

Re: [EXTERNAL] Streaming processing in real life scenario

I don’t think Streaming is a good fit for those use cases. It is best for asynchronous processes. For example, once something is hotel room is booked and paid then send a message to update the rewards account. But if you need to gate access to something (like if a room is booked or not)

Re: Newbie looking for a connector I can configure on my mac

FOSS == Free Open Source Software From: andrew davidson Date: Wednesday, March 30, 2022 at 3:16 PM To: users@kafka.apache.org Subject: [EXTERNAL] Re: Newbie looking for a connector I can configure on my mac Thanks Liam. What is 'FOSS Kafka'? google did not find any useful definitions A

Re: Apache Kafka Questions

From: Jatin Chhabriya Date: Wednesday, March 16, 2022 at 9:20 AM To: users@kafka.apache.org Cc: Murali Krishna Subject: [EXTERNAL] Apache Kafka Questions Hello Team Upon careful perusal of documentation and tutorials, our team has a few open questions, and we would appreciate to have these

Re: Connector is still in 'RUNNING' status even though the heartbeat failed

PR means “Pull Request”. It is a way to have others review your code changes and, when ready, they can merge them in. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests From: Andreas Gillmann Date:

Re: [EXTERNAL] Re: Log4j 1.2

ingly to do that > work. > I'm so sorry that I can't help. > > Best regards > Franziska > > -Ursprüngliche Nachricht- > Von: Tauzell, Dave > Gesendet: Montag, 10. Januar 2022 14:30 > An: users@kafka.apache.org > Betreff: Re: Log4j 1.2 > > Log4j

Re: Log4j 1.2

Log4j 2.x isn’t a drop-in replacement for 1.x. It isn’t a difficult change but somebody does need to go through all the source code and do the work. -Dave From: Brosy, Franziska Date: Monday, January 10, 2022 at 3:16 AM To: users@kafka.apache.org Subject: [EXTERNAL] AW: Log4j 1.2 Hi Roger,

Re: [External] Monitoring Consumer

the consumer publish notifications about messages it has processed to a new topic (or other storage mechanism). You may be able to use the admin api, but I don't think it's a standard use case. On Tue, May 25, 2021, 8:21 AM Tauzell, Dave wrote: > I don’t know about monitoring when a particu

Re: [External] Monitoring Consumer

I don’t know about monitoring when a particular message is reads but you can use something like https://github.com/linkedin/Burrow to monitor consumer lag. Basically you can see that consumer Y has not yet read X number of messages that are ready. -Dave From: Alberto Moio Date: Tuesday,

Re: [External] How to read-your-writes in a side effect

When you send a message the returned RecordMetadata has an offset. If you know the name of the Consumer Group that is reading these messages there is an API to get the latest offset of the Consumer Group so you could poll that and wait for the latest Offset to be greater than the offset of

Re: [External] How to read-your-writes in a side effect

Your consumer will need to put some sort of response message on a different topic and the API will have to subscribe and wait for that response message. You'll need some sort of key so that it can know which one was for that call. I would avoid this type of API, however. Instead I would

Re: [External] key and subkey sequential processing across competing consumer groups?

So if the stream is: A:1:FOO A:3:BAR A:3:BAZ Then A:3* must be processed after A:1 but A:3:BAR and A:3:BAZ can be processed in any order? I don’t think there is a way to do that with topics. -Dave From: Andre Mermegas Reply-To: "users@kafka.apache.org" Date: Wednesday, September 2, 2020

Re: [External] kafka connector distributed mode balance stratety

When clients connect the Kafka broker assigns each client a subset of the topic partitions. When a client becomes unresponsive then the broker assigns those partitions to another client. The state of your connector task isn't going to be transferred but another connector task will eventually

Re: [External] Auto Scaling in Kafka

If you run in the cloud I think there are some solutions. If you think you might need to add brokers in the future then make sure you create enough partitions ahead of time. It is easier to add brokers and have them take over existing partitions than it is to re-partition data. I don't

Re: [External] Allow parallel processing

I would go with #1: 1. It will be easier to add new "batch producers" since you won't need to worry about re-partitioning 2. You have more control over the parallelism since you can have different numbers of partitions for each topic 3. You can easily split out your consumer into N consumers if

Re: [External] AW: Consumer Lags and receive no records anymore

e: Hi Dave, thank you . saw some tutorial where they told it otherwise .. which confuses me a litte. If its done round-robin .. my "world view" makes sense again  Oliver -Ursprüngliche Nachricht----- Von: Tauzell, Dave Gesendet:

Re: [External] AW: Consumer Lags and receive no records anymore

A null key results in the client sending to partitions in a round-robin order. Use a key if you want to ensure that specific messages end up on the same partition. -Dave On 11/8/19, 1:06 AM, "Oliver Eckle" wrote: Hi, Don’t get me wrong, I just want to understand what's going on.

Re: [External] Re: Need help to find references to antipatterns/pitfalls/incorrect ways to use Kafka

If somebody insists on using Kafka as a database you might be able to do the following: 1. Create a "compacted topic". The key for the topic should be the point of sales id. 2. Create a webservice which takes a point of sale id and can read or update the topic 3. Have the point of sale apps

Re: [External] Re: Newbie question using Kafka Producers in Web apps

We are using both and leaning towards a web service fronting Kafka because it gives us the ability to centralize other logic. That said, I don't think the webservice will be much more "stable" and you'll need to consider what to do with your audit records if the web service call fails. -Dave

RE: [External] What happens if the complete Kafka Cluster crashes

It is possible that if all the nodes fail at about the same time and after the broker acknowledged the message, then some messages will be lost because they were in memory and not yet fully written to the disk. If you set ACKS=all then this requires all of your replicas to fail in this way to

RE: Need info

If you size your cluster right, you can send large messages of many megabytes. We send lots (millions per day) of medium sized messages (5-10k) without any issues. -Dave -Original Message- From: Chanchal Chatterji [mailto:chanchal.chatte...@infosys.com] Sent: Wednesday, September 12,

RE: [External] Re: Java API to read metrics via JMX

We use Jolokia (which has a java agent you can load with kafka to expose metrics via HTTP) and Influx/Telegraf which has support for Jolokia. There is a fair bit of configuration but it can be done without any coding. -Dave -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com]

RE: [External] Kafka - Is it possible to achieve low latency (below 5ms?)

What does the hardware side of your brokers look like - do you have enough memory to hold all pending messages in memory (i.e. before consumers get them). At what rate are your clients trying to send messages? -Dave -Original Message- From: Pritam Kadam [mailto:kpri...@thoughtworks.com]

Confluent Replicator

Does anybody have any experience with Confluent Replicator? Has it worked well for you? -Dave This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If

RE: what are common ways to convert info on a web site into a log entry?

I would have a cron that runs every day but somehow tracks if it has pulled data for the month. If it has it just does nothing. This way if you have some sort of failure one day (website is down, etc ...) it would pull data the next day. You could possibly use Kaka itself to store the last

RE: Best practice for publishing byte messages to Kafka

Whatever you use I recommend some sort of wrapper since Kafka doesn't support any sort of metadata (like the version of the serialization format). -Dave -Original Message- From: Matt Farmer [mailto:m...@frmr.me] Sent: Thursday, January 11, 2018 8:56 AM To: users@kafka.apache.org Subject:

RE: Kafka & Canary Release

If you haven’t built in logic from the start (with micro-service version 1) then I think you’ll need some sort of “router” in the middle that knows the routing logic. -Dave From: Assaf Katz [mailto:assaf.k...@amdocs.com] Sent: Wednesday, December 13, 2017 3:12 AM To: Yuval Alon

RE: Multiple brokers - do they share the load?

You then also need to set this up for each topic you create: > bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor > 3 --partitions 3 --topic my-replicated-topic -Dave -Original Message- From: Skip Montanaro [mailto:skip.montan...@gmail.com] Sent: Thursday,

RE: Multiple brokers - do they share the load?

If you create a partitioned topic with at least 3 partitions then you will see your client connect to all of the brokers. The client decides which partition a message should go to and then sends it directly to the broker that is the leader for that partition. If you have replicated topics,

RE: Time based data retrieval from Kafka Topic

What are you going to do with the messages every 15 minutes? One way I can think of is to have two consumers in your application. One of them reads messages and just keeps track of the offsets for each hour. The other consumer then uses this info to pull the data. You could publish these

Re: Strange Kafka throughput issues

Have you tried increasing max.in.flight.requests.per.connection? I wonder if that would be similar to you having multiple producers. Dave Sent using OWA for iPhone From: Sunny Kim Sent: Wednesday, August 30, 2017 4:55:02 PM To:

RE: Different Schemas on same Kafka Topic

, all avro messages holds same id. Then how multiple schemas on same topic possble ? Please clarify Thanks, Sreejith On 17-Aug-2017 9:49 pm, "Tauzell, Dave" <dave.tauz...@surescripts.com> wrote: > > How does consumer know A is the avro class when there could be > &

RE: Different Schemas on same Kafka Topic

h -Original Message- From: Tauzell, Dave [mailto:dave.tauz...@surescripts.com] Sent: Thursday, August 17, 2017 8:30 AM To: users@kafka.apache.org Subject: RE: Different Schemas on same Kafka Topic It does. The way it works is that the Avro serializer precedes each message with a two-byte

RE: Different Schemas on same Kafka Topic

It does. The way it works is that the Avro serializer precedes each message with a two-byte integer that references a schema id in the confluent schema registry. The Avro deserializer looks at this value to determine which schema to de-serialize with. In order for this to work you need to

RE: How performant is the confluent REST proxy

I don't have any concrete numbers but the REST proxy is quite a bit slower. That said, it can still be fast and can scale out so it might meet your needs. -Dave -Original Message- From: Affan Syed [mailto:as...@an10.io] Sent: Thursday, August 10, 2017 1:32 AM To: users@kafka.apache.org

RE: Hello, Help!

As others mentioned this is not a forum to discuss the works of Franz Kafka. Here are some places to get you started: 1. The works of Franz Kafka: https://www.vanderbilt.edu/olli/class-materials/Franz_Kafka.pdf 2. Literature stack exchange: https://literature.stackexchange.com/ -Dave

RE: Using Kafka Producer inside Oracle DB

>> java.lang.NoClassDefFound Error You are missing some dependent classes. Two questions: 1. Does the message have more information about what class it couldn't find? 2. What exactly are you putting into your jar file? -Dave -Original Message- From: Rahul R04

RE: Kafka logs - server.log & controller.log

All the brokers write to server.log. The broker that happens to be the controller will also write to the controller.log file. -Dave -Original Message- From: karan alang [mailto:karan.al...@gmail.com] Sent: Wednesday, June 28, 2017 6:04 PM To: users@kafka.apache.org Subject: Kafka logs

Re: How does Zookeeper node failure impact Kafka cluster?

Losing one out of three should not impact the cluster. Losing more than a majority means certain Kafka operations won't work. Anything that requires the zookeeper data like electing a new leader for example. Dave Sent using OWA for iPhone From: mayank

RE: Handling 2 to 3 Million Events before Kafka

I’m not really familiar with Netty so I won’t be of much help. Maybe try posting on a Netty forum to see what they think? -Dave From: SenthilKumar K [mailto:senthilec...@gmail.com] Sent: Wednesday, June 21, 2017 10:28 AM To: Tauzell, Dave Cc: users@kafka.apache.org; senthilec...@apache.org; d

RE: Handling 2 to 3 Million Events before Kafka

. It seems possible with the right sort of kafka producer tuning. -Dave From: SenthilKumar K [mailto:senthilec...@gmail.com] Sent: Wednesday, June 21, 2017 8:55 AM To: Tauzell, Dave Cc: users@kafka.apache.org; senthilec...@apache.org; d...@kafka.apache.org; Senthil kumar Subject: Re: Handling 2 to 3

RE: Handling 2 to 3 Million Events before Kafka

What are your configurations? - production - brokers - consumers Is the problem that web servers cannot send to Kafka fast enough or your consumers cannot process messages off of kafka fast enough? What is the average size of these messages? -Dave -Original Message- From: SenthilKumar

RE: Larger payload size

Lots of large messages will slow down throughput. From the client side you might want to have a client for large messages and one for the others so that they each have their own queue. -Dave -Original Message- From: Ghosh, Achintya (Contractor) [mailto:achintya_gh...@comcast.com]

RE: Android app produces data in Kafka

Sounds like there are some issues using the Kafka java library on Android. I think instead you should create a REST api (or use the REST proxy provided by Confluent) and have your device make HTTP calls to something that then puts messages onto Kafka. -Dave -Original Message- From:

RE: client recordmetadata meaning

. It's somewhat not clear to me in the documentation, whether log.flush.interval.messages is connected to the ACK sent to the client. From: Tauzell, Dave <dave.tauz...@surescripts.com> Sent: Tuesday, May 30, 2017 5:08:04 PM To: users@kafka.apache.org Subje

RE: Kafka 24/7 support

Both Confluent and Cloudera provide support. -Dave From: Benny Rutten [mailto:brut...@isabel.eu] Sent: Wednesday, April 26, 2017 2:36 AM To: users@kafka.apache.org Subject: Kafka 24/7 support Good morning, I am trying to convince my company to choose Apache Kafka as our standard messaging

RE: Kafka Connect

Also, see this article on streaming changes from MySQL to kafka: https://wecode.wepay.com/posts/streaming-databases-in-realtime-with-mysql-debezium-kafka -Original Message- From: Tauzell, Dave Sent: Monday, February 27, 2017 9:07 AM To: users@kafka.apache.org Subject: RE: Kafka Connect

RE: Kafka Connect

Are you specifically talking about relational databases?Kafka Connect has a JDBC source (http://docs.confluent.io/3.1.1/connect/connect-jdbc/docs/source_connector.html) which can push data changes to kafka. It can only run sql queries, though, so out of the box it will just get you

RE: about producer and consumer api

You'll need to provide some details. At a minimum the error message that you are getting. -Dave -Original Message- From: VIVEK KUMAR MISHRA 13BIT0066 [mailto:vivekkumar.mishra2...@vit.ac.in] Sent: Friday, February 10, 2017 4:22 AM To: users@kafka.apache.org Subject: about producer and

RE: Kafka Connect in different nodes than Kafka.

Yes, you just need to point it to your cluster. -Dave -Original Message- From: Guillermo Ortiz [mailto:konstt2...@gmail.com] Sent: Wednesday, February 1, 2017 1:09 PM To: users@kafka.apache.org Subject: Kafka Connect in different nodes than Kafka. Is it possible to use Kafka Connect in

RE: Taking a long time to roll a new log segment (~1 min)

; > >>> > > > >>> > 2017-01-12T07:16:46.867-0500: 46891.844: Total time for which > > >>> application > > >>> > > threads were stopped: 0.0141281 seconds, Stopping threads took: > > >>> 0.0002171 > > >>&

RE: First time implimention of publish and subscribe

When you say "high transaction" what are your estimates? ActiveMQ will be easier to get started with. If you design your code to hide the queuing it won't be too hard to switch between them. Kafka provides (I think) more scalability and HA but is going to more complicated to operate if you

RE: Taking a long time to roll a new log segment (~1 min)

G1 gc with the following settings: >>> > >>> > -Xmx12G -Xms12G -server -XX:MaxPermSize=48M -verbose:gc >>> > -Xloggc:/var/log/kafka/gc.log -XX:+PrintGCDateStamps >>> -XX:+PrintGCDetails >>> > -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationSto

RE: can i delete topic like this ?

You can set the retention for the topic to a small time and then wait for Kafka to delete the messages before setting it back: bin/kafka-topics.sh --zookeeper zk.prod.yoursite.com --alter --topic TOPIC_NAME --config retention.ms=1000 -Original Message- From: Laxmi Narayan NIT DGP

RE: Taking a long time to roll a new log segment (~1 min)

Can you collect garbage collection stats and verify there isn't a long GC happening at the same time? -Dave -Original Message- From: Stephen Powis [mailto:spo...@salesforce.com] Sent: Thursday, January 12, 2017 8:34 AM To: users@kafka.apache.org Subject: Re: Taking a long time to roll a

Re: Kafka as a data ingest

Can you explain in more detail? Do you want to have files created in hdfs somehow broken into records and put into Kafka? > On Jan 9, 2017, at 19:57, Cas Apanowicz wrote: > > Hi, > > I have general understanding of main Kafka functionality as a streaming tool. >

RE: adding more partition

Kafka persists messages to disk (would be SSD if that is what you have). However, if you have a large enough memory and are pulling off messages quickly, then the receivers will likely get the messages directly from memory and the write to disk will happen asynchronously in the background.

Re: Processing time series data in order

If you specify a key with each message then all messages with the same key get sent to the same partition. > On Dec 26, 2016, at 23:32, Ali Akhtar wrote: > > How would I route the messages to a specific partition? > >> On 27 Dec 2016 10:25 a.m., "Asaf Mesika"

RE: Kafka as a database/repository question

What is the plan for backup and recovery of the kafka data? -Dave -Original Message- From: Susheel Kumar [mailto:susheel2...@gmail.com] Sent: Thursday, December 15, 2016 12:00 PM To: users@kafka.apache.org Subject: Kafka as a database/repository question Hello Folks, I am going thru an

Re: Some general questions...

I don't know if any API to stream a message. I don't suggest putting lots of large messages onto Kafka. As far as documentation I hear that confluent is going to support a C and C# client so you could try asking questions on the confluent mailing list. Dave On Dec 5, 2016, at 17:51, Doyle,

Re: Suggestions

Can you use the console consumer to see the messages on the other topics? > On Dec 2, 2016, at 04:56, Vincenzo D'Amore wrote: > > Hi Kafka Gurus :) > > I'm creating process between few applications. > > First application create a producer and then write a message into a main

RE: I need some help with the production server architecture

and aggregated to a table. Keyed message will be around 1 KB or so. On Thu, Dec 1, 2016 at 9:44 PM, Tauzell, Dave <dave.tauz...@surescripts.com> wrote: > Do you have some idea of the size and number of messages per second > you'll put onto the topics at peak? > > -Dave > &g

RE: I need some help with the production server architecture

with the production server architecture And what about my brokers. Should I hedge them as well. Like say put 2 zk on nodejs server and 1 on db server. Put 2 brokers on db server and 1 on nodejs server, something like that. Thanks Sachin On Thu, Dec 1, 2016 at 8:59 PM, Tauzell, Dave <dave.t

RE: I need some help with the production server architecture

For low volume zookeeper doesn't seem to use many resources. I would put it on nodejs server as that will have less IO and heavy IO could impact zookeeper. Or, you could put some ZK nodes on nodejs and some on DB servers to hedge your bets. As always, you'll find out a lot once you

Re: while publishing message need to add multiple keys in a single message

Kafka doesn't have the concept of message headers like some other messaging systems. You will have to create a payload that contains these headers and whatever bytes you are sending. Dave > On Nov 28, 2016, at 16:47, Prasad Dls wrote: > > Hi, > > While publishing each

Re: Spring Kafka is not working with Spring 3.0.5.RELEASE

part of > spring-core 4.x.x > > On Sun, Nov 27, 2016 at 8:41 AM, Tauzell, Dave <dave.tauz...@surescripts.com >> wrote: > >> It looks like you are missing a spring jar. Can you google to find out >> which jar that class is in? >> >> Dave >> >>

Re: Spring Kafka is not working with Spring 3.0.5.RELEASE

It looks like you are missing a spring jar. Can you google to find out which jar that class is in? Dave > On Nov 27, 2016, at 01:16, Prasad Dls wrote: > > Hi users, > > > My project is already developed with Spring 3.0.5.RELEASE, We are planning > to use Kafka for new

RE: Oversized Message 40k

I ran tests with a mix of messages, some as large as 20MB. These large messages do slow down processing, but it still works. -Dave -Original Message- From: h...@confluent.io [mailto:h...@confluent.io] Sent: Tuesday, November 22, 2016 1:41 PM To: users@kafka.apache.org Subject: Re:

Re: Investigating apparent data loss during preferred replica election

Do you have: Unclean.leader.election.enable = false ? Dave > On Nov 17, 2016, at 19:39, Mark Smith wrote: > > Hey folks, > > I work at Dropbox and I was doing some maintenance yesterday and it > looks like we lost some committed data during a preferred replica > election. As far as

RE: Topics vs Partitions

Partitions are used to distribute the messages in a topic between several different broker instances. This provides higher throughput. Partitions can also be replicate which allows for high availability. -Dave From: Doyle, Keith [mailto:keith.do...@greenwayhealth.com] Sent: Wednesday,

RE: is there a way to make sure two consumers receive the same message from the broker?

Here is a scenario where this could be useful: Add the kafka offset as a field on the record in both Cassandra and Elasticsearch Now when you get search results from Elastic search and look up details in Cassandra you can know if they come from the same kafka record. If you can use the

Re: is there a way to make sure two consumers receive the same message from the broker?

You should have one consumer pull the message and submit the data to each storage using an XA transaction. > On Nov 5, 2016, at 19:49, kant kodali wrote: > > yes this problem can definetly be approached in many ways but given the > hard constraints by our clients we don't

Re: Link read avro from Kafka Connect Issue

Is Kafka connect adding some bytes to the beginning of the avro with the scheme registry id? Dave > On Nov 2, 2016, at 18:43, Will Du wrote: > > By using the kafka-avro-console-consumer I am able to get rich message from > kafka connect with AvroConvert, but it got no

RE: Kafka edge nodes

You want the servers in the primary zone to put messages onto Kafka and applications in the edge nodes to read and process them? -Dave This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or

RE: Kafka client circuit breaker

the brokers before the message is gonna be read/written. Thank you for support! On Mon, 31 Oct 2016 at 02:41 Tauzell, Dave <dave.tauz...@surescripts.com> wrote: > Once enough failures happen the circuit is marked open. The client > would then periodically try some messages un

Re: Kafka client circuit breaker

Once enough failures happen the circuit is marked open. The client would then periodically try some messages until it works again. Others would be failed. There are a number of existing circuit breaker libraries you can use in the meantime like the Netflix one. Dave > On Oct 30, 2016, at

RE: Kafka Multi DataCenter HA/Failover

VALID> > wrote: > > Thanks dave. > Any ways for how we can achieve HA/Failover in kafka across two DC? > Thanks,Mudit > > From: "Tauzell, Dave" <dave.tauz...@surescripts.com> > To: "users@kafka.apache.org" <users@kafka.apache.org>; Mudit Ag

RE: Kafka Multi DataCenter HA/Failover

I don't know of anything to handle that situation for you, but your application can be written to do that. -Dave -Original Message- From: Mudit Agarwal [mailto:mudit...@yahoo.com.INVALID] Sent: Friday, October 28, 2016 11:08 AM To: Tauzell, Dave; users@kafka.apache.org Subject: Re

RE: Kafka Multi DataCenter HA/Failover

...@yahoo.com.INVALID] Sent: Friday, October 28, 2016 10:09 AM To: users@kafka.apache.org Subject: Re: Kafka Multi DataCenter HA/Failover Thanks dave. Any ways for how we can achieve HA/Failover in kafka across two DC? Thanks,Mudit From: "Tauzell, Dave" <dave.tauz...@surescripts.com

RE: Kafka Multi DataCenter HA/Failover

>> without any lag You are going to have some lag at some point between datacenters. I haven't used this but from taking to them they are working or have created a replacement for MirrorMaker using the Connect framework which will fix a number of MirrorMaker issues. I haven't talked to

RE: Removing kafka topic contents from Java

There is a script kafka-topics.sh which calls kafka.admin.TopicCommand. I think you could probably call that same class from your application if you figure out which jars are needed. When I do testing I'll clear out queues by setting the retention time to a second and then wait for kafka to

RE: How to block tests of Kafka Streams until messages processed?

For similar queue related tests we put the check in a loop. Check every second until either the result is found or a timeout happens. -Dave -Original Message- From: Ali Akhtar [mailto:ali.rac...@gmail.com] Sent: Wednesday, October 19, 2016 3:38 PM To: users@kafka.apache.org Subject:

RE: Kafka as A Metrics Collector

I don't see why you couldn't. InfluxDB's Telegraph (https://docs.influxdata.com/telegraf/v1.0/) has plugin to read from Kafka. You could use Spark or Kafka streams to do custom aggregation. -Dave -Original Message- From: Furkan KAMACI [mailto:furkankam...@gmail.com] Sent: Sunday,

librdkafka

Does anybody know if the librdkafka releases are kept in step with kafka releases? -Dave This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have

RE: Architecture recommendations for a tricky use case

Spark Streaming needs to store the output somewhere. Cassandra is a possible target for that. -Dave -Original Message- From: Ali Akhtar [mailto:ali.rac...@gmail.com] Sent: Thursday, September 29, 2016 9:16 AM Cc: users@kafka.apache.org; spark users Subject: Re: Architecture

RE: Kafka connect 2.0.1 - ByteArrayConverter ?

The current converters want you to send Avro records with a "schema id" prepended to the serialized Avro. You also need the schema registry running. I'm guessing this is what Olivier is talking about. I think it is possible to write your own converter that doesn't need this but I haven't

RE: Does Kafka Sync/persist every message from a publisher by default?

If by "sync" you mean "fsync" then, no it does not. There are some properties: log.flush.interval.messages log.flush.interval.ms In theory you could set log.flush.interval.messages to 1 to fsync with each write. I haven't tried this to see what happens but I expect performance will drop

RE: why did Kafka choose pull instead of push for a consumer ?

Kafka writes each message but the OS is writing those to in memory disk cache. Kafka periodically calls fsync() to tell the OS to force the disk cache to actual disk. Kafka gets high availability by replicating messages to other brokers so that the messages are in-memory on several machines

RE: Kafka duplicate offset at Consumer

Are you using the new java consumer? What method are you using to commit offsets? -Dave -Original Message- From: Ghosh, Achintya (Contractor) [mailto:achintya_gh...@comcast.com] Sent: Tuesday, September 20, 2016 8:56 AM To: users@kafka.apache.org Cc: d...@kafka.apache.org Subject:

RE: No error to kafka-producer on broker shutdown

The send() method returns a Future. You need to get the result at some point to see what happened. A simple way would be: m_kafkaProducer.send(prMessage).get(); -Dave -Original Message- From: Agostino Calamita [mailto:agostino.calam...@gmail.com] Sent: Friday, September 9, 2016 9:33

Re: Kafka : Read replica related

The leader for each partition is on a different broker. Example: Three brokers Topic has three partitions and replication of three. In this case each broker will be a leader for one partition and a follower for two. Three consumers would each be reading from a different topic. Dave > On Sep

Production Use Cases

Does anybody do the following in production? If so, what are your experiences? 1. Use .Net applications for producers or consumers 2. Consume messages across the WAN (across datacenters) - I'm wondering if MirrorMaker is always a requirement for cross-WAN -Dave This e-mail and any

RE: Kafka Producer performance - 400GB of transfer on single instance taking > 72 hours?

I would write a python client that writes dummy data to kafka to measure how fast you can write to Kafka without MongoDB in the mix. I've been doing load testing recently can with 3 brokers I can write 100MB/s (using Java clients). -Dave -Original Message- From: Dominik Safaric

RE: 0.10 Metrics Reporter Suggestions

Take a look at jmxtrans: https://github.com/jmxtrans. There are a number of options, including some ones you can ebmed into your java application. -Dave -Original Message- From: Lawrence Weikum [mailto:lwei...@pandora.com] Sent: Monday, August 22, 2016 2:32 PM To:

kakfa-console-consumer multiple topics

Is there a way to have the kafka-console-consumer read from multiple topics? I'm using this for some testing and that would be helpful. -Dave This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual

Re: Kafka java consumer processes duplicate messages

If you kill a broker, then any uncommitted messages will be replayed. -Dave From: R Krishna Sent: Monday, August 1, 2016 1:32 PM To: users@kafka.apache.org Subject: Re: Kafka java consumer processes duplicate messages Remember

RE: compression ratio

give you that info. Regards Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Jul 29, 2016, at 2:58 PM, Tauzell, Dave <dave.tauz...@surescripts.com> > wrote: > > Is there a good way to see what sort of compression ratio is being achieved? > >

RE: Kafka streams Issue

.@supcom.tn] Sent: Friday, July 29, 2016 9:53 AM To: users@kafka.apache.org Subject: RE: Kafka streams Issue Hi Dave, Could you explain a little bit much your idea ? I can't figure out what you are suggesting. Thank you -Hamza ________ De : Tauzell, Dave <dave.tauz...@su

RE: compression ratio

...@confluent.io] Sent: Friday, July 29, 2016 9:27 AM To: users@kafka.apache.org Subject: Re: compression ratio Hi Dave The JMX metric compression-rate-avg should give you that info. Regards Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Jul 29, 2016, at 2:58 PM, Tauz

RE: Kafka streams Issue

You could send the message immediately to an intermediary topic. Then have a consumer of that topic that pull messages off and waits until the minute is up. -Dave Dave Tauzell | Senior Software Engineer | Surescripts O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com

  1   2   >