Hi,
does a Kafka broker batch messages together in order to improve compression
when using broker-side compression only?
If so, what are the prerequisites to get that behavior? Perhaps a batching of
messages at a producer might be beneficial or even mandatory in order to get
batches in the log
One more question:
Is there a way to ask Kafka which ProducerRecord.key is mapped to which
TopicPartition (for debugging purposes etc.)?
Gesendet: Montag, 14. Januar 2019 um 13:49 Uhr
Von: "Sven Ludwig"
An: users@kafka.apache.org
Betreff: Aw: Re: Re: Doubts in Kafka
Hi,
>> .
Uhr
Von: "Peter Levart"
An: users@kafka.apache.org, "Sven Ludwig"
Betreff: Re: Aw: Re: Doubts in Kafka
On 1/10/19 2:26 PM, Sven Ludwig wrote:
> Okay, but
>
> what if one also needs to preserve the order of messages coming from a
> particular device?
>
> With
Okay, but
what if one also needs to preserve the order of messages coming from a
particular device?
With Kafka, this is perhaps possible if all messages from a particular device
go into the same partition.
Would it be a good and efficient solution for this approach to set the key of
each Kaf
Hi Martin,
even though with Kafka I as a fellow user never found it problematic to just
give upgrades a shot no matter the version increase in a test environment,
since you are still on 0.8 you should thoroughly study the details at this link
and depending on whether you care about one or more
Hi,
I have the case very often that I need to find out the Start and End Offset of
a Kafka Topic.
Usually I go to the server to the place where the Topic contents (segments) are
stored and look at the file names. This I do this way to avoid using Console
Consumers since they make use of Offs
messages. Only applicable to new messages.
On Fri, Dec 29, 2017 at 6:59 PM, Ted Yu wrote:
> Looking at https://issues.apache.org/jira/browse/KAFKA-5686 , it seems you
> should have specified LZ4.
>
> FYI
>
> On Fri, Dec 29, 2017 at 5:00 AM, Sven Ludwig wrote:
>
> > Hi,
>
Hi,
we thought we have lz4 applied as broker-side compression on our Kafka Cluster
for storing measurements, but today I looked into the individual .log files and
I was able to read all the measurements in plain text by just using less on the
command line. This is for me an indicator that batc
16:17 Uhr
Von: "Sven Ludwig"
An: users@kafka.apache.org
Betreff: Number of Processes Kafka Server spawns
Hi,
does Kafka ever launch processes in addition to the Kafka server process?
I have a setup with CentOS plus Docker plus Kafka Server container based on the
image of Confluent Plat
Hi,
does Kafka ever launch processes in addition to the Kafka server process?
I have a setup with CentOS plus Docker plus Kafka Server container based on the
image of Confluent Platform 4.0.0
In ps -Alf there is only one Kafka process, but in htop I can see many.
Regards,
Sven
Hello,
with recent Kafka, would it be okay to have about 1000 topics, with between
1000 to 3000 partitions each, on a 6-node Kafka cluster with replication factor
3?
Each partition would be written to by one producer and consumed by one
consumer, with about 2 messages per minute coming in.
Wo
Hello,
assume that all producers and consumers regarding a topic-partition have been
shutdown.
Is it possible in this situation to empty that topic-partition, while the other
topic-partitions keep working?
Like for example, is it possible to trigger a log truncation to 0 on the leader
for
a topic to give it a topic-level override for compression.type
Kind Regards
Sven
Gesendet: Dienstag, 11. Juli 2017 um 11:41 Uhr
Von: "Sven Ludwig"
An: users@kafka.apache.org
Betreff: Need help regarding Compression
Hi,
I need help. We have a Kafka cluster in production with topics
Hi,
I need help. We have a Kafka cluster in production with topics compressed with
snappy. We want to configure compression.type lz4 on the brokers and restart
them (either rolling update, or all down all up if this has benefits).
What happens to the existing topics compressed with snappy afte
by that broker will
be in the new compression type. However, followers don't uncompress data,
so they will store the data as it was in the leader. An easier way to test
what you are trying to test is to use MirrorMaker to mirror the data to
another cluster.
Ismael
On Mon, Jun 26, 2017 at 11:
Hi,
1. I would like to test lz4 compression on a new broker node that I want add to
my cluster, while the other nodes remain in snappy, in order to compare disk
usage etc. I am not sure if this scenario is even possible with only one
cluster, since in the docs it is mentioned that compression
ere.
Hope this helps,
Apurva
On Tue, Dec 6, 2016 at 10:06 AM, Sven Ludwig wrote:
> Hello,
>
> in our Kafka clusters we sometimes observe a specific ERROR log-statement,
> and therefore we have doubts whether it is already running sable in our
> configuration. This occurs every now an
Hi,
in JMX each Kafka broker has a value 1 or 0 for ActiveControllerCount. As I
understood from this thread, the sum of these values across the cluster should
never be something other than 1. The documentation at
http://docs.confluent.io/3.1.0/kafka/monitoring.html should be improved to make
Hello,
in our Kafka clusters we sometimes observe a specific ERROR log-statement, and
therefore we have doubts whether it is already running sable in our
configuration. This occurs every now and then, like two or three times in a
day. It is actually the predominant ERROR log-statement in our c
19 matches
Mail list logo
