You can use something like this to get a comma-separated list of all filed
in a folder:
ls -l | awk '{print $9}' ORS=','
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http:/
Do you know in advance (when sending the first message) how many messages
that batch is going to have?
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://semat
nc replica becomes
the leader.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Thu, Sep 29, 2016 at 7:30 PM, Ezra Stuetzel <e
enough for any type of use case? What do
you think cannot be achieved?
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Thu, Sep 15,
BTW regarding latency:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Con
lag; i.e. lag can fit
in the OS page cache so you're not even hitting disk when consuming)
measured in low 10s of ms.
No read replicas. You only read from partition's master. I.e. replicas are
used to achieve redundancy.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log
Hi Karin,
regarding 5 (fsyncing to disk), take a look at the broker configuration
parameters whose names start with log.flush.
http://kafka.apache.org/documentation.html#brokerconfigs
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
Hi Tom,
if you need a commercially proven lag monitoring solution (and all other
Kafka and ZK metrics) take a look at our SPM.
Hope you don't mind me plugging this one in :)
[image: Inline image 1]
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr
to assign the second consumer with a different group, each
consumer would consume all messages (independently of one another).
BTW, Kafka is not broadcasting anything, your consumers are pulling
messages out of Kafka :)
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log
Instantly reminded me of Streams API, where you can use Java8 streams
semantics (filter being one of them) to do the first thing in Gouzhang's
response (filter messages from one topic into another - I assume that's
what you were looking for).
Marko Bonaći
Monitoring | Alerting | Anomaly Detection
I'm assuming that you created a topic with replication factor 3, while
having only a single broker.
Try with replication factor 1 or add additional brokers.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext &
Also sent to: ggol...@hortonworks.com
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Fri, Apr 15, 2016 at 1:47 AM, Gurudi
Automated reply:
thank you for attempting to subscribe to Kafka mailing list.
To finish the subscription process send email to
users-subscr...@kafka.apache.org < users-subscr...@kafka.apache.org>
:)
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Managemen
+
Has anything changed in 0.9?
Thanks
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
-providers
Amazon Kinesis would also work.
Anything really that would "outsource" the initial effort until you're
ready to commit to Kafka.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sem
regardless of the load.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Mon, Mar 21, 2016 at 6:25 PM, Ben Stopford <b...@
These two issues track progress of Kafka consumer 0.9.
https://github.com/apache/spark/pull/10953
https://github.com/apache/spark/pull/11143
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/>
Hello again Cosmin :)
I think this is because offsets are kept in a special _consumer_offsets
topic, as opposed to ZK previously.
Take a look here:
http://search-hadoop.com/m/uyzND1T1i3BNkRFM1=Re+Kafka+0+8+2+1+how+to+read+from+__consumer_offsets+topic+
Marko Bonaći
Monitoring | Alerting | Anomaly
Hi Cosmin,
do you have default server configuration on these new nodes you're setting
up?
I'd check consumer's socket.timeout.ms, maybe someone set it to 30 instead
of 30 000 :)
Speaking from my own experience (I had the same symptom and this turned out
to be the cause).
Marko Bonaći
Monitoring
Actually, why don't you use the same code as outlined here (that includes
timeout in props):
http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
n requests?
FINALLY: tell us more about your use case.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Mon, Jan 4, 2016 at 1
here:
https://github.com/quantifind/KafkaOffsetMonitor
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Wed, Dec 30, 2015 at
by default)
I'm still on Kafka 0.8, so I can't shed any light on your issue.
Thx for the AdminClient info.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about
I was refering to Dana Powers's answer in the link I posted (to use a
client API). You can find an example here:
http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr
We recently had the same question:
http://search-hadoop.com/m/uyzND1kM7q1gElhy1
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On T
-class.sh kafka.tools.ImportZkOffsets --input-file
/tmp/zk-offsets --zkconnect localhost:2181
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
Hmm, I guess you're right Tod :)
Just to confirm, you meant that, while you're changing the exported file it
might happen that one of the segment files becomes eligible for cleanup by
retention, which would then make the imported offsets out of range?
Marko Bonaći
Monitoring | Alerting | Anomaly
Spotify's image since it contains both ZK and Kafka, but I think
the latest version they built is 0.8.2.1, so you might have to build the
new image yourself if you need 0.9, but that's trivial to do.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr
Hi,
there was a problem with JMX consumer lag in 0.8:
http://search-hadoop.com/m/uyzND14v72215XZpK=Re+Consumer+lag+lies+orphaned+offsets+
Has anything changed now with 0.9?
Thanks
Did not know that quotas landed in 0.9. Very nice!
Being able to throttle clients that don't have real-time SLAs (in favor of
those who do) is a great addition.
Thanks for that Grant.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
case.
Perhaps you could check Consumer offsets from your Producer and then decide
based on that information whether to throttle Producer or not. Could get
complicated really fast, though.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Sup
AFAIK there is no such notion as maximum length of a topic, i.e. offset has
no limit, except Long.MAX_VALUE I think, which should be enough for a
couple of lifetimes (9 * 10E18, or quintillion or million trillions).
What would be the purpose of that, besides being a nice foot-gun :)
Marko Bonaći
running producers, brokers and consumers on AWS.
Is it possible that the network is that much flaky?
What's your experience?
Thanks,
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Con
g the same thing?
Or should I just forget about it and use the recommended approach from the
low-level consumer code example in the wiki (which I currently use as the
fallback)?
Thanks,
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Su
34 matches
Mail list logo
