Re: [ANNOUNCE] New committer: Damian Guy

Congrats! ~ Joe Stein On Fri, Jun 9, 2017 at 6:49 PM, Neha Narkhede <n...@confluent.io> wrote: > Well deserved. Congratulations Damian! > > On Fri, Jun 9, 2017 at 1:34 PM Guozhang Wang <wangg...@gmail.com> wrote: > > > Hello all, > > > > > >

Re: [DISCUSS] Java 8 as a minimum requirement

and communicating their semantic version strategy. If JDK8 is so much desired then jump to 0.11 and only do bug fixes on the 0.10 release (which should be rigorous and not forceful to make folks upgrade unnecessarily to get such improvements). My 0.2824152382 cents. Regards, ~ Joe Stein On Fri

Heron Spouts & Bolts & Example for Apache Kafka

Topology Looking forward to continued contributes, if you haven't tried out Heron yet you should. Kafka is the wind to make Heron fly! /****** Joe Stein Elodina Inc http://www.elodina.net **/

Re: Kafka support needed

If you use Go you can use https://github.com/sclasen/event-shuttle which is nice choice in some cases because footprint, it uses boltdb which is like leveldb which is like embedded k/v ok NiFi is cool too https://nifi.apache.org/ So is bruce https://github.com/ifwe/bruce those are more out of

Re: Kafka Connect - Source Connector for Apache Cassandra

There is one being worked on here https://github.com/tuplejump/kafka-connect-cassandra/ On Wed, Apr 13, 2016 at 5:21 PM, Kaz Chehresa wrote: > Has anyone tried implementing or know whether it's possible to implement a > "Kafka Connect" source connector for apache Cassandra,

Re: Powered by Kafka

ok, you should be gt2g ~ Joe Stein - - - - - - - - - - - - - - - - - - - [image: Logo-Black.jpg] http://www.elodina.net http://www.stealth.ly - - - - - - - - - - - - - - - - - - - On Mon, Nov 30, 2015 at 1:05 PM, Andrew Schofield < andrew_schofi...@uk.ibm.com> wrote: >

Re: Powered by Kafka

or anything lots of info in there too for folks that are helpful. I think more services that folks know have kafka behind it gives it more reliability that it can do what the expectation for it are to-do. ~ Joe Stein On Mon, Nov 30, 2015 at 12:39 PM, Andrew Schofield < andrew_schofi...@uk.ibm.com>

Re: [kafka-clients] Re: 0.9.0.0 RC4

Thanks for everyone that contributed to this release! It has been a long time in the works with some really great new additions for folks waiting with excitement of the new consumer, security and connect (copycat) and everything else baked in. Thanks again! ~ Joe Stein

Re: One more Kafka Meetup hosted by LinkedIn in 2015 (this time in San Francisco) - does anyone want to talk?

They should all be on the user groups section of the confluence page https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations for which there were video. It might need some curating but that is where it has been going so far. ~ Joe Stein On Tue, Nov 3, 2015 at 4:48 PM

Re: Pushing from Pig to Kafka

best support for integrated systems... I suspect that integrated system will support Kafka too :) The Hadoop community has spent A LOT of time since 0.8 making all of that ecosystem more cohesive within that community. I urge support for it always. ~ Joe Stein - - - - - - - - - - - - - - - - - - -

Re: Broker restart, new producer and snappy in 0.8.2.1

so you will need to download into a local mvn and use 0.8.2.2 producer. I don't know it if will fix your problem but if you can reproduce it and see and if not post to a JIRA that would be great. Thanks! ~ Joe Stein - - - - - - - - - - - - - - - - - - - [image: Logo-Black.jpg] http

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Is this something also that should be in 0.8.2.2 https://issues.apache.org/jira/browse/KAFKA-2421 wasn't sure what the lz4 usage has been seems to break on some jdks. +1 on the snappy fixes ~ Joe Stein On Fri, Aug 14, 2015 at 5:39 PM, Guozhang Wang wangg...@gmail.com wrote: +1 for both KAFKA

Re: abstracting ZooKeeper

available in contrib repo we can link to. Other thoughts/comments happy to catch those and update the KIP. Thanks! ~ Joe Stein - - - - - - - - - - - - - - - - - - - [image: Logo-Black.jpg] http://www.elodina.net http://www.stealth.ly - - - - - - - - - - - - - - - - - - - On Sun, Aug 9

Re: Re: Closing socket connection to /192.115.190.61. (kafka.network.Processor)

yup, my bad I really thought we got that one in but it is on trunk so 0.8.3 it is ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, Jun 19, 2015 at 8:52 AM, Todd Palino tpal...@gmail.com wrote: I don't think this got changed until

Re: Closing socket connection to /192.115.190.61. (kafka.network.Processor)

What version of Kafka are you using? This was changed to debug level in 0.8.2. ~ Joestein On Jun 18, 2015 10:39 PM, bit1...@163.com bit1...@163.com wrote: Hi, I have started the kafka server as a backgroud process, however, the following INFO log appears on the console very 10 seconds. Looks

Re: Keeping Zookeeper and Kafka Server Up

+1 to using exhibitor. Besides managing the ensemble it also helps with backups and zk log cleanup (which if you don't do your machine will run out of space). ~ Joestein On Jun 17, 2015 9:44 AM, Dillian Murphey crackshotm...@gmail.com wrote: supervisord is pretty easy to use. Netflix Exhibitor

Re: KAFKA 0.9 release timeline

the discussion again maybe in a few weeks after folks have had a chance to test and try out all what is done/almost done. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, May 29, 2015 at 10:04 AM, Kuchibhotla, Murthy venkata.murthy.kuchibho

Re: kafkacat

Try out bruce https://github.com/ifwe/bruce it's a daemon listening socket producer, does exactly what you are looking for I think. ~ Joestein On May 19, 2015 7:05 AM, clay teahouse clayteaho...@gmail.com wrote: Thanks Magnus. I'll take a look at n2kafka. I have many data sources sending data

Re: Data replication and zero data loss

a enough replica to survive data loss guarantees. min.isr=2 min.sir=3 //depends on data Also take a look at https://github.com/stealthly/go_kafka_client/tree/master/mirrormaker it might be helpful for what you are looking for. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly

Re: Horizontally Scaling Kafka Consumers

the consumergroup API and accomplishes what I was looking for and I assume is implemented by keeping track of memberships w/ Zookeeper. Thank you for the information - it really helped clear up what I failing to understand with kafka. Nimi On Wed, Apr 29, 2015 at 10:10 PM, Joe Stein joe.st

Re: Horizontally Scaling Kafka Consumers

/0.9/comparisons/introduction.html which has a Mesos patch https://issues.apache.org/jira/browse/SAMZA-375 and built in YARN support. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, Apr 29, 2015 at 8:56 AM, David Corley davidcor

Re: Kafka elastic no downtime scalability

https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, Mar 13, 2015 at 3:05 PM, sunil kalva sambarc...@gmail.com wrote: Joe Well, I know it is semantic but right

Re: Kafka elastic no downtime scalability

partitions when increasing/decreasing the size of the cluster and some more goodies too. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, Mar 13, 2015 at 8:43 AM, Stevo Slavić ssla...@gmail.com wrote: Hello Apache Kafka community

Re: Kafka elastic no downtime scalability

with AWS CloudFormation and internal systems they built too. So, it can be done... you just have todo it. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, Mar 13, 2015 at 12:39 PM, Stevo Slavić ssla...@gmail.com wrote: OK, thanks

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2

Thanks Jun for getting this release out the door and everyone that contributed to the work in 0.8.2.1, awesome! ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Mon, Mar 9, 2015 at 2:12 PM, Jun Rao j...@confluent.io wrote

Re: framework to load streamed data from kafka into relational database

requirements or were looking for something higher level like Sqoop or NiFi (neither which I think support both yet) or something. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Mar 10, 2015 at 3:14 AM, Vadim Keylis vkeylis2...@gmail.com

Re: New Errors in 0.8.2 Protocol

everyone needs fluidly enough. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Wed, Mar 4, 2015 at 12:44 PM, Evan Huus evan.h...@shopify.com wrote: Hey all, it seems that 0.8.2 has added a handful more errors to the protocol which

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2

) at kafka.api.ProducerFailureHandlingTest.testNotEnoughReplicasAfterBrokerShutdown(ProducerFailureHandlingTest.scala:355) This happens to me all the time on a few different machines. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Mon, Mar 2, 2015 at 7:36 PM, Jun Rao j...@confluent.io wrote: +1

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2

Ok, lets fix the transient test failure on trunk agreed not a blocker. +1 quick start passed, verified artifacts, updates in scala https://github.com/stealthly/scala-kafka/tree/0.8.2.1 and go https://github.com/stealthly/go_kafka_client/tree/0.8.2.1 look good ~ Joe Stein

Re: can we send pdf, image files in kafka queue

It can be done, sure. We built a prototype a while back https://github.com/stealthly/f2k though I can't say I have bumped into a use case where the tradeoffs worked to do it. Chunking the file in each message and reconstructing it is going to be an overhead or your going to block on waiting to

Re: Anyone using log4j Appender for Kafka?

Sounds like https://issues.apache.org/jira/browse/KAFKA-1788 maybe On Feb 24, 2015 2:28 PM, Scott Chapman sc...@woofplanet.com wrote: Yea, however I don't get async behavior. When kafka is down the log blocks, which is kinda nasty to my app. On Tue Feb 24 2015 at 2:27:09 PM Joe Stein joe.st

Re: Anyone using log4j Appender for Kafka?

and changed it as things came up... e.g. setSource(), setTags() blah blah... Paul Otto has an open source version https://github.com/potto007/kafka-appender-layout that you could try out too that he built to tackle some of the layout things. ~ Joe Stein - - - - - - - - - - - - - - - - - http

Re: Anyone using log4j Appender for Kafka?

and kafka too :) On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein joe.st...@stealth.ly wrote: are you including https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in your project? ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly

Re: Anyone using log4j Appender for Kafka?

are you including https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in your project? ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Tue, Feb 24, 2015 at 2:02 PM, Scott Chapman sc...@woofplanet.com wrote: Yea

Re: Anyone using log4j Appender for Kafka?

Producer type isn't needed anymore with the new producer so in the the logger properties just leave that out in 0.8.2 and it should work. On Tue, Feb 24, 2015 at 2:24 PM, Joe Stein joe.st...@stealth.ly wrote: Interesting, looks like a breaking change from 0.8.1 https://github.com/apache/kafka

Re: Anyone using log4j Appender for Kafka?

PM, Joe Stein joe.st...@stealth.ly wrote: and kafka too :) On Tue, Feb 24, 2015 at 2:21 PM, Joe Stein joe.st...@stealth.ly wrote: are you including https://github.com/stealthly/scala-kafka/blob/master/build.gradle#L122 in your project? ~ Joe Stein

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 1

Jun, Can we also add https://issues.apache.org/jira/browse/KAFKA-1724 to the next RC please? Thanks! ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Sun, Feb 22, 2015 at 11:59 AM, Jun Rao j...@confluent.io wrote: We identified

Re: Kafka on Mesos?

We pushed first bits for the kafka framework we are working on to https://github.com/mesos/kafka. We have been using Aurora and Marathon to run Kafka on Mesos but are cutting over to a framework approach as described in the ticket over the next 5 weeks. ~ Joe Stein

Re: [VOTE] 0.8.2.1 Candidate 1

-1729 please. +1 (binding) ~ Joe Stein On Fri, Feb 20, 2015 at 9:08 PM, ted won realtime...@gmail.com wrote: +1 On Friday, February 20, 2015, Guozhang Wang wangg...@gmail.com wrote: +1 binding. Checked the md5, and quick start. Some minor comments: 1. The quickstart section would

Re: Kafka 0.8.2 New Producer Example

in scala https://github.com/stealthly/scala-kafka/blob/master/src/test/scala/KafkaSpec.scala#L146-L168 ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Sat, Feb 21, 2015 at 8:10 AM, Cdhar B trycd...@gmail.com wrote: Hi, Could you

Re: still too many files open errors for kafka web cosole

The Apache project doesn't have a web console for kafka. Have you taken a look at https://github.com/yahoo/kafka-manager as of yet? I haven't myself hoping to get sometime tonight/this weekend to-do so. ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly

Re: rebalancing leadership quite often

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whypartitionleadersmigratethemselvessometimes ? ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, Feb 20, 2015 at 7:19 PM, Sa Li sal...@gmail.com wrote: Hi, All My dev

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Thanks Harsha, looks good so far. How were you thinking of running the KerberosTicketManager as a standalone process or like controller or is it a layer of code that does the plumbing pieces everywhere? ~ Joestein On Wed, Feb 11, 2015 at 12:18 PM, Harsha ka...@harsha.io wrote: Hi, Here is

Re: Ship Kafka in on prem product

Cool!!! Maybe on the eco system confluence page we should have a packaging and/or distro section ~ Joestein On Wed, Feb 11, 2015 at 1:34 PM, David Morales dmora...@stratio.com wrote: Regarding RPM/DEB packages, we, at stratio.com, have a public repository which includes RPM and dev

Re: java API equivalent of --from-beginning?

You have to delete the znode from zookeeper for your consumer group https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/consumer/ConsoleConsumer.scala#L181 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC

Re: Kafka Architecture diagram

Ankur, There is more from papers and presentations you can check out too https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations if you haven't already. - Joestein On Thu, Feb 5, 2015 at 12:57 PM, Conikee coni...@gmail.com wrote: Michael Noll's blog posting might

Re: How to configure asynchronous replication

too (same line --partitions). /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed

Re: New Producer - ONLY sync mode?

much sync vs async but my crystal ball just back from the shop so well see =8^) /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 3

Huzzah! Thanks Jun for preparing the release candidates and getting this out to the community. - Joe Stein On Mon, Feb 2, 2015 at 2:27 PM, Jun Rao j...@confluent.io wrote: The following are the results of the votes. +1 binding = 3 votes +1 non-binding = 1 votes -1 = 0 votes 0 = 0 votes

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 3

+1 (binding) verified signatures, ran quick start, tests all passed. - Joe Stein On Fri, Jan 30, 2015 at 12:04 PM, Jun Rao j...@confluent.io wrote: This is a reminder that the vote will close tomorrow night. Please test RC3 out and vote before the deadline. Thanks, Jun On Wed, Jan 28

Re: Poll: Producer/Consumer impl/language you use?

is taking a look at that now unless someone is already working on that? We are using the partition static assignment in the new consumer and just using Mesos for handling re-balance for us. When he gets further along and if it makes sense we will shoot a KIP around people can chat about it on dev. - Joe

Re: Regarding Kafka release 0.8.2-beta

Matve wr should add to the documentation experimental so folks that don't know understand. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

build, topic deletion, compaction and random testing On Mon, Jan 26, 2015 at 6:14 AM, Neha Narkhede n...@confluent.io wrote: +1 (binding) Verified keys, quick start, unit tests. On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein joe.st...@stealth.ly wrote: That makes sense, thanks! On Sat

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

ask that question right? -Jay On Fri, Jan 23, 2015 at 9:06 PM, Joe Stein joe.st...@stealth.ly wrote: Should value.serializer in the new java producer be defaulted to Array[Byte] ? I was working on testing some upgrade paths and got this ! return exception in callback when buffer

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

is something you basically have to think about to use Kafka and trying to guess just makes things worse. -Jay On Sat, Jan 24, 2015 at 2:51 PM, Joe Stein joe.st...@stealth.ly wrote: Maybe. I think the StringSerialzer could look more like a typical type of message. Instead of encoding being

Re: Help: Kafka LeaderNotAvailableException

Vishal, Does this error happen every time you are sending? Or just the first time? Joe Stein On Thu, Jan 22, 2015 at 4:33 AM, vishal_ja...@dell.com wrote: Hi, Let me overview on the issue that I am facing on producing message in Kafka: I have horthonworks HDP-2.1 installed, along that we

Re: Kafka Producers and Virtual Kafka Endpoint

(through a metadatarequest) and their current ownership. More on that here https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Partitioningandbootstrapping if you are interested. /*** Joe Stein Founder

Re: Isr difference between Metadata Response vs /kafka-topics.sh --describe

Sounds like you are bumping into this https://issues.apache.org/jira/browse/KAFKA-1367 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: Can I run Kafka Producer in BG and then run a program that outputs to the console?

This is a stdin/stdout producer/consumer that works great for (what I think) you are trying to-do https://github.com/edenhill/kafkacat /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: Issue size message

*fetch.message.max.bytes* in your consumers properties (whoever that might be configured for your specific consumer being used) so that they can read that data otherwise you won't see messages downstream. /*** Joe Stein Founder, Principal Consultant Big

Re: Consumer questions

in zk long term. It is all just 1s and 0s and just a matter of how many you put together yourself vs take out of the box that are given too you =8^) /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly

Re: dumping JMX data

-statsd-metrics2 and sending to https://github.com/kamon-io/docker-grafana-graphite you can quickly chart and see everything going on. There are also software as a service options too. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security

Re: [VOTE] 0.8.2.0 Candidate 1

I think that is a change of behavior that organizations may get burned on. Right now there is no delete data feature. If an operations teams upgrades to 0.8.2 and someone decides to delete a topic then there will be data loss. The organization may not have wanted that to happen. I would argue to

Re: kafka cluster on aws

/master/infrastructure/aws/bastion) and supervisor (https://github.com/stealthly/minotaur/tree/master/supervisor) and after that it is really nice and easy. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly

Re: metric-kafka problems

this help, thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Fri, Jan 9, 2015

Re: kafka monitoring

You need to export the JMX_PORT for kafka to use on startup before starting up /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: Consumer and offset management support in 0.8.2 and 0.9

^) /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Jan 7, 2015 at 1:04 PM, Jay Kreps j...@confluent.io wrote: Hey

Re: no space left error

/hsperfdata_root/19721 I don't think this is Kafka related though so what we are talking about partition and retention is not applicable, even though important. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: messages lost

counting the message from each partition using https://github.com/edenhill/kafkacat and pipe to wc -l it makes for a nice simple sanity check to where the problem might be. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: Kafka getMetadata api

/ConsoleConsumer.scala#L196 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: kafka logs gone after reboot the server

That is because your logs are in /tmp which you can change by setting log.dirs to something else. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com

Re: kafka logs gone after reboot the server

they you should probably have it be consistent on each node. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: kafka-web-console error

is the default likely for your OS. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: Kafka getMetadata api

. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Fri, Jan 2, 2015 at 8:25 PM, Mukesh Jha me.mukesh@gmail.com

Re: under replicated topics

at as such... bursty traffic (especially with batched messages) can upset this, yup. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: csv files to kafka

it works right from the repo better without having to-do much local for others so they don't bump into this and then work around it locally by installing protobuf. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: Kafka in C#

Another option is a HTTP wrapper around the actual producer and doing a HTTP POST from C# to a REST server e.g. https://github.com/stealthly/dropwizard-kafka-http which I know folks have done successfully. /*** Joe Stein Founder, Principal Consultant Big

Re: Kafka in C#

Thunder, can you add that to https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-.net didn't know it existed but cool that it uses Rx On Mon, Dec 22, 2014 at 12:52 PM, Thunder Stumpges tstump...@ntent.com wrote: Hi there, We looked at both of these a while back and ended up

Re: Max. storage for Kafka and impact

see some comments inline On Fri, Dec 19, 2014 at 11:30 AM, Achanta Vamsi Subhash achanta.va...@flipkart.com wrote: We require: - many topics - ordering of messages for every topic Ordering is only on a per partition basis so you might have to pick a partition key that makes sense for what

Re: Max. storage for Kafka and impact

and commit your offset in batch post processing. Then you only need more partitions to fetch batches to process in parallel. Regards, Joestein On Fri, Dec 19, 2014 at 12:01 PM, Joe Stein joe.st...@stealth.ly wrote: see some comments inline On Fri, Dec 19, 2014 at 11:30 AM, Achanta Vamsi Subhash

Re: Better exception handling in kafka.producer.async.DefaultEventHandler

I would suggest to use the new client java producer in 0.8.2-beta http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html it handles the case you brought up (among lots of other goodies). /*** Joe Stein Founder, Principal

Re: csv files to kafka

program, or you can use as reference and rewrite it in Java. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

Re: spark kafka batch integration

for? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Sun, Dec 14, 2014 at 8:22 PM, Koert Kuipers ko...@tresata.com wrote: hello all

Re: how to achieve availability with no data loss when replica = 1?

that no messages will be lost as long as at least one in sync replica remains. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop

New Go Kafka Client

Hi, we open sourced a new Go Kafka Client http://github.com/stealthly/go_kafka_client some more info on blog post http://allthingshadoop.com/2014/12/10/making-big-data-go/ for those working with Go or looking to get into Go and Apache Kafka. /*** Joe Stein

Re: how to achieve availability with no data loss when replica = 1?

, and the topic replica is set to 1, the whole system is still available and the data would go to other partitions without loss. THANKS again. On Thu, Dec 11, 2014 at 12:37 AM, Joe Stein joe.st...@stealth.ly wrote: If you want no data loss then you need to set ack = -1 Copied from https

Re: Kafka Issue [Corrupted broker]

. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Dec 9, 2014 at 2:54 AM, ashendra bansal ashendraban...@gmail.com wrote: Hi

Re: unable to create topic

Cloudera Manager utilizes the chroot znode structures so you should connect with --zookeeper localhost:2181/kafka Or whatever value CM has set for the chroot path of your installation. /*** Joe Stein Founder, Principal Consultant Big Data Open Source

Re: Trying to start Kafka

That is for a previous version. This is for the 0.8.2-beta quick start https://kafka.apache.org/082/documentation.html#quickstart /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: Which consumer to use with Kafka 0.8.2-beta

The new consumer client is slated for 0.9.0 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http

Re: Kafka 0.8.2 release - before Santa Claus?

If we can have a build ready by Dec 26th I think that is feasible. I could prepare and post that if we think we have the votes and a stable version. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter

Re: [DISCUSSION] adding the serializer api back to the new java producer

in easier with a build/binaries to download and use from maven also. +1 on the change now prior to the 0.8.2 release. - Joe Stein On Mon, Nov 24, 2014 at 11:43 PM, Sriram Subramanian srsubraman...@linkedin.com.invalid wrote: Looked at the patch. +1 from me. On 11/24/14 8:29 PM, Gwen Shapira

Re: Is there a plan to build a ubiquitous web service API to manage the kafka cluster

it however they want. Work for that has already gotten underway parent ticket https://issues.apache.org/jira/browse/KAFKA-1694 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http

Re: Partition Key Cannot be Send Out by Producer

iI would helpful to see the full stack trace. Also, how have you instantiated your Producer class? Did you set a value for serializer.class in the property? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: Partition Key Cannot be Send Out by Producer

^) /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Thu, Nov 20, 2014 at 8:42 PM, Haoming

Re: Elastsic Scaling

currently (as per the wire protocol https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol ) it is all about how the consumer is designed and developed. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

Re: Elastsic Scaling

. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Thu, Nov 20, 2014 at 11:57 PM, Daniel Compton d...@danielcompton.net wrote

Re: Elastsic Scaling

Meant to say burst 1,000,000 messages per second on those X partitions for 10 minutes On Fri, Nov 21, 2014 at 12:12 AM, Joe Stein joe.st...@stealth.ly wrote: You need to be thoughtful about adding more partitions. This is paramount if you are doing semantic partitioning in which case adding

Re: selecting java producer (0.8.2 or 0.8.1.1?)

0.8.2-beta java doc https://archive.apache.org/dist/kafka/0.8.2-beta/java-doc/ /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: powered by kafka

Hey, I added you to the power by page and also ekaf to the Client Library page too https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Erlang /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http

  1   2   3   4   >