Re: Poll RESULTS: Producer/Consumer languages

Thanks for the answers I'll be looking for both to see what fits best for our needs :) On Thu, Feb 12, 2015 at 12:37 PM, Colin wrote: > Digital River will be releasing one soon as well that is integrated with the > netflix stack for discovery, load balancing, metrics,etc. > > -- > Colin Clark >

Re: Poll RESULTS: Producer/Consumer languages

Digital River will be releasing one soon as well that is integrated with the netflix stack for discovery, load balancing, metrics,etc. -- Colin Clark +1 612 859 6129 Skype colin.p.clark > On Feb 11, 2015, at 7:20 PM, Jay Kreps wrote: > > Hey Justin, > > I don't think LinkedIn is, but Conflue

Re: Producing message set

Looking at the code, it does look like appending a message set to a partition will succeed or fail as a whole. You can look at Log.append() for the exact logic if you are interested. Gwen On Wed, Feb 11, 2015 at 6:13 AM, Piotr Husiatyński wrote: > Hi, > > I'm writing new client library for kaf

Kafka C++ Windows Client for 0.8 Releases

Hi Tianning Zhang , Did you make this Kafka C++ native client library for windows public? I would like to use your library. Thanks Vindhyavasini

Re: Poll RESULTS: Producer/Consumer languages

Hey Justin, I don't think LinkedIn is, but Confluent has made a pretty complete producer and consumer REST proxy that we will be releasing quite soon. -Jay On Wed, Feb 11, 2015 at 2:40 PM, Justin Maltat wrote: > Hi, > > Is Linkedin planning to release its REST proxy server as an official > rel

Re: understanding partition key

Partition key is on producer level, that if you have multiple partitions for a single topic, then you can pass in a key for the KeyedMessage object, and base on different partition.class, it will return a partition number for the producer, and producer will find the leader for that partition.I d

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Looks good. Thanks for working on this. One note, the Channel implementation from SSL only works on Java7 and up. Since we are still supporting Java 6, I'm working on a lighter wrapper that will be a composite on SocketChannel but will not extend it. Perhaps you'll want to use that. Looking forwa

Re: Poll RESULTS: Producer/Consumer languages

Hi, Is Linkedin planning to release its REST proxy server as an official release? This would be quite interesting for using Kafka in some existing heterogeneous environment. Justin On Thu, Jan 29, 2015 at 3:58 PM, Guozhang Wang wrote: > Thanks for sharing this Otis, I am also quite surprised

understanding partition key

I'm trying to understand how the partition key works and whether I need to specify a partition key for my topics or not. What happens if I don't specify a PK and I have more than one consumer that wants all messages in a topic for a certain period of time? Will those consumers get all the message

Re: Ship Kafka in on prem product

+1. Although... would packages built and published by a third party be considered trustworthy enough by others? Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Wed, Feb 11, 2015 at 1:45 PM, Joe Stein wrote:

Re: Ship Kafka in on prem product

That would be great... we will work on the new kafka version ASAP. Regards. 2015-02-11 19:45 GMT+01:00 Joe Stein : > Cool!!! > > Maybe on the eco system confluence page we should have a packaging and/or > distro section > > ~ Joestein > > On Wed, Feb 11, 2015 at 1:34 PM, David Morales > wr

Re: Ship Kafka in on prem product

Cool!!! Maybe on the eco system confluence page we should have a packaging and/or distro section ~ Joestein On Wed, Feb 11, 2015 at 1:34 PM, David Morales wrote: > Regarding RPM/DEB packages, we, at stratio.com, have a public repository > which includes RPM and dev packages > > http://repo

Re: Ship Kafka in on prem product

Regarding RPM/DEB packages, we, at stratio.com, have a public repository which includes RPM and dev packages http://repository.stratio.com/sds/1.1/ubuntu/13.10/binary/ http://repository.stratio.com/sds/1.1/RHEL/ Regards. 2015-02-11 19:22 GMT+01:00 Otis Gospodnetic : > Hi, > > For what it's

Re: Ship Kafka in on prem product

Hi, For what it's worth, we include Kafka (now 0.8.2) in our On Premises version of SPM and Logsene. Kafka is used to to transport metrics and logs there. I do seem to recall some not-so-elegant stuff we had to do around packaging, though. I think because http://kafka.apache.org/downloads.ht

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Thanks Joe. It will be part of KafkaServer and will run on its own thread. Since each kafka server will run with a keytab we should make sure they are all getting renewed. On Wed, Feb 11, 2015, at 10:00 AM, Joe Stein wrote: > Thanks Harsha, looks good so far. How were you thinking of running > the

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Thanks Harsha, looks good so far. How were you thinking of running the KerberosTicketManager as a standalone process or like controller or is it a layer of code that does the plumbing pieces everywhere? ~ Joestein On Wed, Feb 11, 2015 at 12:18 PM, Harsha wrote: > Hi, > Here is the initial p

ANNOUNCE: Kafka 0.8.2 Monitoring in SPM

Hi, I'm super happy to announce we've added Kafka 0.8.2 monitoring support to SPM. It looks like we capture over 100 Kafka metrics (yes, including that Consumer Lag everyone always wants to see!). Details + charts in: http://blog.sematext.com/2015/02/10/kafka-0-8-2-monitoring/ Thanks to the Ka

Fwd: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi, Here is the initial proposal for sasl/kerberos implementation for kafka https://cwiki.apache.org/confluence/x/YI4WAw and JIRA https://issues.apache.org/jira/browse/KAFKA-1686. I am currently working on prototype which will add more details to the KIP. Just opening the thread to say the

[DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi, Here is the initial proposal for sasl/kerberos implementation for kafka https://cwiki.apache.org/confluence/x/YI4WAw and JIRA https://issues.apache.org/jira/browse/KAFKA-1686. I am currently working on prototype which will add more details to the KIP. Just opening the thread to say the

Producing message set

Hi, I'm writing new client library for kafka and I cannot find information about atomicity of message set publishing. I'm implementing protocol described on wiki [1]. According to documentation, it is possible to send within single request more than one message - message set for any partition. In

Re: createMessageStreams vs createMessageStreamsByFilter

The new consumer will be released in 0.9, which is targeted for end of this quarter. On Tue, Feb 10, 2015 at 7:11 PM, tao xiao wrote: > Do you know when the new consumer API will be publicly available? > > On Wed, Feb 11, 2015 at 10:43 AM, Guozhang Wang > wrote: > > > Yes, it can get stuck. For

Re: Poor performance consuming multiple topics

Thanks again for the help. As a final follow-up we added a fetch.wait.max.ms of 10ms and that got us to a throughput of about 800kb/sec reading from the 20 topics, which is still a bit lower than I'd originally hoped, but should be sufficient for our use case in the near term. Thank you for all

Re: Java APIs - which one to use?

You should use the producer under o.a.k.c. The new consumer implementation is not available in 0.8.2 (although the APIs are there) so you would need to use the kafka.javaapi classes for the consumer. We plan to deprecate kafka.javaapi eventually. Thanks, Joel On Wed, Feb 11, 2015 at 11:42:04AM +

Java APIs - which one to use?

Hi, I'm trying out Kafka for a Java-centric project, and to that end I have downloaded the 0.8.2 source. I find both the kafka.javaapi.* classes (in the core project) and the org.apache.kafka.clients.* classes (in the clients project). I need to use the low-level API, and that seems do-able wit