Re: Cannot access Kafka Streams JMX metrics using jmxterm

I figured out the issue. Try with following info -b kafka.consumer:type=*,client-id==* So type before client id works. Thanks Sachin On 30 Jan 2017 11:22 p.m., "Guozhang Wang" wrote: > Hi Jendrik, > > Could you share with us what is your specified application id and client > id? > > The reaso

Re: Using KafkaConsumer to consume the internal office __consumer_offsets in Kafka 10, using Java

Yes, it is similar to console consumer. Just pass exclude.internal.topics=false config to KafkaConsumer configs. On Wed, Feb 1, 2017 at 9:39 PM, Wang, Howard wrote: > Hi, > > I have a use case where I need to programmatically consume the internal > office __consumer_offsets in Kafka 10 for dat

Re: Anyone have guide for setting up the rest proxy using SSL and Client authentication

so the server is doing a forward instead of a redirect so chrome client *should* be safe would a self-signed cert be flagged as "non-secure connection" in Chrome https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/2LXKVWYkOus%5B1-25%5D

Re: "End of Batch" event

But you cant delete them from the local store like this... you need to process tombstone to get them deleted from there. The idea about the design is, to compute those tombstone an inject them into the source topics. -Matthias On 2/1/17 3:34 PM, Gwen Shapira wrote: > I'm wondering why it has to b

[VOTE] 0.10.2.0 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 0.10.2.0. This is a minor version release of Apache Kafka. It includes 19 new KIPs. See the release notes and release plan ( https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+

Re: "End of Batch" event

I'm wondering why it has to be so complex... Kafka can be configured to delete items older than 24h in a topic. So if you want to get rid of records that did not arrive in the last 24h, just configure the topic accordingly? On Wed, Feb 1, 2017 at 2:37 PM, Matthias J. Sax wrote: > Understood now.

Re: KAFKA Support

Dear Lincu, You have reached the Apache Kafka mailing list. We are an open source community of volunteers, helping each other use Kafka to the best of our abilities. There are several companies that provide paid support for Kafka - including, but not limited to Confluent, Cloudera, and Hortonwork

Re: Kafka Streams punctuate with slow-changing KTables

I am not sure if I can follow... what do you mean by "find or create" semantics? What do you mean by "my first pass processor"? -Matthias On 2/1/17 12:24 PM, Elliot Crosby-McCullough wrote: > Ah, bueno! > > The first use case I have is actually to split a raw event stream into > facts and dime

Re: "End of Batch" event

Understood now. It's a tricky problem you have, and the only solution I can come up with is quite complex -- maybe anybody else has a better idea? Honestly, I am not sure if this will work: For my proposal, the source ID must be part of the key of your records to distinguish records from differen

kafka 0.8 producer when zookeeper 3.4.6 is shut down

I've been reading many posts describing the behavior of the Kafka Consumer when the Zookeeper connection is suddenly dropped. I could definitely use some clarification. The Kafka Consumer accepts a Zookeeper connection string made of a set of zookeeper nodes, for example "zk1a:2181,zk1b:2181,zk1c

RE: Anyone have guide for setting up the rest proxy using SSL and Client authentication

The MSFT Certificate Server does not have a product name per se. We have an internal Certificate Server in our AD infrastructure. In this use case I just generate a key pair on the UNIX side in a JKS. Then create a Certificate Signing Request (CSR) from that key. Then I sign it with our internal

Re: kafka-streams ktable recovery after rebalance crash

Not sure why the locks on the state directory got not release (maybe because of the crash) -- what version do you use? We fixed a couple of bug with this regard lately -- maybe it's fixed in upcoming 0.10.2 For now, you might want to delete the whole state directory (either manually or by calling

Re: Anyone have guide for setting up the rest proxy using SSL and Client authentication

Unfortunately im more of a nix guy I take it windows has a CA server which will parse cert and lookup attrs (CN?) to name-server? what is the Microsoft CA server called? if cert is parsed and keys match which name-server do you use to lookup attrs (CN?) in that scenario? Merci Eugene Marti

Re: Kafka Connect in different nodes than Kafka.

I was asking that exactly. Thank you ;) 2017-02-01 20:39 GMT+01:00 Hans Jespersen : > If you are asking if technically does Kafka Connect need the entire Apache > Kafka distribution to run then, then the answer is no, it does not because > Connectors just remotely connect to Kafka Brokers on sepa

Re: Kafka Streams punctuate with slow-changing KTables

Ah, bueno! The first use case I have is actually to split a raw event stream into facts and dimensions, but it looks like this is still the right solution as interactive queries can be done against it for "find or create" semantics. The KIP mentions that the GlobalKTable "will only be used for do

kafka-consumer-offset-checker complaining about NoNode for X in zk

Hi, For a subset of our topics we get this error: $KAFKA_HOME/bin/kafka-consumer-offset-checker.sh --group argyle-streams --topic topic_name --zookeeper $ZOOKEEPERS [2017-02-01 12:08:56,115] WARN WARNING: ConsumerOffsetChecker is deprecated and will be dropped in releases following 0.9.0. Use C

kafka-streams ktable recovery after rebalance crash

Hi, My kafka-streams application crashed due to a rebalance event (seems like I need to increase max.poll.interval.ms even more!) and then when I restarted the app I noticed existing rocksdb files were gone and while the rest of the pipeline was processing the part dealing with ktable was sitti

Re: Kafka Connect in different nodes than Kafka.

If you are asking if technically does Kafka Connect need the entire Apache Kafka distribution to run then, then the answer is no, it does not because Connectors just remotely connect to Kafka Brokers on separate machines. If you are asking if there is a separate distribution for a “connect node”

RE: Anyone have guide for setting up the rest proxy using SSL and Client authentication

Not really. I have managed to generate a cert with a single common CN, sign that with my internal CA and import that to the keystore that is referenced by the Kafka-Rest Proxy. I then distribute that Java Keystore (JKS) to all of the servers running the Rest API. The last part is to export that

RE: Kafka Connect in different nodes than Kafka.

Yes, you just need to point it to your cluster. -Dave -Original Message- From: Guillermo Ortiz [mailto:konstt2...@gmail.com] Sent: Wednesday, February 1, 2017 1:09 PM To: users@kafka.apache.org Subject: Kafka Connect in different nodes than Kafka. Is it possible to use Kafka Connect in n

Re: Kafka Streams punctuate with slow-changing KTables

Thanks! About your example: in upcoming 0.10.2 release we will have KStream-GlobalKTable join that is designed for exactly your use case to enrich a "fact stream" with dimension tables. If you use this feature, "stream time" will not depend on the GlobalKTables (but only only the "main topology")

Kafka Connect in different nodes than Kafka.

Is it possible to use Kafka Connect in nodes where isn't Kafka installed? I can't see on the documentation anything about installed in others nodes.

Re: Anyone have guide for setting up the rest proxy using SSL and Client authentication

probably fighting an uphill battle sending cleartext POSTS and GETS with REST calls most apache server folk who secure HTTP 1.1 server will front end with CA (provided by thawte,verisign,GeoTrust) https://www.geotrust.com [https://seal.geotrust.com/getgeotrustsslseal?at=0&sealid=1&dn=www.geot

Re: Kafka Streams punctuate with slow-changing KTables

Yes it does make sense, thank you, that's what I thought the behaviour was. I think having the option of triggering at least every n real-time seconds (or whatever period) would be very useful, as I can see a number of situations where the time between updates to some tables might be very infreque

Re: Kafka docs for current trunk

+1 In particular, this would help when there is a doc change submitted to trunk which is applicable to the currently released version. It would help the change get out there faster. -James > On Feb 1, 2017, at 9:03 AM, Guozhang Wang wrote: > > +1 > > Guozhang > > > On Wed, Feb 1, 2017 at

Anyone have guide for setting up the rest proxy using SSL and Client authentication

We are looking at the Rest Proxy for our environment but we want to secure connections to the Rest proxy via SSL AND Client Certificate authentication. The SSL part is no big deal but I am struggling mightly with the client authentication portion. Does someone have some details on how to proper

Re: Kafka Streams punctuate with slow-changing KTables

One thing to add: There are plans/ideas to change punctuate() semantics to "system time" instead of "stream time". Would this be helpful for your use case? -Matthias On 2/1/17 9:41 AM, Matthias J. Sax wrote: > Yes and no. > > It does not depend on the number of tuples but on the timestamps of

Re: Kafka Streams punctuate with slow-changing KTables

Yes and no. It does not depend on the number of tuples but on the timestamps of the tuples. I would assume, that records in the high volume stream have timestamps that are only a few milliseconds from each other, while for the low volume KTable, record have timestamp differences that are much big

Re: [DISCUSS] KIP-114: KTable materialization and improved semantics

Thanks everyone! I think it's time to do a V2 on the KIP so I'll do that and we can see how it looks and continue the discussion from there. Stay tuned. Thanks Eno > On 30 Jan 2017, at 17:23, Matthias J. Sax wrote: > > Hi, > > I think Eno's separation is very clear and helpful. In order to >

Re: Kafka docs for current trunk

+1 Guozhang On Wed, Feb 1, 2017 at 1:30 AM, Damian Guy wrote: > +1 > > On Wed, 1 Feb 2017 at 07:48 Michael Noll wrote: > > > Thanks for bringing this up, Matthias. > > > > +1 > > > > On Wed, Feb 1, 2017 at 8:15 AM, Gwen Shapira wrote: > > > > > +1 > > > > > > On Tue, Jan 31, 2017 at 5:57 PM,

KAFKA Support

Hi Team, We are planning to use KAFKA for our applications, can you please provide me the list of support model with cost per year? Thanks Lincu

Using KafkaConsumer to consume the internal office __consumer_offsets in Kafka 10, using Java

Hi, I have a use case where I need to programmatically consume the internal office __consumer_offsets in Kafka 10 for data recovery. I am able to use the console consumer to consume the internal topic. But I wonder if I can use KafkaConsumer class from my Java program to do the same thing. A

Re: Kafka Connect in different hosts than Kafka

In production it is recommended that you install Kafka Connect on a separate machine from the Kafka Brokers because brokers are recommended to run by themselves on dedicated machines. Connect can be on the same machine(s) as the target source or sink (Apache Ignite in your case) or somewhere els

Kafka Streams punctuate with slow-changing KTables

Hi there, I've been reading through the Kafka Streams documentation and there seems to be a tricky limitation that I'd like to make sure I've understood correctly. The docs[1] talk about the `punctuate` callback being based on stream time and that all incoming partitions of all incoming topics mu

Kafka Connect in different hosts than Kafka

Hello, I'm going to use Kafka Connect (with Apache Ignite). I guess that it's possible to install Kafka-Connect in others machines different than Kafka, is it recommended? Are you going to lose some data locallity? is it enough smart Kafka-connect to get this data locallity? how much resources doe

Re: Kafka docs for current trunk

+1 On Wed, 1 Feb 2017 at 07:48 Michael Noll wrote: > Thanks for bringing this up, Matthias. > > +1 > > On Wed, Feb 1, 2017 at 8:15 AM, Gwen Shapira wrote: > > > +1 > > > > On Tue, Jan 31, 2017 at 5:57 PM, Matthias J. Sax > > wrote: > > > Hi, > > > > > > I want to collect feedback about the ide