Re: kafka connect questions

Hello Gwen, Thanks for the reply. My comments/answers inline. 1. Connectors that listen on sockets typically run in stand-alone mode, so they can tied to a specific machine (in distributed mode, connectors can move around). [Clay:] Even if the connectors move around, they can still listen to a

Re: kafka connect questions

I don't remember seeing one. There is no reason not to write one (let us know if you do, so we can put it on the connector hub!). Few things: 1. Connectors that listen on sockets typically run in stand-alone mode, so they can tied to a specific machine (in distributed mode, connectors can move

Re: [DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

Hey, Nice research and summary. Regarding the ability to have a "nameless" connector - I'm pretty sure we never intended to allow that. I'm confused about breaking something that currently works though - since we get NPE, how will giving more intentional exceptions break anything? Regarding the

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

Hi Matthias, Actually my intend was to provide to RichInitializer and later on we could provide the context of the record as you also mentioned. I remove that not to confuse the users. Regarding the RecordContext and ProcessorContext interfaces, I just realized the InternalProcessorContext class.

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

Hi Matthias, Sorry for long delay. Thanks for the comment. Good to know that the specified issue is not worth to break the backwards-compatibility. I fixed the KIP. Cheers, Jeyhun On Fri, Jun 30, 2017 at 1:38 AM Matthias J. Sax wrote: > Hi Jeyhun, > > thanks for

Re: RocksDB flushing issue on 0.10.2 streams

Upon another run, I see the same error occur during a rebalance, so either my log was showing a rebalance or there is a shared underlying issue with state stores. On Tue, Jul 4, 2017 at 11:35 AM, Greg Fodor wrote: > Also, I am on 0.10.2.1, so poll interval was already set to

Re: RocksDB flushing issue on 0.10.2 streams

Also, I am on 0.10.2.1, so poll interval was already set to MAX_VALUE. On Tue, Jul 4, 2017 at 11:28 AM, Greg Fodor wrote: > I've nuked the nodes this happened on, but the job had been running for > about 5-10 minutes across 5 nodes before this happened. Does the log show a >

Re: RocksDB flushing issue on 0.10.2 streams

I've nuked the nodes this happened on, but the job had been running for about 5-10 minutes across 5 nodes before this happened. Does the log show a rebalance was happening? It looks to me like the standby task was just committing as part of normal operations. On Tue, Jul 4, 2017 at 7:40 AM,

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Vahid, no we are not relying on parsing the current output. I just thought that keeping the full output isn't necessarily that bad as it shows some sort of history of how a group was used. ciao Edo -- Edoardo Comar IBM Message Hub IBM UK

kafka connect

Hello All, I have a few questions regarding kafka connect. I'd appreciate your replies. 1) Is there a kafka connector for listening to tcp sockets? 2) If there a protobuf converter that can deal with variable length messages? Meaning read the prefix that specifies the length and use the

Jenkins build is back to normal : kafka-trunk-jdk8 #1782

See

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Edo, Thanks for reviewing the KIP. Modifying the default behavior of `--describe` was suggested in the related JIRA. We could poll the community to see whether they go for that option, or, as you suggested, introducing a new `--only-xxx` ( can't also think of a proper name right now :) )

[jira] [Created] (KAFKA-5556) KafkaConsumer throws: java.lang.IllegalStateException: > Attempt to retrieve exception from future which hasn't failed

Damian Guy created KAFKA-5556: - Summary: KafkaConsumer throws: java.lang.IllegalStateException: > Attempt to retrieve exception from future which hasn't failed Key: KAFKA-5556 URL:

[jira] [Created] (KAFKA-5555) unable to set sasl Mechanism via KafkaLog4jAppender

Ahmed JEMAI created KAFKA-: -- Summary: unable to set sasl Mechanism via KafkaLog4jAppender Key: KAFKA- URL: https://issues.apache.org/jira/browse/KAFKA- Project: Kafka Issue Type:

kafka connect questions

Hello All, I'd appreciate your help with the following questions. 1) Is there kafka connect for listening to tcp sockets? 2) If, can the messages be in protobuf, with each messaged prefixed with the length of the message? thanks Clay

Re: RocksDB flushing issue on 0.10.2 streams

Hi Greg, Obviously a bit difficult to read the RocksDBException, but my guess is it is because the state directory gets deleted right before the flush happens: 2017-07-04 10:54:46,829 [myid:] - INFO [StreamThread-21:StateDirectory@213] - Deleting obsolete state directory 0_10 for task 0_10 Yes

[GitHub] kafka pull request #3487: MINOR: Fixed misleading reference to HTTPS instead...

GitHub user ppatierno opened a pull request: https://github.com/apache/kafka/pull/3487 MINOR: Fixed misleading reference to HTTPS instead of SSL support in the doc You can merge this pull request into a Git repository by running: $ git pull https://github.com/ppatierno/kafka

[GitHub] kafka pull request #3368: KAFKA-5469: Created state changelog topics not log...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3368 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3486: KAFKA-5554 doc config common

GitHub user tombentley opened a pull request: https://github.com/apache/kafka/pull/3486 KAFKA-5554 doc config common This is more a request for review than it is a request to merge, as I don't think this is complete yet. However, this illustrates the idea of drawing attention to

[jira] [Created] (KAFKA-5554) Hilight config settings for particular common use cases

Tom Bentley created KAFKA-5554: -- Summary: Hilight config settings for particular common use cases Key: KAFKA-5554 URL: https://issues.apache.org/jira/browse/KAFKA-5554 Project: Kafka Issue

[GitHub] kafka pull request #3485: KAFKA-5531: throw concrete exceptions in streams t...

GitHub user evis opened a pull request: https://github.com/apache/kafka/pull/3485 KAFKA-5531: throw concrete exceptions in streams tests 1. Now instead of just generic Exception methods declare more concrete exceptions throwing or don't declare any throwing at all, if not

[GitHub] kafka pull request #3484: MINOR: add IQ docs to streams documentation

GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/3484 MINOR: add IQ docs to streams documentation You can merge this pull request into a Git repository by running: $ git pull https://github.com/dguy/kafka iq-doc Alternatively you can review and

Build failed in Jenkins: kafka-trunk-jdk8 #1781

See Changes: [rajinisivaram] MINOR: Add test brokers to list as they are created to ensure cleanup -- [...truncated 4.85 MB...]

[GitHub] kafka pull request #3483: MINOR: Wait for tasks to terminate to avoid except...

GitHub user rajinisivaram opened a pull request: https://github.com/apache/kafka/pull/3483 MINOR: Wait for tasks to terminate to avoid exception in test teardown You can merge this pull request into a Git repository by running: $ git pull

[DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

While working on KAFKA-4930 and KAFKA-4938 I came across some sort of fundamental questions about the rest api for creating connectors in Kafka Connect that I'd like to put up for discussion. Currently requests that do not contain a "name" element on the top level are not accepted by the API, but

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Thanks Vahid, I like the KIP. One question - could we keep the current "--describe" behavior unchanged and introduce "--only-xxx" options to filter down the full output as you proposed ? ciao, Edo -- Edoardo Comar IBM Message Hub IBM UK Ltd,

[GitHub] kafka pull request #3471: MINOR: Add test brokers to list as they are create...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3471 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS]: KIP-161: streams record processing exception handlers

As part of the PR review we decided to add a metric to keep track of the number of skipped records due to deserialization. I updated the KIP to reflect that. Thanks Eno > On Jun 23, 2017, at 10:59 AM, Eno Thereska wrote: > > Done, thanks. I'll open a vote thread now. >

Opened PRs

Hi devs, I remember discussion about delayed PR resolution due to major release. Now, given that 0.11.0.0 released five days ago, it would be nice to know, when someone from Team will be able to review PRs? Some approximation like today, next week, next month, etc. would be good to have. :)

[GitHub] kafka pull request #3482: MINOR: Make topic config a section, with TOC entry...

GitHub user tombentley opened a pull request: https://github.com/apache/kafka/pull/3482 MINOR: Make topic config a section, with TOC entry. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tombentley/kafka minor-doc-topic-config

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Hi Jan, Thanks very much for the input. On Tue, 4 Jul 2017 at 08:54 Jan Filipiak wrote: > Hi Damian, > > I do see your point of something needs to change. But I fully agree with > Gouzhang when he says. > --- > > But since this is a incompatibility change, and we are

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Hi Damian, I do see your point of something needs to change. But I fully agree with Gouzhang when he says. --- But since this is a incompatibility change, and we are going to remove the compatibility annotations soon it means we only have one chance and we really have to make it right.

[jira] [Created] (KAFKA-5553) Delete topic failed to change from OnlineReplica to ReplicaDeletionStarted if ISR not created

Pengwei created KAFKA-5553: -- Summary: Delete topic failed to change from OnlineReplica to ReplicaDeletionStarted if ISR not created Key: KAFKA-5553 URL: https://issues.apache.org/jira/browse/KAFKA-5553