[jira] [Created] (KAFKA-4678) Create separate page for Connect docs

2017-01-20 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4678: -- Summary: Create separate page for Connect docs Key: KAFKA-4678 URL: https://issues.apache.org/jira/browse/KAFKA-4678 Project: Kafka Issue Type

[GitHub] kafka pull request #2374: KAFKA-3209: KIP-66: more single message transforms

2017-01-13 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2374 KAFKA-3209: KIP-66: more single message transforms WIP, in this PR I'd also like to add doc generation for transformations. You can merge this pull request into a Git repository by running

[GitHub] kafka pull request #2365: MINOR: avoid closing over both pre & post-transfor...

2017-01-12 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2365 MINOR: avoid closing over both pre & post-transform record in WorkerSourceTask Followup to #2299 for KAFKA-3209 You can merge this pull request into a Git repository by running: $ git

[GitHub] kafka pull request #2182: ConfigDef experimentation - support List and Ma...

2017-01-11 Thread shikhar
Github user shikhar closed the pull request at: https://github.com/apache/kafka/pull/2182 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

Re: [VOTE] KIP-66: Single Message Transforms for Kafka Connect

2017-01-09 Thread Shikhar Bhushan
Thanks all. The vote passed with +5 (binding). On Fri, Jan 6, 2017 at 11:37 AM Shikhar Bhushan <shik...@confluent.io> wrote: That makes sense to me, I'll fold that into the PR and update the KIP if it gets committed in that form. On Fri, Jan 6, 2017 at 9:44 AM Jason Gustafs

Re: [VOTE] KIP-66: Single Message Transforms for Kafka Connect

2017-01-06 Thread Shikhar Bhushan
Configurable` and remove the `init` method? > > On Thu, Jan 5, 2017 at 5:48 PM, Neha Narkhede <n...@confluent.io> wrote: > > > +1 (binding) > > > > On Wed, Jan 4, 2017 at 2:36 PM Shikhar Bhushan <shik...@confluent.io> > > wrote: > > > >

[jira] [Commented] (KAFKA-4598) Create new SourceTask commit callback method that takes offsets param

2017-01-06 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805460#comment-15805460 ] Shikhar Bhushan commented on KAFKA-4598: Yeah, that's a reasonable alternative with the caveat you

Re: KafkaConnect SinkTask::put

2017-01-06 Thread Shikhar Bhushan
Sorry I forgot to specify, this needs to go into your Connect worker configuration. On Fri, Jan 6, 2017 at 02:57 <david.frank...@bt.com> wrote: > Hi Shikhar, > > I've just added this to ~config/consumer.properties in the Kafka folder > but it doesn't appear to have made any d

[jira] [Commented] (KAFKA-4598) Create new SourceTask commit callback method that takes offsets param

2017-01-05 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802110#comment-15802110 ] Shikhar Bhushan commented on KAFKA-4598: In the meantime the workaround is to use

[jira] [Created] (KAFKA-4598) Create new SourceTask commit callback method that takes offsets param

2017-01-05 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4598: -- Summary: Create new SourceTask commit callback method that takes offsets param Key: KAFKA-4598 URL: https://issues.apache.org/jira/browse/KAFKA-4598 Project

Re: KafkaConnect SinkTask::put

2017-01-05 Thread Shikhar Bhushan
Hi David, You can override the underlying consumer's `max.poll.records` setting for this. E.g. consumer.max.poll.records=500 Best, Shikhar On Thu, Jan 5, 2017 at 3:59 AM <david.frank...@bt.com> wrote: > Is there any way of limiting the number of events that are passed into t

[GitHub] kafka pull request #2313: KAFKA-4575: ensure topic created before starting s...

2017-01-04 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2313 KAFKA-4575: ensure topic created before starting sink for ConnectDistributedTest.test_pause_resume_sink Otherwise in this test the sink task goes through the pause/resume cycle with 0 assigned

[jira] [Commented] (KAFKA-4575) Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming source connector

2017-01-04 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799807#comment-15799807 ] Shikhar Bhushan commented on KAFKA-4575: By the way, the error message is misleading, it should

[jira] [Updated] (KAFKA-4575) Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming sink connector

2017-01-04 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4575: --- Summary: Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming

[jira] [Work started] (KAFKA-4575) Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming source connector

2017-01-04 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4575 started by Shikhar Bhushan. -- > Transient failure in ConnectDistributedTest.test_pause_and_resume_s

Re: [VOTE] KIP-66: Single Message Transforms for Kafka Connect

2017-01-04 Thread Shikhar Bhushan
> wrote: > > > +1 > > > > On Wed, Jan 4, 2017 at 1:29 PM, Gwen Shapira <g...@confluent.io> wrote: > > > > > I would have preferred not to bundle transformations, but since SMT > > > capability is a much needed feature, I'll take it in its cu

[VOTE] KIP-66: Single Message Transforms for Kafka Connect

2017-01-04 Thread Shikhar Bhushan
Hi all, I'd like to start voting on KIP-66: https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect Best, Shikhar

[GitHub] kafka pull request #2299: KAFKA-3209: KIP-66: single message transforms

2017-01-03 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2299 KAFKA-3209: KIP-66: single message transforms *WIP* particularly around testing You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

2017-01-03 Thread Shikhar Bhushan
Makes sense Ewen, I edited the KIP to include this criteria. I'd like to start a voting thread soon unless anyone has additional points for discussion. On Fri, Dec 30, 2016 at 12:14 PM Ewen Cheslack-Postava <e...@confluent.io> wrote: On Thu, Dec 15, 2016 at 7:41 PM, Shikhar Bhushan

[jira] [Work started] (KAFKA-3209) Support single message transforms in Kafka Connect

2017-01-02 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3209 started by Shikhar Bhushan. -- > Support single message transforms in Kafka Conn

[jira] [Commented] (KAFKA-3513) Transient failure of OffsetValidationTest

2017-01-01 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15791790#comment-15791790 ] Shikhar Bhushan commented on KAFKA-3513: There was a failure in the last run http://confluent

[jira] [Assigned] (KAFKA-4575) Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming source connector

2016-12-29 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan reassigned KAFKA-4575: -- Assignee: Shikhar Bhushan > Transient fail

[jira] [Updated] (KAFKA-4575) Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming source connector

2016-12-29 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4575: --- Component/s: system tests KafkaConnect > Transient fail

[jira] [Created] (KAFKA-4575) Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming source connector

2016-12-29 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4575: -- Summary: Transient failure in ConnectDistributedTest.test_pause_and_resume_sink in consuming messages after resuming source connector Key: KAFKA-4575 URL: https

[jira] [Created] (KAFKA-4574) Transient failure in ZooKeeperSecurityUpgradeTest.test_zk_security_upgrade with security_protocol = SASL_PLAINTEXT, SSL

2016-12-29 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4574: -- Summary: Transient failure in ZooKeeperSecurityUpgradeTest.test_zk_security_upgrade with security_protocol = SASL_PLAINTEXT, SSL Key: KAFKA-4574 URL: https

[GitHub] kafka pull request #2277: KAFKA-4527: task status was being updated before a...

2016-12-19 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2277 KAFKA-4527: task status was being updated before actual pause/resume h/t @ewencp for pointing out the issue You can merge this pull request into a Git repository by running: $ git pull https

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

2016-12-15 Thread Shikhar Bhushan
before that we have ExtractAvroMetadata that may fit? and ExtractEmailHeaders doesn't sound totally outlandish either... Nothing in the baked-in list by Shikhar looks out of place. I am concerned about slipperly slope. Or the arbitrariness of the decision if we say that this list is final and nothin

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

2016-12-15 Thread Shikhar Bhushan
e project later than to remove functionality. On Thu, Dec 15, 2016 at 11:59 AM, Shikhar Bhushan <shik...@confluent.io> wrote: > I have updated KIP-66 > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 66%3A+Single+Message+Transforms+for+Kafka+Connect > with > the ch

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

2016-12-15 Thread Shikhar Bhushan
it out? Or at least keeping the built-in set super minimal (Flume has like 3 built-in interceptors)? Gwen On Wed, Dec 14, 2016 at 1:36 PM, Shikhar Bhushan <shik...@confluent.io> wrote: > With regard to a), just using `ConnectRecord` with `newRecord` as a new > abstract method woul

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

2016-12-14 Thread Shikhar Bhushan
o data enrichment by querying external systems, so building a bunch of transformations in could potentially open the floodgates, or at least make decisions about what is included vs what should be 3rd party muddy. -Ewen On Wed, Dec 7, 2016 at 11:46 AM, Shikhar Bhushan <shik...@confluent.io> wrot

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

2016-12-12 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743300#comment-15743300 ] Shikhar Bhushan commented on KAFKA-3209: Thanks [~snisarg]. I self-assigned it as I don't believe

[jira] [Created] (KAFKA-4524) ConfigDef.Type.LIST does not handle escaping

2016-12-12 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4524: -- Summary: ConfigDef.Type.LIST does not handle escaping Key: KAFKA-4524 URL: https://issues.apache.org/jira/browse/KAFKA-4524 Project: Kafka Issue Type

[jira] [Assigned] (KAFKA-3209) Support single message transforms in Kafka Connect

2016-12-09 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan reassigned KAFKA-3209: -- Assignee: Shikhar Bhushan > Support single message transforms in Kafka Conn

[GitHub] kafka pull request #2232: HOTFIX: Fix HerderRequest.compareTo()

2016-12-08 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2232 HOTFIX: Fix HerderRequest.compareTo() With KAFKA-3008 (#1788), the implementation does not respect the contract that 'sgn(x.compareTo(y)) == -sgn(y.compareTo(x))' This fix addresses

[jira] [Commented] (KAFKA-3209) Support single message transforms in Kafka Connect

2016-12-07 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729737#comment-15729737 ] Shikhar Bhushan commented on KAFKA-3209: [~snisarg] and [~jjchorrobe], I revived the discussion

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

2016-12-07 Thread Shikhar Bhushan
Hi all, I have another iteration at a proposal for this feature here: https://cwiki.apache.org/confluence/display/KAFKA/Connect+Transforms+-+Proposed+Design I'd welcome your feedback and comments. Thanks, Shikhar On Tue, Aug 2, 2016 at 7:21 PM Ewen Cheslack-Postava <e...@confluent.io>

[GitHub] kafka pull request #2196: KAFKA-3910: prototype of another approach to cycli...

2016-11-30 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2196 KAFKA-3910: prototype of another approach to cyclic schemas You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka KAFKA-3910

[GitHub] kafka pull request #2182: ConfigDef experimentation - support List and Ma...

2016-11-28 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2182 ConfigDef experimentation - support List and Map<String, T> You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka con

[GitHub] kafka pull request #2139: KAFKA-4161: KIP-89: Allow sink connectors to decou...

2016-11-15 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2139 KAFKA-4161: KIP-89: Allow sink connectors to decouple flush and offset commit You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka

[GitHub] kafka pull request #2131: Remove failing ConnectDistributedTest.test_bad_con...

2016-11-14 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2131 Remove failing ConnectDistributedTest.test_bad_connector_class Since #1911 was merged it is hard to externally test a connector transitioning to FAILED state due to an initialization failure, which

[jira] [Updated] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-11-14 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-3462: --- Resolution: Fixed Status: Resolved (was: Patch Available) This will be handled

Re: [VOTE] KIP-89: Allow sink connectors to decouple flush and offset commit

2016-11-14 Thread Shikhar Bhushan
The vote passed with +3 binding votes. Thanks all! On Sun, Nov 13, 2016 at 1:42 PM Gwen Shapira <g...@confluent.io> wrote: +1 (binding) On Nov 9, 2016 2:17 PM, "Shikhar Bhushan" <shik...@confluent.io> wrote: > Hi, > > I would like to initiate a vote on KIP-

[jira] [Work started] (KAFKA-4161) Decouple flush and offset commits

2016-11-09 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4161 started by Shikhar Bhushan. -- > Decouple flush and offset comm

[VOTE] KIP-89: Allow sink connectors to decouple flush and offset commit

2016-11-09 Thread Shikhar Bhushan
Hi, I would like to initiate a vote on KIP-89 https://cwiki.apache.org/confluence/display/KAFKA/KIP-89%3A+Allow+sink+connectors+to+decouple+flush+and+offset+commit Best, Shikhar

[jira] [Assigned] (KAFKA-3910) Cyclic schema support in ConnectSchema and SchemaBuilder

2016-11-07 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan reassigned KAFKA-3910: -- Assignee: Shikhar Bhushan (was: Ewen Cheslack-Postava) > Cyclic schema supp

[jira] [Commented] (KAFKA-4161) Decouple flush and offset commits

2016-11-04 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637718#comment-15637718 ] Shikhar Bhushan commented on KAFKA-4161: Created KIP-89 for this https://cwiki.apache.org

[DISCUSS] KIP-89: Allow sink connectors to decouple flush and offset commit

2016-11-04 Thread Shikhar Bhushan
Hi all, I created KIP-89 for making a Connect API change that allows for sink connectors to decouple flush and offset commits. https://cwiki.apache.org/confluence/display/KAFKA/KIP-89%3A+Allow+sink+connectors+to+decouple+flush+and+offset+commit I'd welcome your input. Best, Shikhar

[GitHub] kafka pull request #1968: MINOR: missing whitespace in doc for `ssl.cipher.s...

2016-11-04 Thread shikhar
Github user shikhar closed the pull request at: https://github.com/apache/kafka/pull/1968 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Resolved] (KAFKA-4356) o.a.k.common.utils.SystemTime.sleep() swallows InterruptedException

2016-11-03 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan resolved KAFKA-4356. Resolution: Duplicate > o.a.k.common.utils.SystemTime.sleep() swallows InterruptedExcept

[jira] [Commented] (KAFKA-4375) Kafka consumer may swallow some interrupts meant for the calling thread

2016-11-03 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634044#comment-15634044 ] Shikhar Bhushan commented on KAFKA-4375: Good to have a report of this being a problem, I opened

[jira] [Created] (KAFKA-4356) o.a.k.common.utils.SystemTime.sleep() swallows InterruptedException

2016-10-28 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4356: -- Summary: o.a.k.common.utils.SystemTime.sleep() swallows InterruptedException Key: KAFKA-4356 URL: https://issues.apache.org/jira/browse/KAFKA-4356 Project: Kafka

[GitHub] kafka pull request #2040: KAFKA-4161: prototype for exploring API change

2016-10-26 Thread shikhar
Github user shikhar closed the pull request at: https://github.com/apache/kafka/pull/2040 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Resolved] (KAFKA-4342) Kafka-connect- support tinyint values

2016-10-25 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan resolved KAFKA-4342. Resolution: Not A Problem The Connect schema type {{Schema.Type.INT8}} accurately maps

[jira] [Commented] (KAFKA-4306) Connect workers won't shut down if brokers are not available

2016-10-18 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586568#comment-15586568 ] Shikhar Bhushan commented on KAFKA-4306: KAFKA-4154 is another issue relating to broker

[jira] [Updated] (KAFKA-3462) Allow SinkTasks to disable consumer offset commit

2016-10-18 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-3462: --- Issue Type: Improvement (was: Bug) > Allow SinkTasks to disable consumer offset com

[jira] [Updated] (KAFKA-4161) Decouple flush and offset commits

2016-10-18 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4161: --- Issue Type: Improvement (was: New Feature) > Decouple flush and offset comm

[GitHub] kafka pull request #2040: KAFKA-4161: prototype for exploring API change

2016-10-18 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/2040 KAFKA-4161: prototype for exploring API change You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka kafka-4161 Alternatively you can

[GitHub] kafka pull request #1969: MINOR: missing fullstop in doc for `max.partition....

2016-10-04 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1969 MINOR: missing fullstop in doc for `max.partition.fetch.bytes` You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka patch-2

[GitHub] kafka pull request #1968: MINOR: missing whitespace in doc for `ssl.cipher.s...

2016-10-04 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1968 MINOR: missing whitespace in doc for `ssl.cipher.suites` You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka patch-1 Alternatively

[GitHub] kafka pull request #1964: KAFKA-4010: add ConfigDef toEnrichedRst() for addi...

2016-10-04 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1964 KAFKA-4010: add ConfigDef toEnrichedRst() for additional fields in output followup on https://github.com/apache/kafka/pull/1696 cc @rekhajoshm You can merge this pull request into a Git

[jira] [Resolved] (KAFKA-3906) Connect logical types do not support nulls.

2016-09-23 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan resolved KAFKA-3906. Resolution: Fixed I think we should handle null values at the converter layer to avoid

[GitHub] kafka pull request #1872: KAFKA-4183: centralize checking for optional and d...

2016-09-16 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1872 KAFKA-4183: centralize checking for optional and default values to avoid bugs Cleaner to just check once for optional & default value from the `convertToConnect()` function. It also h

[jira] [Reopened] (KAFKA-4183) Logical converters in JsonConverter don't properly handle null values

2016-09-16 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan reopened KAFKA-4183: Assignee: Shikhar Bhushan (was: Ewen Cheslack-Postava) [~rhauch] Reopening this, I

[jira] [Commented] (KAFKA-3906) Connect logical types do not support nulls.

2016-09-16 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497533#comment-15497533 ] Shikhar Bhushan commented on KAFKA-3906: [~jcustenborder] did this come up in the context

[GitHub] kafka pull request #1865: KAFKA-4173: SchemaProjector should successfully pr...

2016-09-16 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1865 KAFKA-4173: SchemaProjector should successfully project missing Struct field when target field is optional You can merge this pull request into a Git repository by running: $ git pull https

[jira] [Updated] (KAFKA-4161) Decouple flush and offset commits

2016-09-16 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4161: --- Summary: Decouple flush and offset commits (was: Allow connectors to request flush via

[jira] [Commented] (KAFKA-4161) Allow connectors to request flush via the context

2016-09-16 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496820#comment-15496820 ] Shikhar Bhushan commented on KAFKA-4161: We could also implement KAFKA-3462 here by having

[jira] [Created] (KAFKA-4173) SchemaProjector should successfully project when source schema field is missing and target schema field is optional

2016-09-14 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4173: -- Summary: SchemaProjector should successfully project when source schema field is missing and target schema field is optional Key: KAFKA-4173 URL: https://issues.apache.org

[jira] [Commented] (KAFKA-4161) Allow connectors to request flush via the context

2016-09-14 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15491296#comment-15491296 ] Shikhar Bhushan commented on KAFKA-4161: bq. Probably worth clarifying whether we're really

[jira] [Created] (KAFKA-4161) Allow connectors to request flush via the context

2016-09-13 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4161: -- Summary: Allow connectors to request flush via the context Key: KAFKA-4161 URL: https://issues.apache.org/jira/browse/KAFKA-4161 Project: Kafka Issue

[jira] [Created] (KAFKA-4159) Allow overriding producer & consumer properties at the connector level

2016-09-13 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4159: -- Summary: Allow overriding producer & consumer properties at the connector level Key: KAFKA-4159 URL: https://issues.apache.org/jira/browse/KAFKA-4159 Pro

[jira] [Updated] (KAFKA-4154) Kafka Connect fails to shutdown if it has not completed startup

2016-09-12 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4154: --- Fix Version/s: (was: 0.10.0.2) > Kafka Connect fails to shutdown if it has not comple

[jira] [Updated] (KAFKA-4154) Kafka Connect fails to shutdown if it has not completed startup

2016-09-12 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4154: --- Fix Version/s: (was: 0.10.1.0) 0.10.0.2 > Kafka Connect fa

[jira] [Created] (KAFKA-4154) Kafka Connect fails to shutdown if it has not completed startup

2016-09-12 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4154: -- Summary: Kafka Connect fails to shutdown if it has not completed startup Key: KAFKA-4154 URL: https://issues.apache.org/jira/browse/KAFKA-4154 Project: Kafka

[jira] [Resolved] (KAFKA-4048) Connect does not support RetriableException consistently for sinks

2016-09-06 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan resolved KAFKA-4048. Resolution: Not A Problem Turns out all exceptions from {{task.flush()}} are treated

[jira] [Commented] (KAFKA-3962) ConfigDef support for resource-specific configuration

2016-09-06 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468352#comment-15468352 ] Shikhar Bhushan commented on KAFKA-3962: This is also realizable today by using

[jira] [Comment Edited] (KAFKA-3962) ConfigDef support for resource-specific configuration

2016-09-06 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468352#comment-15468352 ] Shikhar Bhushan edited comment on KAFKA-3962 at 9/6/16 7:49 PM

[jira] [Commented] (KAFKA-4127) Possible data loss

2016-09-06 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468012#comment-15468012 ] Shikhar Bhushan commented on KAFKA-4127: Dupe of KAFKA-3968 > Possible data l

[jira] [Resolved] (KAFKA-4127) Possible data loss

2016-09-06 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan resolved KAFKA-4127. Resolution: Duplicate > Possible data loss > -- > >

[jira] [Closed] (KAFKA-4127) Possible data loss

2016-09-06 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan closed KAFKA-4127. -- > Possible data loss > -- > > Key: KAFKA-4127 >

[GitHub] kafka pull request #1815: KAFKA-4115: grow default heap size for connect-dis...

2016-09-01 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1815 KAFKA-4115: grow default heap size for connect-distributed.sh to 1G You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka connect-heap

[jira] [Created] (KAFKA-4115) Grow default heap settings for distributed Connect from 256M to 1G

2016-09-01 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4115: -- Summary: Grow default heap settings for distributed Connect from 256M to 1G Key: KAFKA-4115 URL: https://issues.apache.org/jira/browse/KAFKA-4115 Project: Kafka

[jira] [Work started] (KAFKA-4100) Connect Struct schemas built using SchemaBuilder with no fields cause NPE in Struct constructor

2016-08-29 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4100 started by Shikhar Bhushan. -- > Connect Struct schemas built using SchemaBuilder with no fields cause

[GitHub] kafka pull request #1800: KAFKA-4100: ensure 'fields' and 'fieldsByName' are...

2016-08-29 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1800 KAFKA-4100: ensure 'fields' and 'fieldsByName' are not null for Struct schemas You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka

[jira] [Created] (KAFKA-4100) Connect Struct schemas built using SchemaBuilder with no fields cause NPE in Struct constructor

2016-08-29 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4100: -- Summary: Connect Struct schemas built using SchemaBuilder with no fields cause NPE in Struct constructor Key: KAFKA-4100 URL: https://issues.apache.org/jira/browse/KAFKA-4100

[GitHub] kafka pull request #1790: KAFKA-4070: implement Connect Struct.toString()

2016-08-25 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1790 KAFKA-4070: implement Connect Struct.toString() You can merge this pull request into a Git repository by running: $ git pull https://github.com/shikhar/kafka add-struct-tostring

[GitHub] kafka pull request #1745: KAFKA-4042: prevent DistributedHerder thread from ...

2016-08-24 Thread shikhar
Github user shikhar closed the pull request at: https://github.com/apache/kafka/pull/1745 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] kafka pull request #1778: KAFKA-4042: Contain connector & task start/stop fa...

2016-08-24 Thread shikhar
GitHub user shikhar opened a pull request: https://github.com/apache/kafka/pull/1778 KAFKA-4042: Contain connector & task start/stop failures within the Worker Invoke the statusListener.onFailure() callback on start failures so that the statusBackingStore is updated. This invo

[jira] [Resolved] (KAFKA-4068) FileSinkTask - use JsonConverter to serialize

2016-08-20 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan resolved KAFKA-4068. Resolution: Not A Problem I was thinking JSON since it would be easy to serialize

[jira] [Updated] (KAFKA-4070) Implement a useful Struct.toString()

2016-08-20 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4070: --- Description: Logging of {{Struct}}'s does not currently provide any useful output, and users

[jira] [Updated] (KAFKA-4070) Implement a useful Struct.toString()

2016-08-20 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4070: --- Description: Logging of {{Struct}}'s does not currently provide any useful output, and users

[jira] [Created] (KAFKA-4070) Implement a useful Struct.toString()

2016-08-20 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4070: -- Summary: Implement a useful Struct.toString() Key: KAFKA-4070 URL: https://issues.apache.org/jira/browse/KAFKA-4070 Project: Kafka Issue Type

[jira] [Created] (KAFKA-4068) FileSinkTask - use JsonConverter to serialize

2016-08-19 Thread Shikhar Bhushan (JIRA)
Shikhar Bhushan created KAFKA-4068: -- Summary: FileSinkTask - use JsonConverter to serialize Key: KAFKA-4068 URL: https://issues.apache.org/jira/browse/KAFKA-4068 Project: Kafka Issue Type

[jira] [Updated] (KAFKA-4042) DistributedHerder thread can die because of connector & task lifecycle exceptions

2016-08-19 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4042: --- Fix Version/s: 0.10.1.0 > DistributedHerder thread can die because of connector &am

[GitHub] kafka pull request #1745: KAFKA-4042: prevent DistributedHerder thread from ...

2016-08-19 Thread shikhar
GitHub user shikhar reopened a pull request: https://github.com/apache/kafka/pull/1745 KAFKA-4042: prevent DistributedHerder thread from dying from connector/task lifecycle exceptions - `worker.startConnector()` and `worker.startTask()` can throw (e.g. `ClassNotFoundException

[GitHub] kafka pull request #1745: KAFKA-4042: prevent DistributedHerder thread from ...

2016-08-19 Thread shikhar
Github user shikhar closed the pull request at: https://github.com/apache/kafka/pull/1745 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[jira] [Resolved] (KAFKA-3054) Connect Herder fail forever if sent a wrong connector config or task config

2016-08-18 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan resolved KAFKA-3054. Resolution: Done > Connect Herder fail forever if sent a wrong connector config or t

[jira] [Commented] (KAFKA-3054) Connect Herder fail forever if sent a wrong connector config or task config

2016-08-18 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427362#comment-15427362 ] Shikhar Bhushan commented on KAFKA-3054: Addressing this in KAFKA-4042, which should take care

[jira] [Updated] (KAFKA-4048) Connect does not support RetriableException consistently for sinks

2016-08-16 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4048: --- Description: We only allow for handling {{RetriableException}} from calls to {{SinkTask.put

[jira] [Updated] (KAFKA-4048) Connect does not support RetriableException consistently for sinks

2016-08-16 Thread Shikhar Bhushan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shikhar Bhushan updated KAFKA-4048: --- Summary: Connect does not support RetriableException consistently for sinks (was: Connect

  1   2   >