Re: Connect: SourceTask poll & commit interaction

I have created https://issues.apache.org/jira/browse/KAFKA-4598 for this. On Wed, Dec 14, 2016 at 2:58 PM Shikhar Bhushan wrote: > Hi Mathieu, > > I think you are right, there is currently no mutual exclusion between > `task.commit()` and `task.poll()`. The solution you are thin

Re: Connect: SourceTask poll & commit interaction

Hi Mathieu, I think you are right, there is currently no mutual exclusion between `task.commit()` and `task.poll()`. The solution you are thinking of with maintaining the committed offset state yourself seems reasonable, though inconvenient. It probably makes sense to add a new parameterized `com

Re: Kafka connect jdbc Sink connector issues when moving from MySQL to Oracle DB

Hi Srikrishna, For future please address questions related to Confluent's connectors to the relevant ML (https://groups.google.com/forum/#!forum/confluent-platform ). The NVARCHAR2(4000) mapping for string types for Oracle was based on my reading of the documentation which states it can hold up t

Re: Schema for jsonConverter

Hi Srikrishna, The issue is that you are using "name" to specify the field name for the struct's fields. The correct key to use is "field". Best, Shikhar On Thu, Sep 15, 2016 at 4:23 PM Gwen Shapira wrote: > ah, never mind - I just noticed you do use a schema... Maybe you are > running into t

Re: Sink Connector feature request: SinkTask.putAndReport()

cords. > > In fact, this mechanism is really more general than just volume based > commits; its really about providing sinks a flexible commit capability > (e.g., some sink event requires premature commit, or otherwise requires > modification of the commit interval). > > On Tue, Sep 1

Re: Sink Connector feature request: SinkTask.putAndReport()

Hi Dean, I agree, it would be good to support volume-based offset commits. For giving more control on flushes to a sink connector, rather than adding a new task.put() variant, I think it may be better to add an API like "requestFlush()" to the `SinkTaskContext` (and perhaps also `SinkTaskContext`

Re: Kafka Connect startup issues

Hi Jeff, If you are using the source tar, you will first need to compile the JAR's as per the instructions on the README (https://github.com/apache/kafka). Then you will indeed be able to run with "./bin/connect-distributed.sh config/connect-distributed.properties". Alternately, you can just use t

Re: Kafka Source Connector tasks run only in same node

Hi Dhanuka, For something like a TCP connector where you have a dependency on the placement, the standalone mode makes more sense. With distributed mode you can't have guarantees of what instances your connector's tasks will be distributed to. Best, Shikhar On Wed, Aug 31, 2016 at 8:05 AM dhanu

Re: How do I remove myself from mailing list?

Just to confirm: after you unsubscribe by sending an email to users-unsubscr...@kafka.apache.org, you should receive an email titled "confirm unsubscribe from users@kafka.apache.org", which you need to reply to in order for the unsubscribe to take effect. Did you folks do that? On Tue, Aug 30, 201

Re: Kafka ETL for Parquet

d not reach to run correctly. > I could not find how to avoid SR for writing AvroConverter. > > Could you give me some concrete implementation of AvroConverter to support > for instance, Classpath Avro Schema Registry? > > - Kidong. > > > > > > 2016-08-02 7:40

Re: Kafka ETL for Parquet

Er, mislinked HDFS connector :) https://github.com/confluentinc/kafka-connect-hdfs On Mon, Aug 1, 2016 at 3:39 PM Shikhar Bhushan wrote: > Hi Kidong, > > That's pretty cool! I'm curious what this offers over the Confluent HDFS > connector <https://github.com/my

Re: Kafka ETL for Parquet

Hi Kidong, That's pretty cool! I'm curious what this offers over the Confluent HDFS connector , though. The README mentions not depending on the Schema Registry, and that the schema can be retrieved via the classpath and Consul. This functionality s

Re: Heartbeating during long processing times

Hi Elias, KIP-62 has a discussion of current options, and the improvements that are coming. Best, Shikhar On Thu, Jun 30, 2016 at 6:02 PM Elias Levy wrote: > What is the of

Re: broker randomly shuts down

ot see stderr log file anywhere. Can you point me to where kafka > would write such a file? > > On Thu, Jun 30, 2016 at 5:10 PM, Shikhar Bhushan > wrote: > > > Perhaps it's a JVM crash? You might not see anything in the standard > > application-level logs, you

Re: broker randomly shuts down

Perhaps it's a JVM crash? You might not see anything in the standard application-level logs, you'd need to look for the stderr. On Thu, Jun 30, 2016 at 5:07 PM allen chan wrote: > Anyone else have ideas? > > This is still happening. I moved off zookeeper from the server to its own > dedicated VM