On Mon, May 15, 2017 at 2:06 PM, Phillip Mann <[email protected]> wrote:
> Currently, Kafka Connect experiences a spike in CPU usage which causes > Kafka Connect to crash. What kind of crash? Can you provide an error or stacktrace? > There is really no useful information from the logs to help me > understand what is causing this to happen. Is this a known issue? If it > matters, my configuration settings are as follows: > > format.class":"com.trulia.footprint.eventexportkafkaconnect.SequenceFileFormat", > (our custom Sequence File format output) > "connector.class":"io.confluent.connect.s3.S3SinkConnector", > "tasks.max":"10", > "topics":"truliaTopic", > "flush.size":"150000", > "s3.part.size":"5242880", > "s3.bucket.name":"truliaBucket", > "storage.class":"io.confluent.connect.s3.storage.S3Storage", > "partitioner.class":"com.trulia.footprint.eventexportkafkaconnect.FootprintEventPartitioner", > (our custom partitioner) > "schema.generator.class": > "io.confluent.connect.storage.hive.schema.DefaultSchemaGenerator", > "schema.compatibility": "NONE", > "topics.dir":"truliaOutput" > There are multiple custom plugins here -- the format.class and partitioner.class implementations are not built-in. What are the implementations here, and could they affect CPU usage? I'd normally expect these to have more of an effect on memory usage, but could they also be affecting CPU usage? Is there any process, compression, etc? -Ewen
