Hi Debasish,
> flush.size=3 this means every 3 messages in that topic will end up in its own HDFS file, which is probably why you end up with so many files that ls hurts. You should flush a bigger batch or after a high enough interval. > tasks.max=1 Unless you have a single partition topic, you need to up this number for better parallelism. HTH, Abdoulaye On Tue, Jul 18, 2017 at 11:12 AM, Debasish Ghosh <[email protected]> wrote: > Hi - > > I have a Kafka Streams application that generates Avro records in a topic, > which is being read by a Kafka Connect process that uses HDFS Sink > connector. The topic has around 1.6 million messages. And the Kafka Connect > script is as follows .. > > bin/connect-standalone > > etc/schema-registry/connect-avro-standalone.properties > > etc/kafka-connect-hdfs/quickstart-hdfs.properties > > > where quickstart-hdfs.properties contains the following .. > > name=hdfs-sink > > connector.class=io.confluent.connect.hdfs.HdfsSinkConnector > > tasks.max=1 > > topics=avro-topic > > hdfs.url=hdfs://0.0.0.0:9000 > > flush.size=3 > > > The problem is that the Kafka Connect process looks to be running in an > infinite loop with messages like the following .. > > [2017-07-18 20:02:04,487] INFO Starting commit and rotation for topic > > partition avro-topic-0 with start offsets {partition=0=1143033} and end > > offsets {partition=0=1143035} > > (io.confluent.connect.hdfs.TopicPartitionWriter:297) > > [2017-07-18 20:02:04,491] INFO Committed hdfs:// > > 0.0.0.0:9000/topics/avro-topic/partition=0/avro-topic+ > 0+0001143033+0001143035.avro > > for avro-topic-0 (io.confluent.connect.hdfs.TopicPartitionWriter:625) > > > The result is that the avro files created are so many in numbers that I > cannot do an ls on the folder. > > $ hdfs dfs -ls /topics/avro-topic > > Found 1 items > > drwxr-xr-x - debasishghosh supergroup 0 2017-07-18 20:02 > > /topics/avro-topic/partition=0 > > > Trying to list to more depth in the HDFS folder results in an > OutOfMemoryError .. > > $ hdfs dfs -ls /topics/avro-topic/partition=0 > > 17/07/18 20:02:19 WARN util.NativeCodeLoader: Unable to load > native-hadoop > > library for your platform... using builtin-java classes where applicable > > Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit > > exceeded > > at java.util.Arrays.copyOfRange(Arrays.java:3664) > > at java.lang.String.<init>(String.java:207) > > at java.lang.String.substring(String.java:1969) > > at java.net.URI$Parser.substring(URI.java:2869) > > at java.net.URI$Parser.parseHierarchical(URI.java:3106) > > ... > > > Why is the Kafka Connect program going in an infinite loop ? How can I > prevent it ? > > I am using Confluent 3.2.2 for the schema registry, Avro serialization part > and Apache Kafka 0.10.2.1 for Kafka Streams client and the broker part. > > Help ? > > regards. > > -- > Debasish Ghosh > http://manning.com/ghosh2 > http://manning.com/ghosh > > Twttr: @debasishg > Blog: http://debasishg.blogspot.com > Code: http://github.com/debasishg > -- Abdoulaye Diallo
