[ https://issues.apache.org/jira/browse/KAFKA-2365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642323#comment-14642323 ]
Gwen Shapira commented on KAFKA-2365: ------------------------------------- I keep waiting for [~ewencp] to create the "log connector" jira so I can add specific requirements, but its getting late, so I'm adding them here for reference and will move the comment to the right place tomorrow :) Top log features missing from Flume (more or less in order of importance): * Ability to safely tail a log (vs. only ingest completed files) * rename / move file after done ingesting * recursively ingest from sub-directories * handling gzipped files (i.e. unzip and split into records) * pulling from FTP / http / SFTP * some intelligence regarding record splitting (support for CSV and JSON is pretty high on the list) * conversion to Avro (duh!), bonus if the schema can be magically inferred and lated editted. > Copycat checklist > ----------------- > > Key: KAFKA-2365 > URL: https://issues.apache.org/jira/browse/KAFKA-2365 > Project: Kafka > Issue Type: New Feature > Reporter: Ewen Cheslack-Postava > Assignee: Ewen Cheslack-Postava > Labels: feature > Fix For: 0.8.3 > > > This covers the development plan for > [KIP-26|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767]. > There are a number of features that can be developed in sequence to make > incremental progress, and often in parallel: > * Initial patch - connector API and core implementation > * Runtime data API > * Standalone CLI > * REST API > * Distributed copycat - CLI > * Distributed copycat - coordinator > * Distributed copycat - config storage > * Distributed copycat - offset storage > * Log/file connector (sample source/sink connector) > * Elasticsearch sink connector (sample sink connector for full log -> Kafka > -> Elasticsearch sample pipeline) > * Copycat metrics > * System tests (including connector tests) > * Mirrormaker connector > * Copycat documentation > This is an initial list, but it might need refinement to allow for more > incremental progress and may be missing features we find we want before the > initial release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)