[
https://issues.apache.org/jira/browse/CHUKWA-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14044290#comment-14044290
]
Eric Yang commented on CHUKWA-674:
----------------------------------
Shreyas,
SeqFileWriter was built prior to introduction of PipelinableWriter. This is
the reason that it can not write and blocks the chunk to be passed to the next
writer. If the configuration is done with SeqFileWriter being last, it will
work fine. In the event, if the writer fails for bad data, the chunk can be
dropped. In the event that writer failed due to down stream unavailability,
then the same chunk can be retried. It is possible to have duplicated data
this way, and the sequence id helps to eliminate the duplication. Hence, this
should be working as designed.
> Integrate Chukwa collector feature to Chukwa agent
> --------------------------------------------------
>
> Key: CHUKWA-674
> URL: https://issues.apache.org/jira/browse/CHUKWA-674
> Project: Chukwa
> Issue Type: Improvement
> Components: Data Collection
> Environment: MacOSX, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: CHUKWA-674.patch
>
>
> Feature offered in Chukwa collector can be integrated into Chukwa agent, and
> use multi-tier Chukwa agent to collect data for large scale cluster. For
> small cluster, agents can talk directly to HDFS cluster to reduce the
> complexity of deployment. The required features to reduce the need of Chukwa
> collectors are:
> - Enhance agent rest api to receive chunk data.
> - Pipeline writer to channel data to storage destinations (HDFS, HBASE).
> - Improve connector interface and replace http connector with collector
> connector for bandwidth balance.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
