[
https://issues.apache.org/jira/browse/HADOOP-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12663455#action_12663455
]
Ari Rabkin commented on HADOOP-5018:
------------------------------------
Both 1 and 2 are worthy goals. I think that pipelines are a fairly natural way
to accomplish both. I intended to write a pipeline stage for doing
subscriptions for real-time delivery; if you're also working on that, it's
pretty awesome, and we should open a JIRA.
I hadn't thought of log-to-local-storage, but it should be easy to write a
pipeline stage that stores everything, passes it through, and that also has a
worker thread that does the write to HDFS.
What do you mean by removing the Hadoop dependency? I assume you don't
literally mean breaking all dependence on Hadoop-core. But you can already
point the SeqFileWriter at a local filesystem; you don't need an HDFS cluster.
> Chukwa should support pipelined writers
> ---------------------------------------
>
> Key: HADOOP-5018
> URL: https://issues.apache.org/jira/browse/HADOOP-5018
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/chukwa
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Attachments: pipeline.patch
>
>
> We ought to support chaining together writers; this will radically increase
> flexibility and make it practical to add new features without major surgery
> by putting them in pass-through or filter classes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
