Hi Jhhani, Yes, we can use two (or several) channels to fan out data to different sinks. Then we will have two channels with same data, which may not be an optimized solution. So I want to use just ONE channel, creating a processor to pull the data once from the channel, then distributing to different sinks.
Regards, Yongkun Wang On 12/08/10 18:07, "Juhani Connolly" <[email protected]> wrote: >Hi Yongkun, > >I'm curious why you need to pull the data twice from the sink? Do you >need all sinks to have read the same amount of data? Normally for the >case of splitting data into batch and analytics, we will send data from >the source to two separate channels and have the sinks read from >separate channels. > >On 08/10/2012 02:48 PM, Wang, Yongkun | Yongkun | BDD wrote: >> Hi Denny, >> >> I am working on the patch now, it's not difficult. I have listed the >> changes in that JIRA. >> I think you misunderstand my design, I didn't maintain the order of the >> events. Instead I make sure that each sink will get the same events (or >> different events specified by selector). >> >> Suppose Channel (mc) contains the following events: 4,3,2,1 >> >> If simply enable it by configuration, it may work like this: >> Sink "hsa" may get 1,3; >> Sink "hsb" may get 2,4; >> So different sink will get different data. Is this what user wants? >> >> >> In my design, "hsa" and "hsb" will both get "4,3,2,1". This is a typical >> case when user want to fan-out the data into two places (eg. One for >>batch >> and and another for real-time analysis). >> >> Regards, >> Yongkun Wang >> >> >> On 12/08/10 14:29, "Denny Ye" <[email protected]> wrote: >> >>> hi Yongkun, >>> >>> JIRA can be accessed now. >>> >>> I think it might be difficult to understand the order of events from >>> your thought. If we don't care about the order, can discuss the value >>>and >>> feasibility. In my opinion, data ingest flow is order unawareness, at >>> least, not such important for us. You can try to verify your proposal >>>and >>> give us result. It may be some difficulties in keeping transaction with >>> several Sinks. >>> >>> -Regards >>> Denny Ye >>> >>> >>> 2012/8/10 Wang, Yongkun | Yongkun | BDD <[email protected]> >>> >>>> JIRA is down again? I cannot connect to it and comment there. >>>> >>>> I have a proposal in "Transactional Multiplex (fan out) Sink"): >>>> https://issues.apache.org/jira/browse/FLUME-1435 >>>> Which contains the design of one channel to multiple sinks. >>>> >>>> You can search the email since JIRA cannot be accessed. >>>> >>>> I think this is more than a configuration issue. If simply enable >>>> several >>>> sinks on the same channel, they will take it either in a round-robin >>>> mode >>>> or in a unpredictable mode if the speed of sinks are different. >>>> >>>> So it's better to have a even higher level transaction control instead >>>> of >>>> the transaction in the process() of each sink, as I describe in >>>> FLUME-1435. >>>> >>>> Regards, >>>> Yongkun Wang >>>> >>>> >>>> On 12/08/10 12:30, "Denny Ye (JIRA)" <[email protected]> wrote: >>>> >>>>> Denny Ye created FLUME-1479: >>>>> ------------------------------- >>>>> >>>>> Summary: Multiple Sinks can connect to single Channel >>>>> Key: FLUME-1479 >>>>> URL: >>>>>https://issues.apache.org/jira/browse/FLUME-1479 >>>>> Project: Flume >>>>> Issue Type: Bug >>>>> Components: Configuration >>>>> Affects Versions: v1.2.0 >>>>> Reporter: Denny Ye >>>>> Assignee: Denny Ye >>>>> Fix For: v1.3.0 >>>>> >>>>> >>>>> If we has one Channel (mc) and two Sinks (hsa, hsb), then they may be >>>>> connected with each other with configuration example >>>>> {quote} >>>>> agent.sinks.hsa.channel = mc >>>>> agent.sinks.hsb.channel = mc >>>>> {quote} >>>>> It means that there have multiple Sinks can connect to single >>>>>Channel. >>>>> Normally, one Sink only can connect to unified Channel >>>>> >>>>> -- >>>>> This message is automatically generated by JIRA. >>>>> If you think it was sent incorrectly, please contact your JIRA >>>>> administrators: >>>>> >>>>>https://issues.apache.org/jira/secure/ContactAdministrators!default.js >>>>>pa >>>>> For more information on JIRA, see: >>>> http://www.atlassian.com/software/jira >>>>> >>>>> >>>> >>>> >> >> > >
