With multiple sinks reading from one channel, will each sink read each event, or will the events be distributed between the sinks?
Regards, Guy Needham | Data Discovery Virgin Media | Technology and Transformation | Data Bartley Wood Business Park, Hook, Hampshire RG27 9UP D 01256 75 3362 I welcome VSRE emails. Learn more at http://vsre.info/ From: Hari Shreedharan [mailto:[email protected]] Sent: 19 May 2015 16:30 To: [email protected] Subject: Re: Channel locks Each sink does not hold a lock on the channel. Channels can handle multiple sinks reading from them and in fact, that is encouraged. On Tuesday, May 19, 2015, Sverre Bakke <[email protected]<mailto:[email protected]>> wrote: Hi, When using a single source, channel and sink setup with Syslog, memory channel and KafkaSink (from https://github.com/thilinamb/flume-ng-kafka-sink) I am experiencing performance issues. After looking at the source code of the sink, it seems that the sink will begin a transaction, then do everything (e.g. compress, send over the network, wait for ack), and eventually close/commit the transaction after everything is done. Is there any way to increase the performance of this setup? Earlier I have seen people proposing adding more sinks to get higher throughput, but if every sink holds a lock on the channel until done with processing, then I would assume that this only causes additional performance issues? Are my understanding correct on how channel locking works? Regards, Sverre -- Thanks, Hari -------------------------------------------------------------------- Save Paper - Do you really need to print this e-mail? Visit www.virginmedia.com for more information, and more fun. This email and any attachments are or may be confidential and legally privileged and are sent solely for the attention of the addressee(s). If you have received this email in error, please delete it from your system: its use, disclosure or copying is unauthorised. Statements and opinions expressed in this email may not represent those of Virgin Media. Any representations or commitments in this email are subject to contract. Registered office: Media House, Bartley Wood Business Park, Hook, Hampshire, RG27 9UP Registered in England and Wales with number 2591237
