Got it. Thanx. On Apr 7, 2014 12:39 PM, "Jeff Lord" <[email protected]> wrote:
> No not at all. Flume's transactional model guarantees delivery between > hops. > https://blogs.apache.org/flume/entry/flume_ng_architecture > > > On Mon, Apr 7, 2014 at 10:16 AM, Christopher Shannon < > [email protected]> wrote: > >> So, this basically means that Flume's transactional model is also >> unreliable. That would have to mean that the downstream agent is sending an >> ack to the upstream agent before it actually persists the event. >> >> On Apr 7, 2014 10:48 AM, "Jeff Lord" <[email protected]> wrote: >> > >> > No. If you need to guarantee delivery of events please use a file >> channel. >> > https://blogs.apache.org/flume/entry/apache_flume_filechannel >> > >> > >> > On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon < >> [email protected]> wrote: >> >> >> >> >> >> On Apr 7, 2014 9:35 AM, "Jeff Lord" <[email protected]> wrote: >> >> > >> >> > >> >> > >> >> > >> >> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal < >> [email protected]> wrote: >> >> >> >> >> >> Hi Jeff, >> >> >> >> >> >> Yes, I am using the memory channel, and that's because I want it to >> be more reliable and not miss any events/messages. >> >> >> As I've read in flume documentation that the memory channel is fast >> but there could be a chance of missing events if the in-memory buffer fills >> up. >> >> > >> >> > >> >> > Memory channel is not reliable, meaning if the flume agent goes down >> or is restarted while there are events in the channel than this data will >> be lost. >> >> > For reliability please use the file channel. >> >> > >> >> >> >> Jeff, >> >> >> >> I am using an upstream agent with a spooling directory source and a >> memory channel, and the downstream agent uses a memory channel and an HDFS >> sink. If my downstream agent goes down for any reason, are the entries lost >> in the downstream agent's memory channel still preserved in the memory >> channel / file directory of the upstream agent? >> >> >> >> All the best, Chris >> > >> > >> No. If you need to guarantee delivery of events please use a file >> channel. >> https://blogs.apache.org/flume/entry/apache_flume_filechannel >> >> >> On Mon, Apr 7, 2014 at 8:38 AM, Christopher Shannon < >> [email protected]> wrote: >> >>> >>> On Apr 7, 2014 9:35 AM, "Jeff Lord" <[email protected]> wrote: >>> > >>> > >>> > >>> > >>> > On Thu, Apr 3, 2014 at 11:27 AM, Mohit Durgapal < >>> [email protected]> wrote: >>> >> >>> >> Hi Jeff, >>> >> >>> >> Yes, I am using the memory channel, and that's because I want it to >>> be more reliable and not miss any events/messages. >>> >> As I've read in flume documentation that the memory channel is fast >>> but there could be a chance of missing events if the in-memory buffer fills >>> up. >>> > >>> > >>> > Memory channel is not reliable, meaning if the flume agent goes down >>> or is restarted while there are events in the channel than this data will >>> be lost. >>> > For reliability please use the file channel. >>> > >>> >>> Jeff, >>> >>> I am using an upstream agent with a spooling directory source and a >>> memory channel, and the downstream agent uses a memory channel and an HDFS >>> sink. If my downstream agent goes down for any reason, are the entries lost >>> in the downstream agent's memory channel still preserved in the memory >>> channel / file directory of the upstream agent? >>> >>> All the best, Chris >>> >> >> >
