[jira] [Created] (FLUME-1435) Proposal of Transactional Multiplex (fan out) Sink

[Yongkun Wang (JIRA)]

Yongkun Wang created FLUME-1435:
---

 Summary: Proposal of Transactional Multiplex (fan out) Sink
 Key: FLUME-1435
 URL: https://issues.apache.org/jira/browse/FLUME-1435
 Project: Flume
  Issue Type: New Feature
  Components: Channel, Sinks+Sources
Affects Versions: v1.2.0
Reporter: Yongkun Wang
Assignee: Yongkun Wang


Hi, 

I have proposed this design by email several weeks ago. I received comment from 
Brock. I guess your guys are very busy, so I think I'd better create this JIRA, 
and put slides and patch here to explain it more clearly.

Regards,
Yongkun

Following is the design from previous email, I will attach slides later.

From: "Wang, Yongkun" 
Date: Wed, 25 Jul 2012 10:32:31 GMT
To: "dev@flume.apache.org" 
Cc: "u...@flume.apache.org" 
Subject: Transactional Multiplex (fan out) Sink


Hi,

In our system, we need to fan out the aggregated flow to several destinations. 
Usually the flow to each destination is identical. 

There is a nice feature of NG, the "multiplexing flow", which can satisfy our 
requirements. It is implemented by using separated channels, which is easy to 
do transaction control.

But in our case, the fan out is replicating in most cases. If using the current 
"Replicating to Channels" configuration, we will get several identical channels 
on the same host, which may consume a large amount resources (memory, disk, 
etc.). The performance may possibly drop. And the events to each destination 
may not be synchronized.

I read NG source, I think I could move the multiplex from Channel to Sink, that 
is, using single Channel, fan out to different Sinks, which may solve the 
problems (resource usage, performance, event synchronization) of multiple 
Channels.

I see that there are "LoadBalancingSinkProcessor" and "SinkSelector" classes, 
but they cannot be used to achieve the target of replicating events from one 
Channel to different Sinks.

The following is an optional implementation of the Transactional Multiplex (or 
fan out) Sink:
1. Add a Transactional Multiplex Sink Processor, which will group the 
operations of all fan out Sinks into one transaction, and use a certain policy 
to commit the transaction.  
2. Add MultiplexSink, which simply processes the Events and report status, no 
transaction.
3. Add "peek()" and "remove()" to Channel and Transaction.

The policy of committing a transaction can be defined as follow (suppose we 
have N Sinks) :
1. When M(0=https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1045) Proposal to support disk based spooling

[Inder SIngh (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430869#comment-13430869
 ] 

Inder SIngh commented on FLUME-1045:


@Patrick, @Jarec, @Venkatesh

what your thoughts about the above configuration to combine file and memory 
channel.
Having the flush frequency in file channel configurable could achieve both 
tight & relaxed semantics for data-loss scenarios.

This kind of setup would be a shared hosted setup in any org and they may want 
to combine different streams on the same instance, wherein depending on data 
importance different semantics would be required for different streams.



> Proposal to support disk based spooling
> ---
>
> Key: FLUME-1045
> URL: https://issues.apache.org/jira/browse/FLUME-1045
> Project: Flume
>  Issue Type: New Feature
>Affects Versions: v1.0.0
>Reporter: Inder SIngh
>Priority: Minor
>  Labels: patch
> Attachments: FLUME-1045-1.patch, FLUME-1045-2.patch
>
>
> 1. Problem Description 
> A sink being unavailable at any stage in the pipeline causes it to back-off 
> and retry after a while. Channel's associated with such sinks start buffering 
> data with the caveat that if you are using a memory channel it can result in 
> a domino effect on the entire pipeline. There could be legitimate down times 
> eg: HDFS sink being down for name node maintenance, hadoop upgrades. 
> 2. Why not use a durable channel (JDBC, FileChannel)?
> Want high throughput and support sink down times as a first class use-case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1434) Distinguish background worker with channel name

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430847#comment-13430847
 ] 

Hari Shreedharan commented on FLUME-1434:
-

Patch committed. Thanks Brock!

> Distinguish background worker with channel name 
> 
>
> Key: FLUME-1434
> URL: https://issues.apache.org/jira/browse/FLUME-1434
> Project: Flume
>  Issue Type: Improvement
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Brock Noland
>Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1434.patch
>
>
> All checkpoint workers in multiple channel with same thread name, it's hard 
> to track checkpoint progress for specified channel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1045) Proposal to support disk based spooling

[Patrick Wendell (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430846#comment-13430846
 ] 

Patrick Wendell commented on FLUME-1045:


Hey I'm just getting caught up on this discussion. One issue (or 
misunderstanding) that I have with Sharad's proposal, and any of the proposals 
that seem to suggest a composed "MemoryChannel + File Channel" is what we want 
here, is that the existing FileChannel has certain transaction guarantees that 
you would not want in this case.

If you are running a memory channel and you want to spill over to disk, you are 
already accepting "best effort" delivery semantics for the normal case where 
all of the data is fitting in memory.

If our spillover implementation directly uses, or functionally mirrors, the 
existing FileChannel, we'll be offering much stronger semantics once the data 
has spilled over to disk, at a high throughput cost.

For instance, the FileChannel flushes to disk on every transaction to avoid 
data loss. If we were to build a disk-spilling extension to the existing 
MemoryChannel, we'd likely want to batch these disk flushes to make the 
aggregate disk throughput better. We just wouldn't want the strong semantics 
offered by the FileChannel.

That is why I think that just extending the Memory Channel to have some type of 
best effort disk spilling would be best, since it differs in fundamental ways 
from what is accomplished with the FileChannel.

> Proposal to support disk based spooling
> ---
>
> Key: FLUME-1045
> URL: https://issues.apache.org/jira/browse/FLUME-1045
> Project: Flume
>  Issue Type: New Feature
>Affects Versions: v1.0.0
>Reporter: Inder SIngh
>Priority: Minor
>  Labels: patch
> Attachments: FLUME-1045-1.patch, FLUME-1045-2.patch
>
>
> 1. Problem Description 
> A sink being unavailable at any stage in the pipeline causes it to back-off 
> and retry after a while. Channel's associated with such sinks start buffering 
> data with the caveat that if you are using a memory channel it can result in 
> a domino effect on the entire pipeline. There could be legitimate down times 
> eg: HDFS sink being down for name node maintenance, hadoop upgrades. 
> 2. Why not use a durable channel (JDBC, FileChannel)?
> Want high throughput and support sink down times as a first class use-case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1434) Distinguish background worker with channel name

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430841#comment-13430841
 ] 

Hari Shreedharan commented on FLUME-1434:
-

Brock - I am committing the patch from RB. Please attach the same patch to this 
jira as well.

> Distinguish background worker with channel name 
> 
>
> Key: FLUME-1434
> URL: https://issues.apache.org/jira/browse/FLUME-1434
> Project: Flume
>  Issue Type: Improvement
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Brock Noland
>Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1434.patch
>
>
> All checkpoint workers in multiple channel with same thread name, it's hard 
> to track checkpoint progress for specified channel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: FLUME-1434: Distinguish background worker with channel name

[Hari Shreedharan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6459/#review10007
---

Ship it!


Ship It!

- Hari Shreedharan


On Aug. 8, 2012, 2:23 a.m., Brock Noland wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6459/
> ---
> 
> (Updated Aug. 8, 2012, 2:23 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> Fixes test failures introduced in the first patch.
> 
> 
> This addresses bug FLUME-1434.
> https://issues.apache.org/jira/browse/FLUME-1434
> 
> 
> Diffs
> -
> 
>   
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
>  5f7c83e 
>   
> flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
>  f286b66 
> 
> Diff: https://reviews.apache.org/r/6459/diff/
> 
> 
> Testing
> ---
> 
> unit tests pass
> 
> 
> Thanks,
> 
> Brock Noland
> 
>

NYC Flume Meetup

[Kathleen Ting]

Hi,

A Flume Meetup will be held in New York on the Thursday (10/25) of
Strata + Hadoop World 2012 from 6 - 8 pm.

If you are interested in attending, please RSVP at the following location:

http://www.meetup.com/Flume-User-Meetup/events/76809642/

Thanks,
Kathleen Ting

Review Request: FLUME-1434: Distinguish background worker with channel name

[Brock Noland]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6459/
---

Review request for Flume.


Description
---

Fixes test failures introduced in the first patch.


This addresses bug FLUME-1434.
https://issues.apache.org/jira/browse/FLUME-1434


Diffs
-

  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
 5f7c83e 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
 f286b66 

Diff: https://reviews.apache.org/r/6459/diff/


Testing
---

unit tests pass


Thanks,

Brock Noland

[jira] [Reopened] (FLUME-1434) Distinguish background worker with channel name

[Brock Noland (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brock Noland reopened FLUME-1434:
-

  Assignee: Brock Noland  (was: Denny Ye)

FLUME-1432 is a big patch. I should have just re-opened this one. Patch coming 
soon.

> Distinguish background worker with channel name 
> 
>
> Key: FLUME-1434
> URL: https://issues.apache.org/jira/browse/FLUME-1434
> Project: Flume
>  Issue Type: Improvement
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Brock Noland
>Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1434.patch
>
>
> All checkpoint workers in multiple channel with same thread name, it's hard 
> to track checkpoint progress for specified channel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1434) Distinguish background worker with channel name

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430613#comment-13430613
 ] 

Brock Noland commented on FLUME-1434:
-

Shoot, this introduced test failures. The fix for the test failures is in 
FLUME-1432.

> Distinguish background worker with channel name 
> 
>
> Key: FLUME-1434
> URL: https://issues.apache.org/jira/browse/FLUME-1434
> Project: Flume
>  Issue Type: Improvement
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Denny Ye
>Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1434.patch
>
>
> All checkpoint workers in multiple channel with same thread name, it's hard 
> to track checkpoint progress for specified channel.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1385) Add a multiport syslog source

[Mike Percy (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430611#comment-13430611
 ] 

Mike Percy commented on FLUME-1385:
---

I've got something I've been working on for a little while. I'll post a patch 
soon.

> Add a multiport syslog source
> -
>
> Key: FLUME-1385
> URL: https://issues.apache.org/jira/browse/FLUME-1385
> Project: Flume
>  Issue Type: New Feature
>Reporter: Hari Shreedharan
>Assignee: Mike Percy
>
> A common use case for syslog to be sent to different ports. This will require 
> one syslog source per flume agent, each of which runs a Netty Server, 
> requiring at several threads more than what is required.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1432) FileChannel should replay logs in the order they were written

[Brock Noland (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430601#comment-13430601
 ] 

Brock Noland commented on FLUME-1432:
-

That is a difficult question. The answer is no there is no format changes, only 
how the the "timestamp" in the format is assigned and how we replay the file.

In flume 1.2 there are log files/checkpoints we cannot replay. This is what the 
patch addresses. This change does not allow us to replay those invalid 
files/checkpoints. I have looked at the problem and I don't see how we can 
address that, they are just invalid. The invalid files/checkpoints occur when 
the channel is full. 

I have merged this JIRA with FLUME-1431 because that JIRA added a regression 
test framework for the file channel format. Additionally, I added an additional 
test which specifically tests log files/checkpoints that were created with 
flume-1.2. Review board has been updated.

Note: this patch includes binary content so once we get a +1 I will have to 
commit it.

> FileChannel should replay logs in the order they were written
> -
>
> Key: FLUME-1432
> URL: https://issues.apache.org/jira/browse/FLUME-1432
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Brock Noland
>Assignee: Brock Noland
>
> Currently we replay the logs one at a time causing us to build large queue of 
> pending takes. Additionally, there maybe scenerios where this simply will not 
> work. Take a queue which is full (via checkpoint) and two files:
> 1:
> put
> commit
> put
> commit
> 2:
> take
> commit
> take
> commit
> take
> commit
> Replaying these logs in the current form will not work because we will we try 
> and reply the puts first and exceed our queue size. For these reasons, we 
> should replay them in the order they were written.
> However, at present there is no way to do this. Currently we have two 
> identifers in each record we write, a transaction id and a timestamp. Neither 
> can be used in replaying logs in order because the transaction id is created 
> when we create the transaction not when we write to the log. Someone could 
> create transaction, sleep, and then do work. The timestamp its not granular 
> enough as we could have duplicates.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: FLUME-1432: FileChannel should replay logs in order of them being written

[Brock Noland]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6411/
---

(Updated Aug. 7, 2012, 8:38 p.m.)


Review request for Flume.


Changes
---

Updated patch to include FLUME-1431.

Note that the patch includes binary content so I'll have to commit this one.


Description
---

This resolves FLUME-1432 which describes a problem with how we replayed logs. 
Instead of replaying one log at a time, we replay all the logs in the order 
they were written. We do this by changing the "timestamp" field to be a 
globally ordered number which allows us to do a merge of all the log files. 
This should be faster and also fix some problems where we could not replay some 
checkpoints.

Its worth noting that the condition in testRaceFoundInFLUME1432 was observed 
prior to this patch and will not pass if we simply used the transaction id to 
order edits due to that being generated on transaction create or the previous 
value of the field, timestamp, because they were observed to be non-unique on a 
2 core host.

If committed FLUME-1433 can be closed as well since this change fixes that test 
as well.


This addresses bug FLUME-1432.
https://issues.apache.org/jira/browse/FLUME-1432


Diffs (updated)
-

  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Commit.java
 03b1060 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
 cc8f89a 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
 e692934 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
 2b733b1 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
 5615c6d 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogRecord.java
 PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Put.java
 bcd37ab 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
 da2d068 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Rollback.java
 b42501f 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Take.java
 42b197f 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java
 c222bd1 
  
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/WriteOrderOracle.java
 PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
 17a7cf9 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
 5f7c83e 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
 f286b66 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java
 8995089 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogRecord.java
 PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecord.java
 a46526d 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-checkpoint.gz
 PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-log-1.gz 
PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-log-2.gz 
PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-log-3.gz 
PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-checkpoint.gz
 PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-log-1.gz
 PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-log-2.gz
 PRE-CREATION 
  
flume-ng-channels/flume-file-channel/src/test/resources/fileformat-v2-pre-FLUME-1432-log-3.gz
 PRE-CREATION 
  
flume-ng-sinks/flume-ng-hbase-sink/src/test/java/org/apache/flume/sink/hbase/TestAsyncHBaseSink.java
 641952a 

Diff: https://reviews.apache.org/r/6411/diff/


Testing
---

All unit tests pass. A unit test is added, testRaceFoundInFLUME1432 which found 
a problem with the way we previously replayed logs. It passes with these 
changes. I also tested on a two node flume installation stopping and restarting 
a node several times.


Thanks,

Brock Noland

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430530#comment-13430530
 ] 

Jarek Jarcec Cecho commented on FLUME-1227:
---

I believe that we agreed in FLUME-1045 to create an separate channel that is 
actually described in this JIRA. So I believe that solving this issue will also 
solve FLUME-1045. Does that make sense?

Jarcec

> Introduce some sort of SpillableChannel
> ---
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
>  Issue Type: New Feature
>  Components: Channel
>Reporter: Jarek Jarcec Cecho
>Assignee: Patrick Wendell
>
> I would like to introduce new channel that would behave similarly as scribe 
> (https://github.com/facebook/scribe). It would be something between memory 
> and file channel. Input events would be saved directly to the memory (only) 
> and would be served from there. In case that the memory would be full, we 
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend 
> servers that are generating events. We want to send all events to just 
> limited number of machines from where we would send the data to HDFS (some 
> sort of staging layer). Reason for this second layer is our need to decouple 
> event aggregation and front end code to separate machines. Using memory 
> channel is fully sufficient as we can survive lost of some portion of the 
> events. However in order to sustain maintenance windows or networking issues 
> we would have to end up with a lot of memory assigned to those "staging" 
> machines. Referenced "scribe" is dealing with this problem by implementing 
> following logic - events are saved in memory similarly as our MemoryChannel. 
> However in case that the memory gets full (because of maintenance, networking 
> issues, ...) it will spill data to disk where they will be sitting until 
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's 
> durability guarantees would be same as MemoryChannel - in case that someone 
> would remove power cord, this channel would lose data. Based on the 
> discussion in FLUME-1201, I would propose to have the implementation 
> completely independent on any other channel internal code.
> Jarcec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1045) Proposal to support disk based spooling

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430529#comment-13430529
 ] 

Jarek Jarcec Cecho commented on FLUME-1045:
---

@Venkatesh - I did not consider it as an non-solvable problem. I was actually 
planning to add some generic methods about capacity and fulfillment of a 
channel to main Channel interface. What I wanted to express is that it's not 
easily doable at the moment - which is actually a reason why I did not yet 
started and let Patrik to handle it.

Jacec

> Proposal to support disk based spooling
> ---
>
> Key: FLUME-1045
> URL: https://issues.apache.org/jira/browse/FLUME-1045
> Project: Flume
>  Issue Type: New Feature
>Affects Versions: v1.0.0
>Reporter: Inder SIngh
>Priority: Minor
>  Labels: patch
> Attachments: FLUME-1045-1.patch, FLUME-1045-2.patch
>
>
> 1. Problem Description 
> A sink being unavailable at any stage in the pipeline causes it to back-off 
> and retry after a while. Channel's associated with such sinks start buffering 
> data with the caveat that if you are using a memory channel it can result in 
> a domino effect on the entire pipeline. There could be legitimate down times 
> eg: HDFS sink being down for name node maintenance, hadoop upgrades. 
> 2. Why not use a durable channel (JDBC, FileChannel)?
> Want high throughput and support sink down times as a first class use-case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1432) FileChannel should replay logs in the order they were written

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430516#comment-13430516
 ] 

Hari Shreedharan commented on FLUME-1432:
-

Brock - does this change the file channel format? Would older channels have to 
be completely drained before starting an agent running the new channel or is it 
backward compatible? I haven't looked at the code you posted yet - I apologize 
for that.

> FileChannel should replay logs in the order they were written
> -
>
> Key: FLUME-1432
> URL: https://issues.apache.org/jira/browse/FLUME-1432
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Brock Noland
>Assignee: Brock Noland
>
> Currently we replay the logs one at a time causing us to build large queue of 
> pending takes. Additionally, there maybe scenerios where this simply will not 
> work. Take a queue which is full (via checkpoint) and two files:
> 1:
> put
> commit
> put
> commit
> 2:
> take
> commit
> take
> commit
> take
> commit
> Replaying these logs in the current form will not work because we will we try 
> and reply the puts first and exceed our queue size. For these reasons, we 
> should replay them in the order they were written.
> However, at present there is no way to do this. Currently we have two 
> identifers in each record we write, a transaction id and a timestamp. Neither 
> can be used in replaying logs in order because the transaction id is created 
> when we create the transaction not when we write to the log. Someone could 
> create transaction, sleep, and then do work. The timestamp its not granular 
> enough as we could have duplicates.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: FLUME-1425: Create Standalone Spooling Client

[Hari Shreedharan]

> On Aug. 7, 2012, 7:09 a.m., Jarek Cecho wrote:
> > Hi Patrik,
> > thank you very much for your patch. I would personally prefer if it would 
> > be implemented as a source (I do understand windows incompatibility at the 
> > moment). However would be great to hear other opinions.
> > 
> > I was thinking about use cases of such functionality and I've come with an 
> > improvement idea. Do you think that you could allow user to specify header 
> > name where flume will automatically populate original file name? For 
> > example if user specifies header name to "file" and configuration directory 
> > as /directory than flume will populate header "file" with content 
> > "cool.txt" for each line of "/directory/cool.txt" file.
> 
> Patrick Wendell wrote:
> Let me re-implement this as a source, and maybe we can find a way to also 
> include it in the avro client.
> 
> The filename suggestion is also a good idea. I'll plan to add this in the 
> next rev of the patch. We could also add that as a feature and do in a second 
> patch depending on how much review bandwidth you have.

Patrick & Jarcec,

Not to interfere in this conversation, but would it be possible if this is 
implemented as a library of some sort, which can be used by a source and a 
client? That way we could have a client which supports this feature as well. 
Just a suggestion, upto both of you to decide :-)

Thanks!


- Hari


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6377/#review9950
---


On Aug. 4, 2012, 6:59 a.m., Patrick Wendell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6377/
> ---
> 
> (Updated Aug. 4, 2012, 6:59 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> This patch extends the existing avro client to support a file-based spooling 
> mechanism. See in-line documentation for precise details, but the basic idea 
> is that a user can have a spool directory where files are deposited for 
> ingestion into flume. Once ingested the files are clearly renamed and the 
> implementation guarantees at-least-once delivery semantics similar to those 
> achieved within flume itself, even across failures and restarts of the JVM 
> running the code.
> 
> I feel vaguely uneasy about building this as part of the standlone avro 
> client rather than as its own source. An alternative would be to build this 
> as a proper source (in fact, there are some ad-hoc transaction semantics used 
> here which would really be a better fit for a source). Interested in hearing 
> feedback on that as well. The benefit of having this in the avro client is 
> that you don't need the flume runner scripts which are not windows compatible.
> 
> 
> This addresses bug FlUME-1425.
> https://issues.apache.org/jira/browse/FlUME-1425
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 
> 4a5ecae 
>   
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
>  PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 
> PRE-CREATION 
>   
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/6377/diff/
> 
> 
> Testing
> ---
> 
> Extensive unit tests and I also built and played with this using a stub flume 
> agent. If you look at the JIRA I have a configuration file for an agent that 
> will print out Avro events to the command line - that's helpful when testing 
> this.
> 
> 
> Thanks,
> 
> Patrick Wendell
> 
>

[jira] [Commented] (FLUME-1418) Improvement for Log4j configuration

[Mike Percy (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430498#comment-13430498
 ] 

Mike Percy commented on FLUME-1418:
---

(Note that there were ^M characters at the end of the lines in this patch which 
I manually removed.)

> Improvement for Log4j configuration
> ---
>
> Key: FLUME-1418
> URL: https://issues.apache.org/jira/browse/FLUME-1418
> Project: Flume
>  Issue Type: Improvement
>  Components: Build
>Affects Versions: v1.2.0
>Reporter: Denny Ye
>Assignee: Denny Ye
>  Labels: log4j
> Fix For: v1.3.0
>
> Attachments: FLUME-1418.patch
>
>
> Updated log4j.properties 
> 1. Add 'logs' folder. It's better to aggregate logs into unified folder.
> 2. Log file using DailyRollingFileAppender
> 3. More useful log pattern. Add thread and code line information
> 4. Set debug level for Hadoop. If we use debug level, Hadoop log takes too 
> many lines.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1045) Proposal to support disk based spooling

[Seetharam Venkatesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430489#comment-13430489
 ] 

Seetharam Venkatesh commented on FLUME-1045:


IMHO, what Sharad outlines is a very reasonable approach and should not be hard 
to tie things together. 

bq.  I'm just worried here that current channel interface do not expose number 
of stored entries. So questions like "is memory channel full" or "do file 
channel contains less than memory channel capacity might not work
Jarcec, why do you consider this a problem? Do you consider this leaky? Metrics 
on top of the channel is necessary IMO and is not leaky.

> Proposal to support disk based spooling
> ---
>
> Key: FLUME-1045
> URL: https://issues.apache.org/jira/browse/FLUME-1045
> Project: Flume
>  Issue Type: New Feature
>Affects Versions: v1.0.0
>Reporter: Inder SIngh
>Priority: Minor
>  Labels: patch
> Attachments: FLUME-1045-1.patch, FLUME-1045-2.patch
>
>
> 1. Problem Description 
> A sink being unavailable at any stage in the pipeline causes it to back-off 
> and retry after a while. Channel's associated with such sinks start buffering 
> data with the caveat that if you are using a memory channel it can result in 
> a domino effect on the entire pipeline. There could be legitimate down times 
> eg: HDFS sink being down for name node maintenance, hadoop upgrades. 
> 2. Why not use a durable channel (JDBC, FileChannel)?
> Want high throughput and support sink down times as a first class use-case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

[Seetharam Venkatesh (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430470#comment-13430470
 ] 

Seetharam Venkatesh commented on FLUME-1227:


Does this mean there is no effort going into FLUME-1045?

> Introduce some sort of SpillableChannel
> ---
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
>  Issue Type: New Feature
>  Components: Channel
>Reporter: Jarek Jarcec Cecho
>Assignee: Patrick Wendell
>
> I would like to introduce new channel that would behave similarly as scribe 
> (https://github.com/facebook/scribe). It would be something between memory 
> and file channel. Input events would be saved directly to the memory (only) 
> and would be served from there. In case that the memory would be full, we 
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend 
> servers that are generating events. We want to send all events to just 
> limited number of machines from where we would send the data to HDFS (some 
> sort of staging layer). Reason for this second layer is our need to decouple 
> event aggregation and front end code to separate machines. Using memory 
> channel is fully sufficient as we can survive lost of some portion of the 
> events. However in order to sustain maintenance windows or networking issues 
> we would have to end up with a lot of memory assigned to those "staging" 
> machines. Referenced "scribe" is dealing with this problem by implementing 
> following logic - events are saved in memory similarly as our MemoryChannel. 
> However in case that the memory gets full (because of maintenance, networking 
> issues, ...) it will spill data to disk where they will be sitting until 
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's 
> durability guarantees would be same as MemoryChannel - in case that someone 
> would remove power cord, this channel would lose data. Based on the 
> discussion in FLUME-1201, I would propose to have the implementation 
> completely independent on any other channel internal code.
> Jarcec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: FLUME-1425: Create Standalone Spooling Client

[Patrick Wendell]

> On Aug. 7, 2012, 7:09 a.m., Jarek Cecho wrote:
> > Hi Patrik,
> > thank you very much for your patch. I would personally prefer if it would 
> > be implemented as a source (I do understand windows incompatibility at the 
> > moment). However would be great to hear other opinions.
> > 
> > I was thinking about use cases of such functionality and I've come with an 
> > improvement idea. Do you think that you could allow user to specify header 
> > name where flume will automatically populate original file name? For 
> > example if user specifies header name to "file" and configuration directory 
> > as /directory than flume will populate header "file" with content 
> > "cool.txt" for each line of "/directory/cool.txt" file.

Let me re-implement this as a source, and maybe we can find a way to also 
include it in the avro client.

The filename suggestion is also a good idea. I'll plan to add this in the next 
rev of the patch. We could also add that as a feature and do in a second patch 
depending on how much review bandwidth you have.


- Patrick


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6377/#review9950
---


On Aug. 4, 2012, 6:59 a.m., Patrick Wendell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6377/
> ---
> 
> (Updated Aug. 4, 2012, 6:59 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> This patch extends the existing avro client to support a file-based spooling 
> mechanism. See in-line documentation for precise details, but the basic idea 
> is that a user can have a spool directory where files are deposited for 
> ingestion into flume. Once ingested the files are clearly renamed and the 
> implementation guarantees at-least-once delivery semantics similar to those 
> achieved within flume itself, even across failures and restarts of the JVM 
> running the code.
> 
> I feel vaguely uneasy about building this as part of the standlone avro 
> client rather than as its own source. An alternative would be to build this 
> as a proper source (in fact, there are some ad-hoc transaction semantics used 
> here which would really be a better fit for a source). Interested in hearing 
> feedback on that as well. The benefit of having this in the avro client is 
> that you don't need the flume runner scripts which are not windows compatible.
> 
> 
> This addresses bug FlUME-1425.
> https://issues.apache.org/jira/browse/FlUME-1425
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 
> 4a5ecae 
>   
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
>  PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 
> PRE-CREATION 
>   
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/6377/diff/
> 
> 
> Testing
> ---
> 
> Extensive unit tests and I also built and played with this using a stub flume 
> agent. If you look at the JIRA I have a configuration file for an agent that 
> will print out Avro events to the command line - that's helpful when testing 
> this.
> 
> 
> Thanks,
> 
> Patrick Wendell
> 
>

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430402#comment-13430402
 ] 

Jarek Jarcec Cecho commented on FLUME-1227:
---

Good luck with coding Patrik - I'll happy to help with review!

Jarcec

> Introduce some sort of SpillableChannel
> ---
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
>  Issue Type: New Feature
>  Components: Channel
>Reporter: Jarek Jarcec Cecho
>Assignee: Patrick Wendell
>
> I would like to introduce new channel that would behave similarly as scribe 
> (https://github.com/facebook/scribe). It would be something between memory 
> and file channel. Input events would be saved directly to the memory (only) 
> and would be served from there. In case that the memory would be full, we 
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend 
> servers that are generating events. We want to send all events to just 
> limited number of machines from where we would send the data to HDFS (some 
> sort of staging layer). Reason for this second layer is our need to decouple 
> event aggregation and front end code to separate machines. Using memory 
> channel is fully sufficient as we can survive lost of some portion of the 
> events. However in order to sustain maintenance windows or networking issues 
> we would have to end up with a lot of memory assigned to those "staging" 
> machines. Referenced "scribe" is dealing with this problem by implementing 
> following logic - events are saved in memory similarly as our MemoryChannel. 
> However in case that the memory gets full (because of maintenance, networking 
> issues, ...) it will spill data to disk where they will be sitting until 
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's 
> durability guarantees would be same as MemoryChannel - in case that someone 
> would remove power cord, this channel would lose data. Based on the 
> discussion in FLUME-1201, I would propose to have the implementation 
> completely independent on any other channel internal code.
> Jarcec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Comment Edited] (FLUME-1227) Introduce some sort of SpillableChannel

[Jarek Jarcec Cecho (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429971#comment-13429971
 ] 

Jarek Jarcec Cecho edited comment on FLUME-1227 at 8/7/12 4:04 PM:
---

Thanks Denny and Patrick for support!

This ticket is currently assigned to me and I still do have intent to work on 
it. However I doubt that I'll have time for at least month and half. So please, 
if anyone have interest in implementing this feature, feel free to reassign 
this JIRA.

Jarcec

  was (Author: jarcec):
Thanks Denny and Patrick for support!

This ticket is currently assigned to me and I do have intent to work on it. 
However I doubt that I'll have to start working for at least month and half. So 
please, if anyone have interest in implementing this feature, feel free to 
reassign this JIRA.

Jarcec
  
> Introduce some sort of SpillableChannel
> ---
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
>  Issue Type: New Feature
>  Components: Channel
>Reporter: Jarek Jarcec Cecho
>Assignee: Patrick Wendell
>
> I would like to introduce new channel that would behave similarly as scribe 
> (https://github.com/facebook/scribe). It would be something between memory 
> and file channel. Input events would be saved directly to the memory (only) 
> and would be served from there. In case that the memory would be full, we 
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend 
> servers that are generating events. We want to send all events to just 
> limited number of machines from where we would send the data to HDFS (some 
> sort of staging layer). Reason for this second layer is our need to decouple 
> event aggregation and front end code to separate machines. Using memory 
> channel is fully sufficient as we can survive lost of some portion of the 
> events. However in order to sustain maintenance windows or networking issues 
> we would have to end up with a lot of memory assigned to those "staging" 
> machines. Referenced "scribe" is dealing with this problem by implementing 
> following logic - events are saved in memory similarly as our MemoryChannel. 
> However in case that the memory gets full (because of maintenance, networking 
> issues, ...) it will spill data to disk where they will be sitting until 
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's 
> durability guarantees would be same as MemoryChannel - in case that someone 
> would remove power cord, this channel would lose data. Based on the 
> discussion in FLUME-1201, I would propose to have the implementation 
> completely independent on any other channel internal code.
> Jarcec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1227) Introduce some sort of SpillableChannel

[Patrick Wendell (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430386#comment-13430386
 ] 

Patrick Wendell commented on FLUME-1227:


I grabbed it, thanks! Will take a look at discussion in the related JIRA as 
well.

> Introduce some sort of SpillableChannel
> ---
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
>  Issue Type: New Feature
>  Components: Channel
>Reporter: Jarek Jarcec Cecho
>Assignee: Patrick Wendell
>
> I would like to introduce new channel that would behave similarly as scribe 
> (https://github.com/facebook/scribe). It would be something between memory 
> and file channel. Input events would be saved directly to the memory (only) 
> and would be served from there. In case that the memory would be full, we 
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend 
> servers that are generating events. We want to send all events to just 
> limited number of machines from where we would send the data to HDFS (some 
> sort of staging layer). Reason for this second layer is our need to decouple 
> event aggregation and front end code to separate machines. Using memory 
> channel is fully sufficient as we can survive lost of some portion of the 
> events. However in order to sustain maintenance windows or networking issues 
> we would have to end up with a lot of memory assigned to those "staging" 
> machines. Referenced "scribe" is dealing with this problem by implementing 
> following logic - events are saved in memory similarly as our MemoryChannel. 
> However in case that the memory gets full (because of maintenance, networking 
> issues, ...) it will spill data to disk where they will be sitting until 
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's 
> durability guarantees would be same as MemoryChannel - in case that someone 
> would remove power cord, this channel would lose data. Based on the 
> discussion in FLUME-1201, I would propose to have the implementation 
> completely independent on any other channel internal code.
> Jarcec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (FLUME-1227) Introduce some sort of SpillableChannel

[Patrick Wendell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-1227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Wendell reassigned FLUME-1227:
--

Assignee: Patrick Wendell  (was: Jarek Jarcec Cecho)

> Introduce some sort of SpillableChannel
> ---
>
> Key: FLUME-1227
> URL: https://issues.apache.org/jira/browse/FLUME-1227
> Project: Flume
>  Issue Type: New Feature
>  Components: Channel
>Reporter: Jarek Jarcec Cecho
>Assignee: Patrick Wendell
>
> I would like to introduce new channel that would behave similarly as scribe 
> (https://github.com/facebook/scribe). It would be something between memory 
> and file channel. Input events would be saved directly to the memory (only) 
> and would be served from there. In case that the memory would be full, we 
> would outsource the events to file.
> Let me describe the use case behind this request. We have plenty of frontend 
> servers that are generating events. We want to send all events to just 
> limited number of machines from where we would send the data to HDFS (some 
> sort of staging layer). Reason for this second layer is our need to decouple 
> event aggregation and front end code to separate machines. Using memory 
> channel is fully sufficient as we can survive lost of some portion of the 
> events. However in order to sustain maintenance windows or networking issues 
> we would have to end up with a lot of memory assigned to those "staging" 
> machines. Referenced "scribe" is dealing with this problem by implementing 
> following logic - events are saved in memory similarly as our MemoryChannel. 
> However in case that the memory gets full (because of maintenance, networking 
> issues, ...) it will spill data to disk where they will be sitting until 
> everything start working again.
> I would like to introduce channel that would implement similar logic. It's 
> durability guarantees would be same as MemoryChannel - in case that someone 
> would remove power cord, this channel would lose data. Based on the 
> discussion in FLUME-1201, I would propose to have the implementation 
> completely independent on any other channel internal code.
> Jarcec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Issues with flume while trying to transfer big size files [1GB every hour]

[Hari Shreedharan]

Chandrashekhar,  

Is there a specific reason you cannot use Flume NG. I believe that has already 
been suggested on this thread. Flume OG is no longer actively developed, and it 
is better to move to Flume NG - which had its second release a few weeks back - 
Flume v 1.2.0. 


Thanks
Hari

-- 
Hari Shreedharan


On Tuesday, August 7, 2012 at 12:16 AM, Chandrashekhar Shaw wrote:

> Hi
> 
> I was testing flume for log transfer in DFO mode. I was firing the single 
> shot command as cron every 10th minute from the agent box. So the flume 
> process starts every 10 minutes sends the logs to collector box and ends.
> 
> It was pushing the logs very well. To test the reliability I stopped the 
> collector and brought it back after 1 hour, what I see is
> 
> * The flume process (started by the agent cron) kept on running in the agent 
> machine and not able to transfer the rest of the file to collector.
> * The process did not end even after 5/6 hours. (I had to manually kill it).
> 
> As I know flume agent should continue sending the log once the collector is 
> re-started. ???
> Is there something I am missing ??
> 
> Thanks
> Chandrashekhar Shaw
> 
> 
> 
> 
> On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:
> 
> Hello,
> 
> Not too sure about Flume OG - I barely used it for a week before shifting to 
> NG. Would there be any particular reason you are using OG?
> 
> Yes, you can take a look at SyslogTcpSource (in NG), 
> https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html
> 
> This can be modified according to this Netty example (for asynchronous large 
> file streaming) : 
> http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html
> 
> This is pretty much what I did when I needed a Flume HTTP source to match my 
> requirements.
> 
> Regards,
> Harish
> 
> 
> On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw 
> mailto:sha...@yahoo-inc.com>> wrote:
> I am using Flume - OG 
> https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz
>  .
> 
> It there an available version which implements netty??
> 
> Thanks
> Chandrashekhar Shaw
> 
> 
> On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:
> 
> Hi,
> 
> Which one of the sources are you using? Maybe a flume source that
> implements something like
> http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.
> 
> Regards,
> Harish
> 
> 
> On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
> mailto:sha...@yahoo-inc.com>>wrote:
> 
> Hi,
> 
> I am using flume for transferring logs from 20 agent boxes to 1 collector
> box. The file size which I am trying to transfer is of about 1 GB per hour.
> So every hour my agents run to transfer the log file generated during the
> last hour.
> I am using DFO sink for file transfer.
> 
> But then I am facing following problems:
> 
> 1) Some of the agents are not able to transfer the whole file even in one
> hour and the processes run continuously for hours.
> 2) Some of the agents stop within 5-6 minutes without sending the file
> completely.
> 
> I want to know
> 
> 1) if somebody else has faced the same issues?
> 2) What could be the problem?
> 
> Thanks
> Chandrashekhar Shaw
> 
> 

[jira] [Commented] (FLUME-1424) File Channel should support encryption

[Arvind Prabhakar (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992
 ] 

Arvind Prabhakar commented on FLUME-1424:
-

Yes, the put records do store the data in them. We can perhaps start with that 
as a first step and if more requirements pop-up, we can address them in 
follow-up Jiras as necessary.

> File Channel should support encryption
> --
>
> Key: FLUME-1424
> URL: https://issues.apache.org/jira/browse/FLUME-1424
> Project: Flume
>  Issue Type: Bug
>Reporter: Arvind Prabhakar
>Assignee: Arvind Prabhakar
>
> When persisting the data to disk, the File Channel should allow some form of 
> encryption to ensure safety of data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Issues with flume while trying to transfer big size files [1GB every hour]

[Chandrashekhar Shaw]

Hi

I was testing flume for log transfer in DFO mode. I was firing the single shot 
command as cron every 10th minute from the agent box. So the flume process 
starts every 10 minutes sends the logs to collector box and ends.

It was pushing the logs very well. To test the reliability I stopped the 
collector and brought it back after 1 hour, what I see is

* The flume process (started by the agent cron) kept on running in the agent 
machine and not able to transfer the rest of the file to collector.
* The process did not end even after 5/6 hours. (I had to manually kill it).

As I know flume agent should continue sending the log once the collector is 
re-started. ???
Is there something I am missing ??

Thanks
Chandrashekhar Shaw




On Jul 24, 2012, at 6:50 PM, Harish Mandala wrote:

Hello,

Not too sure about Flume OG - I barely used it for a week before shifting to 
NG. Would there be any particular reason you are using OG?

Yes, you can take a look at SyslogTcpSource (in NG), 
https://people.apache.org/~mpercy/flume/flume-1.2.0-incubating-SNAPSHOT/docs/apidocs/org/apache/flume/source/SyslogTcpSource.html

This can be modified according to this Netty example (for asynchronous large 
file streaming) : 
http://static.netty.io/3.5/xref/org/jboss/netty/example/http/file/package-summary.html

This is pretty much what I did when I needed a Flume HTTP source to match my 
requirements.

Regards,
Harish


On Tue, Jul 24, 2012 at 3:02 AM, Chandrashekhar Shaw 
mailto:sha...@yahoo-inc.com>> wrote:
I am using Flume - OG   
https://github.com/downloads/cloudera/flume/flume-distribution-0.9.4-bin.tar.gz 
.

It there an available version which implements netty??

Thanks
Chandrashekhar Shaw


On Jul 23, 2012, at 8:00 PM, Harish Mandala wrote:

Hi,

Which one of the sources are you using? Maybe a flume source that
implements something like
http://docs.jboss.org/netty/3.2/xref/org/jboss/netty could be useful.

Regards,
Harish


On Mon, Jul 23, 2012 at 7:52 AM, Chandrashekhar Shaw
mailto:sha...@yahoo-inc.com>>wrote:

Hi,

I am using flume for transferring logs from 20 agent boxes to 1 collector
box. The file size which I am trying to transfer is of about 1 GB per hour.
So every hour my agents run to transfer the log file generated during the
last hour.
I am using DFO sink for file transfer.

But then I am facing following problems:

1) Some of the agents are not able to transfer the whole file even in one
hour and the processes run continuously for hours.
2) Some of the agents stop within 5-6 minutes without sending the file
completely.

I want to know

1) if somebody else has faced the same issues?
2) What could be the problem?

Thanks
Chandrashekhar Shaw

Re: Review Request: FLUME-1425: Create Standalone Spooling Client

[Jarek Cecho]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6377/#review9950
---


Hi Patrik,
thank you very much for your patch. I would personally prefer if it would be 
implemented as a source (I do understand windows incompatibility at the 
moment). However would be great to hear other opinions.

I was thinking about use cases of such functionality and I've come with an 
improvement idea. Do you think that you could allow user to specify header name 
where flume will automatically populate original file name? For example if user 
specifies header name to "file" and configuration directory as /directory than 
flume will populate header "file" with content "cool.txt" for each line of 
"/directory/cool.txt" file.


flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java


Nit: Could you please change the tab characters to spaces here?


Jarcec

- Jarek Cecho


On Aug. 4, 2012, 6:59 a.m., Patrick Wendell wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6377/
> ---
> 
> (Updated Aug. 4, 2012, 6:59 a.m.)
> 
> 
> Review request for Flume.
> 
> 
> Description
> ---
> 
> This patch extends the existing avro client to support a file-based spooling 
> mechanism. See in-line documentation for precise details, but the basic idea 
> is that a user can have a spool directory where files are deposited for 
> ingestion into flume. Once ingested the files are clearly renamed and the 
> implementation guarantees at-least-once delivery semantics similar to those 
> achieved within flume itself, even across failures and restarts of the JVM 
> running the code.
> 
> I feel vaguely uneasy about building this as part of the standlone avro 
> client rather than as its own source. An alternative would be to build this 
> as a proper source (in fact, there are some ad-hoc transaction semantics used 
> here which would really be a better fit for a source). Interested in hearing 
> feedback on that as well. The benefit of having this in the avro client is 
> that you don't need the flume runner scripts which are not windows compatible.
> 
> 
> This addresses bug FlUME-1425.
> https://issues.apache.org/jira/browse/FlUME-1425
> 
> 
> Diffs
> -
> 
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/AvroCLIClient.java 
> 4a5ecae 
>   
> flume-ng-core/src/main/java/org/apache/flume/client/avro/BufferedLineReader.java
>  PRE-CREATION 
>   flume-ng-core/src/main/java/org/apache/flume/client/avro/LineReader.java 
> PRE-CREATION 
>   
> flume-ng-core/src/main/java/org/apache/flume/client/avro/SpoolingFileLineReader.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestBufferedLineReader.java
>  PRE-CREATION 
>   
> flume-ng-core/src/test/java/org/apache/flume/client/avro/TestSpoolingFileLineReader.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/6377/diff/
> 
> 
> Testing
> ---
> 
> Extensive unit tests and I also built and played with this using a stub flume 
> agent. If you look at the JIRA I have a configuration file for an agent that 
> will print out Avro events to the command line - that's helpful when testing 
> this.
> 
> 
> Thanks,
> 
> Patrick Wendell
> 
>

[jira] [Assigned] (FLUME-1428) File Channel should not consider a file as inactive until all takes are committed.

[Hari Shreedharan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLUME-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hari Shreedharan reassigned FLUME-1428:
---

Assignee: Hari Shreedharan

> File Channel should not consider a file as inactive until all takes are 
> committed.
> --
>
> Key: FLUME-1428
> URL: https://issues.apache.org/jira/browse/FLUME-1428
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Hari Shreedharan
>Assignee: Hari Shreedharan
> Fix For: v1.3.0
>
>
> FlumeEventQueue removes fileID from fileIDCounts before commits are 
> completed. See FLUME-1417 for background. This series of events would be 
> problematic:
> File ( i )
> n puts
> commit
> n takes (file i is no longer in flume event queue fileIDCounts)
> —
> file roll -> current active file in directory (File i+1)
> background worker kicks in -> removes file i.
> Note that the commit for the n takes has not taken place. 
> We do keep 2 files per directory, but if a commit/rollback does not come and 
> 2 files are rolled, we might have some issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (FLUME-1428) File Channel should not consider a file as inactive until all takes are committed.

[Hari Shreedharan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429982#comment-13429982
 ] 

Hari Shreedharan commented on FLUME-1428:
-

Assigning to myself for now. I am trying a couple of methods:
1. Move the reference counting to the Log class, and pass it to the queue while 
checkpointing, and retrieve from queue before replaying. Replay handler 
provides the refcounts from the replay and the log class applies this delta to 
the original values in the checkpoint.

2. If 1 is too complex and causes issues, I am going to get something checked 
in which will work reasonably fine. Include a configuration variable which sets 
a cleanUpInterval and delete the "inactive" files after the clean up interval. 
The downside is there might be extra "old" data, but this will work around the 
current problem. Not yet sure how this will work on restarts - shutdown and 
restart after several hours.

> File Channel should not consider a file as inactive until all takes are 
> committed.
> --
>
> Key: FLUME-1428
> URL: https://issues.apache.org/jira/browse/FLUME-1428
> Project: Flume
>  Issue Type: Bug
>  Components: Channel
>Affects Versions: v1.2.0
>Reporter: Hari Shreedharan
> Fix For: v1.3.0
>
>
> FlumeEventQueue removes fileID from fileIDCounts before commits are 
> completed. See FLUME-1417 for background. This series of events would be 
> problematic:
> File ( i )
> n puts
> commit
> n takes (file i is no longer in flume event queue fileIDCounts)
> —
> file roll -> current active file in directory (File i+1)
> background worker kicks in -> removes file i.
> Note that the commit for the n takes has not taken place. 
> We do keep 2 files per directory, but if a commit/rollback does not come and 
> 2 files are rolled, we might have some issues.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira