Re: [DISCUSS] Release Flume 1.5.0
+1, There are some minor doc patches pending for review, is it possible to include them? On Fri, Jan 31, 2014 at 3:31 AM, Jeff Lord wrote: > +1 for a release > +1 for resuming the contrib discussion > > > On Thu, Jan 30, 2014 at 12:27 PM, Wolfgang Hoschek >wrote: > > > +1 There a many important new features and fixes ready to go. > > > > Wolfgang. > > > > On Jan 30, 2014, at 7:43 PM, Chiwan Park wrote: > > > > > +1 on new release! > > > > > > -- > > > Regards, > > > Chiwan Park > > > > > > On Jan 31, 2014, at 2:17 AM, Hari Shreedharan < > hshreedha...@cloudera.com> > > wrote: > > > > > >> Hi folks, > > >> > > >> It has been about 6 months since we did a release. We have added > several > > >> new features and fixed a lot of bugs. What do you guys think about > > >> releasing Flume 1.5.0? > > >> > > >> > > >> Thanks > > >> Hari > > > > > > > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [DISCUSS] Release Flume 1.5.0
+1 for a release +1 for resuming the contrib discussion On Thu, Jan 30, 2014 at 12:27 PM, Wolfgang Hoschek wrote: > +1 There a many important new features and fixes ready to go. > > Wolfgang. > > On Jan 30, 2014, at 7:43 PM, Chiwan Park wrote: > > > +1 on new release! > > > > -- > > Regards, > > Chiwan Park > > > > On Jan 31, 2014, at 2:17 AM, Hari Shreedharan > wrote: > > > >> Hi folks, > >> > >> It has been about 6 months since we did a release. We have added several > >> new features and fixed a lot of bugs. What do you guys think about > >> releasing Flume 1.5.0? > >> > >> > >> Thanks > >> Hari > > > >
Re: [DISCUSS] Release Flume 1.5.0
+1 There a many important new features and fixes ready to go. Wolfgang. On Jan 30, 2014, at 7:43 PM, Chiwan Park wrote: > +1 on new release! > > -- > Regards, > Chiwan Park > > On Jan 31, 2014, at 2:17 AM, Hari Shreedharan > wrote: > >> Hi folks, >> >> It has been about 6 months since we did a release. We have added several >> new features and fixed a lot of bugs. What do you guys think about >> releasing Flume 1.5.0? >> >> >> Thanks >> Hari >
[jira] [Commented] (FLUME-2306) onIdleCallback is not canceled when stop hdfs sink
[
https://issues.apache.org/jira/browse/FLUME-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886985#comment-13886985
]
Hari Shreedharan commented on FLUME-2306:
-
We already (before FLUME-2265) closed the bucket writer on rollInterval, since
we really did not know if more events would come into that bucket - so keeping
the bucket writer open didn't make sense. That behavior did not change - the
only change was that we removed a reference to an old bucket writer and created
a new one (BucketWriter#open is the more expensive call -- which had to be
called earlier too). We don't gain any significant advantage from keeping the
bucket writer - but that risks a memory leak if idle timeout is not set --
which IMO, is incorrect behavior since rollInterval should not force the user
to set idleTimeout too.
> onIdleCallback is not canceled when stop hdfs sink
> --
>
> Key: FLUME-2306
> URL: https://issues.apache.org/jira/browse/FLUME-2306
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.4.0
>Reporter: chenshangan
>
> the hdfs sink cached 5000 open files by default and it cost quite a lot of
> memory in total when using lzo CompressedStream. We should open the
> idleTimeout feature to resolve it. But there seems to be a bug with this
> feature. When stopping flume, HDFSWriter does not cancel the idle scheduler,
> which might cause flume not to stop. So I extend the current close() method
> in HDFSWriter as follows, and use it in HDFSEventSink when stop the sink
> component :
> /**
>* when stop flume, all schedulers should be canceled
>* @param cancelIdleCallback
>* @throws IOException
>* @throws InterruptedException
>*/
> public void close(boolean cancelIdleCallback) throws IOException,
> InterruptedException{
> close();
> if(cancelIdleCallback){
> if (idleFuture != null && !idleFuture.isDone()) {
> idleFuture.cancel(false); // do not cancel myself if running!
> idleFuture = null;
> }
> }
> }
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
Re: [DISCUSS] Release Flume 1.5.0
Agreed, we need more discussion on the contrib module and I would like to decouple that discussion from the release discussion. There are several fixes and features we should release soon, so I think we should go ahead with a release and not wait for the contrib discussion to come to a conclusion. Thanks, Hari On Thursday, January 30, 2014 at 10:03 AM, Chiwan Park wrote: > We need more discussion about contrib module (like coverage of core) and > new release is overdue, so I think focusing new release now is more important. > > -- > Regards, > Chiwan Park > > On Jan 31, 2014, at 2:31 AM, Jeremy Karlson (mailto:jeremykarl...@gmail.com)> wrote: > > > Since I enjoy stirring the pot... Do we want to come to a decision on the > > contrib thing before the release, so those things can be included (or not)? > > > > -- Jeremy > > > > > > > > On Thu, Jan 30, 2014 at 9:17 AM, Hari Shreedharan > > mailto:hshreedha...@cloudera.com) > > > wrote: > > > > > > > Hi folks, > > > > > > It has been about 6 months since we did a release. We have added several > > > new features and fixed a lot of bugs. What do you guys think about > > > releasing Flume 1.5.0? > > > > > > > > > Thanks > > > Hari > > > > > > > > > >
Re: [DISCUSS] Release Flume 1.5.0
We need more discussion about contrib module (like coverage of core) and new release is overdue, so I think focusing new release now is more important. -- Regards, Chiwan Park On Jan 31, 2014, at 2:31 AM, Jeremy Karlson wrote: > Since I enjoy stirring the pot... Do we want to come to a decision on the > contrib thing before the release, so those things can be included (or not)? > > -- Jeremy > > > > On Thu, Jan 30, 2014 at 9:17 AM, Hari Shreedharan > wrote: > >> Hi folks, >> >> It has been about 6 months since we did a release. We have added several >> new features and fixed a lot of bugs. What do you guys think about >> releasing Flume 1.5.0? >> >> >> Thanks >> Hari >>
Re: [DISCUSS] Release Flume 1.5.0
Thanks for bringing this up Hari. A new release for Flume is overdue in my opinion. Regards, Arvind Prabhakar On Thu, Jan 30, 2014 at 9:43 AM, Chiwan Park wrote: > +1 on new release! > > -- > Regards, > Chiwan Park > > On Jan 31, 2014, at 2:17 AM, Hari Shreedharan > wrote: > > > Hi folks, > > > > It has been about 6 months since we did a release. We have added several > > new features and fixed a lot of bugs. What do you guys think about > > releasing Flume 1.5.0? > > > > > > Thanks > > Hari > >
Re: [DISCUSS] Release Flume 1.5.0
+1 on new release! -- Regards, Chiwan Park On Jan 31, 2014, at 2:17 AM, Hari Shreedharan wrote: > Hi folks, > > It has been about 6 months since we did a release. We have added several > new features and fixed a lot of bugs. What do you guys think about > releasing Flume 1.5.0? > > > Thanks > Hari
Re: [DISCUSS] Release Flume 1.5.0
+1 Sent from Yahoo Mail on Android
Re: [DISCUSS] Release Flume 1.5.0
Since I enjoy stirring the pot... Do we want to come to a decision on the contrib thing before the release, so those things can be included (or not)? -- Jeremy On Thu, Jan 30, 2014 at 9:17 AM, Hari Shreedharan wrote: > Hi folks, > > It has been about 6 months since we did a release. We have added several > new features and fixed a lot of bugs. What do you guys think about > releasing Flume 1.5.0? > > > Thanks > Hari >
Re: [DISCUSS] Release Flume 1.5.0
Strong +1 on creating a new release. Jarcec On Thu, Jan 30, 2014 at 09:17:43AM -0800, Hari Shreedharan wrote: > Hi folks, > > It has been about 6 months since we did a release. We have added several > new features and fixed a lot of bugs. What do you guys think about > releasing Flume 1.5.0? > > > Thanks > Hari signature.asc Description: Digital signature
[DISCUSS] Release Flume 1.5.0
Hi folks, It has been about 6 months since we did a release. We have added several new features and fixed a lot of bugs. What do you guys think about releasing Flume 1.5.0? Thanks Hari
[jira] [Commented] (FLUME-2306) onIdleCallback is not canceled when stop hdfs sink
[
https://issues.apache.org/jira/browse/FLUME-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886693#comment-13886693
]
chenshangan commented on FLUME-2306:
[~hshreedharan]
It makes sense. My philosophy is that we don't need to create a new
BucketWriter every time when a file is closed as we may reuse the bucketwriter
and only roll files in the bucket. So will it be better that idleTimeout deals
with bucket but not the file in the bucket ?
> onIdleCallback is not canceled when stop hdfs sink
> --
>
> Key: FLUME-2306
> URL: https://issues.apache.org/jira/browse/FLUME-2306
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.4.0
>Reporter: chenshangan
>
> the hdfs sink cached 5000 open files by default and it cost quite a lot of
> memory in total when using lzo CompressedStream. We should open the
> idleTimeout feature to resolve it. But there seems to be a bug with this
> feature. When stopping flume, HDFSWriter does not cancel the idle scheduler,
> which might cause flume not to stop. So I extend the current close() method
> in HDFSWriter as follows, and use it in HDFSEventSink when stop the sink
> component :
> /**
>* when stop flume, all schedulers should be canceled
>* @param cancelIdleCallback
>* @throws IOException
>* @throws InterruptedException
>*/
> public void close(boolean cancelIdleCallback) throws IOException,
> InterruptedException{
> close();
> if(cancelIdleCallback){
> if (idleFuture != null && !idleFuture.isDone()) {
> idleFuture.cancel(false); // do not cancel myself if running!
> idleFuture = null;
> }
> }
> }
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
