[GitHub] flume pull request #72: Fix startup order for ExecSource

[mpercy]

GitHub user mpercy opened a pull request:

https://github.com/apache/flume/pull/72

Fix startup order for ExecSource

This patch starts the SourceCounter in the ExecSource before starting
the runner thread that processes input from the forked command.

Starting the SourceCounter before calling its methods is required to get
correct metric counts because the metrics are all reset to 0 when
SourceCounter.start() is called.

A follow-up patch will attempt to enforce that a SourceCounter or any
other MonitoredCounterGroup is running at the time that calls to modify
the metrics occur.

After applying this patch, TestExecSource.testMonitoredCounterGroup no
longer fails.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mpercy/flume flaky-exec-source-test-2a

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/72.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #72


commit b385dc76090287ce2a91d2982e8df77d64438d05
Author: Mike Percy 
Date:   2016-10-13T23:24:37Z

Fix startup order for ExecSource

This patch starts the SourceCounter in the ExecSource before starting
the runner thread that processes input from the forked command.

Starting the SourceCounter before calling its methods is required to get
correct metric counts because the metrics are all reset to 0 when
SourceCounter.start() is called.

A follow-up patch will attempt to enforce that a SourceCounter or any
other MonitoredCounterGroup is running at the time that calls to modify
the metrics occur.

After applying this patch, TestExecSource.testMonitoredCounterGroup no
longer fails.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLUME-2834) TIBCO JMS with SSL enable

[Kevin Tran (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573768#comment-15573768
 ] 

Kevin Tran commented on FLUME-2834:
---

Hi,
Have we had solution on this?

Thanks,
Kevin Tran.

> TIBCO JMS with SSL enable
> -
>
> Key: FLUME-2834
> URL: https://issues.apache.org/jira/browse/FLUME-2834
> Project: Flume
>  Issue Type: Question
>  Components: Sinks+Sources
>Affects Versions: v1.4.0
> Environment: linux
>Reporter: Sushma
>  Labels: jms, ssl
>
> I am trying to use JMS source with ssl enabled, but i am facing issues, below 
> is my config for the jms source ...
> agent.sources.mySrc.type = jms
> agent.sources.mySrc.initialContextFactory = 
> com.tibco.tibjms.naming.TibjmsInitialContextFactory
> agent.sources.mySrc.providerURL = tibjmsnaming://:
> agent.sources.mySrc.destinationName = MYQUEUE
> agent.sources.mySrc.destinationType = queue
> agent.sources.mySrc.batchSize=50
> agent.sources.mySrc.connectionFactory = QueueConnectionFactory
> agent.sources.mySrc.userName=user
> agent.sources.mySrc.passwordFile=/opt/flume/conf/password.txt
> agent.sources.mySrc.converter.type=org.apache.flume.source.jms.ELFJMSMessageEventUUIDConverter$Builder
> agent.sources.mySrc.selector.type = multiplexing
> agent.sources.mySrc.selector.header=zone
> agent.sources.mySrc.selector.optional.TIBCO = myChannel1 myChannel2
> agent.sources.mySrc.selector.optional.COMMON = myChannel1 
> # SSL properties for JMS source mySrc 
> agent.sources.mySrc.ssl=true
> agent.sources.mySrc.keystore=/opt/flume/conf/jmsSslKey.store
> agent.sources.mySrc.keystore-password=/opt/flume/conf/jmsPassword.txt
> agent.sources.mySrc.keystore-type= store



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Flume bechmarks

[Roshan Naik]

You may want to take a look at
- 
https://cwiki.apache.org/confluence/display/FLUME/Performance+Measurements+
-+round+2


and the older..
- 
https://cwiki.apache.org/confluence/display/FLUME/Flume+NG+Performance+Meas
urements


when coming up with a list of configurations to benchmark.

-roshan


On 10/13/16, 9:12 AM, "Balazs Donat Bessenyei"  wrote:

>I have just proposed enabling Travis on a different thread. That should
>help with this. (Having a separate machine would be best, but I don't know
>how we could get one. I'll do the homework for this.)
>
>On Oct 13, 2016 5:57 PM, "Lior Zeno"  wrote:
>
>> Maybe getting an isolated environment? The CI environment might be
>>shared
>> among multiple users, adding too much noise to the performance test.
>>
>> On Thu, Oct 13, 2016 at 6:53 PM, Balazs Donat Bessenyei <
>> bes...@cloudera.com
>> > wrote:
>>
>> > +1
>> >
>> > I think this is a good idea!
>> >
>> > How can I help with setting it up?
>> >
>> > On Oct 13, 2016 5:20 PM, "Lior Zeno"  wrote:
>> >
>> > > Hi All,
>> > >
>> > > Monitoring Flume's performance over time is an important step in
>>every
>> > > production-level application.  Benchmarking Flume on a nightly basis
>> has
>> > > the following advantages:
>> > >
>> > > * Better understanding of Flume's bottlenecks.
>> > > * Allow users to compare the performance of different solutions,
>>such
>> as
>> > > Logstash and Fluentd.
>> > > * Better understanding of the influence of recent commits on
>> performance.
>> > >
>> > > Logstash already conducts various performance tests, more details in
>> this
>> > > link:
>> > > http://logstash-benchmarks.elastic.co/
>> > >
>> > > I propose adding a few micro-benchmarks showing Flume's TPS vs date
>>(of
>> > > course, in the ideal case where the input and/or output do not
>> bottleneck
>> > > the system), e.g. using the SeqGen source.
>> > >
>> > > Thoughts?
>> > >
>> > > Thanks
>> > >
>> >
>>

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Hari Shreedharan]

+1 (binding)

Signatures and checksums look good
Top level files are all good.
Build runs fine, simple agent with Seq source, memory channel and HDFS
sink run fine as well.

On Thu, Oct 13, 2016 at 8:42 AM, Mike Percy  wrote:
> +1 (binding)
>
> There are some flaky tests which are listed below but I don't think they
> are release blockers.
>
> I performed the following checks:
>
> Binary convenience artifact:
> * Signature and checksums match
> * LICENSE, NOTICE, and README.md files in the binary convenience artifact
> look accurate and complete relative to the jars in lib/
> * Ran a very quick test with the binary artifact and it
> worked: ./bin/flume-ng agent -c conf -f
> conf/flume-conf.properties.template -n
> agent -Dflume.root.logger=DEBUG,console
> * Checked that the documentation in docs/ renders: Flume User Guide and
> Flume Dev Guide are OK. Also spot-checked that the new Kafka security
> documentation was included in the User Guide
>
> Source artifact:
> * Signature and checksums match
> * Built Flume from the source artifact using Oracle 1.7.0_80 on Ubuntu
> Linux 16.04, sanity tested the resulting binary using the above method and
> it worked
> * RAT checks passed
> * Built a new source artifact out of the official source artifact and
> compiled it
> * I ran the unit tests. Most passed but the below two failed. These are
> flaky tests (we have a bunch of them in Flume) so I think it's fine not to
> block the release on them.
>   * TestExecSource.testMonitoredCounterGroup - looks like a racy test
>   * TestSpillableMemoryChannel - didn't investigate
>
> RC2 looks good to me.
>
> Thanks for running this release, Donat!
>
> Mike
>
> On Wed, Oct 12, 2016 at 9:29 PM, Balazs Donat Bessenyei > wrote:
>
>> Hi All,
>>
>> This is the tenth release for Apache Flume as a top-level project,
>> version 1.7.0. We are voting on release candidate RC2.
>>
>> It fixes the following issues:
>>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>>
>> *** Please cast your vote within the next 72 hours ***
>>
>> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
>> for the source and binary artifacts can be found here:
>>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
>>
>> Maven staging repo:
>>   https://repository.apache.org/content/repositories/orgapacheflume-1020/
>>
>> The tag to be voted on:
>>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868
>>
>> Flume's KEYS file containing PGP keys we use to sign the release:
>>   https://www.apache.org/dist/flume/KEYS
>>
>>
>> Thank you,
>>
>> Donat
>>

Re: Flume bechmarks

[Balazs Donat Bessenyei]

I have just proposed enabling Travis on a different thread. That should
help with this. (Having a separate machine would be best, but I don't know
how we could get one. I'll do the homework for this.)

On Oct 13, 2016 5:57 PM, "Lior Zeno"  wrote:

> Maybe getting an isolated environment? The CI environment might be shared
> among multiple users, adding too much noise to the performance test.
>
> On Thu, Oct 13, 2016 at 6:53 PM, Balazs Donat Bessenyei <
> bes...@cloudera.com
> > wrote:
>
> > +1
> >
> > I think this is a good idea!
> >
> > How can I help with setting it up?
> >
> > On Oct 13, 2016 5:20 PM, "Lior Zeno"  wrote:
> >
> > > Hi All,
> > >
> > > Monitoring Flume's performance over time is an important step in every
> > > production-level application.  Benchmarking Flume on a nightly basis
> has
> > > the following advantages:
> > >
> > > * Better understanding of Flume's bottlenecks.
> > > * Allow users to compare the performance of different solutions, such
> as
> > > Logstash and Fluentd.
> > > * Better understanding of the influence of recent commits on
> performance.
> > >
> > > Logstash already conducts various performance tests, more details in
> this
> > > link:
> > > http://logstash-benchmarks.elastic.co/
> > >
> > > I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> > > course, in the ideal case where the input and/or output do not
> bottleneck
> > > the system), e.g. using the SeqGen source.
> > >
> > > Thoughts?
> > >
> > > Thanks
> > >
> >
>

Re: Enabling Travis-CI on Flume

[Lior Zeno]

Jenkins can do PRs as well. If we can upgrade Jenkins to 2.0, we will be
able to define the build step via Jenkinsfile which becomes very similar to
Travis.
Is there any reason to prefer Travis over Jenkins in our case?

On Thu, Oct 13, 2016 at 7:01 PM, Balazs Donat Bessenyei  wrote:

> Hi All,
>
> Having something that checks proposed patches (PR-s especially)
> automatically would help a lot with the development on Flume.
>
> I think, Travis-CI could be an easy solution and (afaik) we'd only have to
> ask infra to enable it for us.
>
> Please, let me know your thoughts.
>
> Thank you,
>
> Donat
>

Enabling Travis-CI on Flume

[Balazs Donat Bessenyei]

Hi All,

Having something that checks proposed patches (PR-s especially)
automatically would help a lot with the development on Flume.

I think, Travis-CI could be an easy solution and (afaik) we'd only have to
ask infra to enable it for us.

Please, let me know your thoughts.

Thank you,

Donat

Re: Flume bechmarks

[Lior Zeno]

Maybe getting an isolated environment? The CI environment might be shared
among multiple users, adding too much noise to the performance test.

On Thu, Oct 13, 2016 at 6:53 PM, Balazs Donat Bessenyei  wrote:

> +1
>
> I think this is a good idea!
>
> How can I help with setting it up?
>
> On Oct 13, 2016 5:20 PM, "Lior Zeno"  wrote:
>
> > Hi All,
> >
> > Monitoring Flume's performance over time is an important step in every
> > production-level application.  Benchmarking Flume on a nightly basis has
> > the following advantages:
> >
> > * Better understanding of Flume's bottlenecks.
> > * Allow users to compare the performance of different solutions, such as
> > Logstash and Fluentd.
> > * Better understanding of the influence of recent commits on performance.
> >
> > Logstash already conducts various performance tests, more details in this
> > link:
> > http://logstash-benchmarks.elastic.co/
> >
> > I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> > course, in the ideal case where the input and/or output do not bottleneck
> > the system), e.g. using the SeqGen source.
> >
> > Thoughts?
> >
> > Thanks
> >
>

Re: Flume bechmarks

[Lior Zeno]

I think that we can come up with an initial version with little efforts.
The simplest scenario I can think of is running a Flume instance (with a
SeqGen source and a Null sink) for one minute, and then report the average
events per second.

On Thu, Oct 13, 2016 at 6:43 PM, Attila Simon  wrote:

> Good idea! What would be required to set up something similar for Flume?
> ie initial time cost for setting up the infrastructure and periodic time
> cost to add new use-cases.
>
> Cheers,
> Attila
>
>
>
> On Thu, Oct 13, 2016 at 5:19 PM, Lior Zeno  wrote:
>
> > Hi All,
> >
> > Monitoring Flume's performance over time is an important step in every
> > production-level application.  Benchmarking Flume on a nightly basis has
> > the following advantages:
> >
> > * Better understanding of Flume's bottlenecks.
> > * Allow users to compare the performance of different solutions, such as
> > Logstash and Fluentd.
> > * Better understanding of the influence of recent commits on performance.
> >
> > Logstash already conducts various performance tests, more details in this
> > link:
> > http://logstash-benchmarks.elastic.co/
> >
> > I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> > course, in the ideal case where the input and/or output do not bottleneck
> > the system), e.g. using the SeqGen source.
> >
> > Thoughts?
> >
> > Thanks
> >
>

Re: Flume bechmarks

[Balazs Donat Bessenyei]

+1

I think this is a good idea!

How can I help with setting it up?

On Oct 13, 2016 5:20 PM, "Lior Zeno"  wrote:

> Hi All,
>
> Monitoring Flume's performance over time is an important step in every
> production-level application.  Benchmarking Flume on a nightly basis has
> the following advantages:
>
> * Better understanding of Flume's bottlenecks.
> * Allow users to compare the performance of different solutions, such as
> Logstash and Fluentd.
> * Better understanding of the influence of recent commits on performance.
>
> Logstash already conducts various performance tests, more details in this
> link:
> http://logstash-benchmarks.elastic.co/
>
> I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> course, in the ideal case where the input and/or output do not bottleneck
> the system), e.g. using the SeqGen source.
>
> Thoughts?
>
> Thanks
>

Re: Flume bechmarks

[Attila Simon]

Good idea! What would be required to set up something similar for Flume?
ie initial time cost for setting up the infrastructure and periodic time
cost to add new use-cases.

Cheers,
Attila



On Thu, Oct 13, 2016 at 5:19 PM, Lior Zeno  wrote:

> Hi All,
>
> Monitoring Flume's performance over time is an important step in every
> production-level application.  Benchmarking Flume on a nightly basis has
> the following advantages:
>
> * Better understanding of Flume's bottlenecks.
> * Allow users to compare the performance of different solutions, such as
> Logstash and Fluentd.
> * Better understanding of the influence of recent commits on performance.
>
> Logstash already conducts various performance tests, more details in this
> link:
> http://logstash-benchmarks.elastic.co/
>
> I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
> course, in the ideal case where the input and/or output do not bottleneck
> the system), e.g. using the SeqGen source.
>
> Thoughts?
>
> Thanks
>

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Mike Percy]

+1 (binding)

There are some flaky tests which are listed below but I don't think they
are release blockers.

I performed the following checks:

Binary convenience artifact:
* Signature and checksums match
* LICENSE, NOTICE, and README.md files in the binary convenience artifact
look accurate and complete relative to the jars in lib/
* Ran a very quick test with the binary artifact and it
worked: ./bin/flume-ng agent -c conf -f
conf/flume-conf.properties.template -n
agent -Dflume.root.logger=DEBUG,console
* Checked that the documentation in docs/ renders: Flume User Guide and
Flume Dev Guide are OK. Also spot-checked that the new Kafka security
documentation was included in the User Guide

Source artifact:
* Signature and checksums match
* Built Flume from the source artifact using Oracle 1.7.0_80 on Ubuntu
Linux 16.04, sanity tested the resulting binary using the above method and
it worked
* RAT checks passed
* Built a new source artifact out of the official source artifact and
compiled it
* I ran the unit tests. Most passed but the below two failed. These are
flaky tests (we have a bunch of them in Flume) so I think it's fine not to
block the release on them.
  * TestExecSource.testMonitoredCounterGroup - looks like a racy test
  * TestSpillableMemoryChannel - didn't investigate

RC2 looks good to me.

Thanks for running this release, Donat!

Mike

On Wed, Oct 12, 2016 at 9:29 PM, Balazs Donat Bessenyei  wrote:

> Hi All,
>
> This is the tenth release for Apache Flume as a top-level project,
> version 1.7.0. We are voting on release candidate RC2.
>
> It fixes the following issues:
>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>
> *** Please cast your vote within the next 72 hours ***
>
> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
> for the source and binary artifacts can be found here:
>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
>
> Maven staging repo:
>   https://repository.apache.org/content/repositories/orgapacheflume-1020/
>
> The tag to be voted on:
>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868
>
> Flume's KEYS file containing PGP keys we use to sign the release:
>   https://www.apache.org/dist/flume/KEYS
>
>
> Thank you,
>
> Donat
>

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Attila Simon]

Hi Folks,

+1 on this Release Candidate

* hashes and signatures match
* build and run environment was Mac OSX El Capitan 10.11.5, Java
HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
* in src tarball
  * Sources of flume-checkstyle module is not included. Rest of the files
are there compared to apache git repo. As discussed it is not a release
blocker (checkstyle was removed intentionally from src tarball).
  * "mvn clean install -DskipTest" builds
  * "mvn test" passed
* in bin tarball
  * verified content of lib dir against the LICENCE file whether "each jar
shipped is included in the LICENCE"  and  "each jar mentioned in the
LICENCE is shipped".
  * sanity checked DEVNOTES, README.md, NOTICE, RELEASE-NOTES,
doap_Flume.rdf and the content of bin, conf, tools dirs
  * in docs dir I verified main page, user guide, dev guide, javadocs
  * executed the "bin/flume-ng  agent -n agent -c conf -f
conf/flume-conf.properties.template" from user guide which is passed (logs
were generated in the logs dir)

Cheers,
Attila



On Thu, Oct 13, 2016 at 3:10 PM, Denes Arvay  wrote:

> Hi,
>
> +1 for the RC2
> - checksums and signatures match
> - source package successfully built (but skipped the tests), rat and
> checkstyle pass
> - I was able to start up flume extracted from the binary package with the
> sample configuration
>
> Denes
>
> On Wed, Oct 12, 2016 at 9:29 PM Balazs Donat Bessenyei <
> bes...@cloudera.com>
> wrote:
>
> Hi All,
>
>
>
> This is the tenth release for Apache Flume as a top-level project,
>
> version 1.7.0. We are voting on release candidate RC2.
>
>
>
> It fixes the following issues:
>
>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>
>
>
> *** Please cast your vote within the next 72 hours ***
>
>
>
> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
>
> for the source and binary artifacts can be found here:
>
>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
>
>
>
> Maven staging repo:
>
>   https://repository.apache.org/content/repositories/orgapacheflume-1020/
>
>
>
> The tag to be voted on:
>
>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868
>
>
>
> Flume's KEYS file containing PGP keys we use to sign the release:
>
>   https://www.apache.org/dist/flume/KEYS
>
>
>
>
>
> Thank you,
>
>
>
> Donat
>

Flume bechmarks

[Lior Zeno]

Hi All,

Monitoring Flume's performance over time is an important step in every
production-level application.  Benchmarking Flume on a nightly basis has
the following advantages:

* Better understanding of Flume's bottlenecks.
* Allow users to compare the performance of different solutions, such as
Logstash and Fluentd.
* Better understanding of the influence of recent commits on performance.

Logstash already conducts various performance tests, more details in this
link:
http://logstash-benchmarks.elastic.co/

I propose adding a few micro-benchmarks showing Flume's TPS vs date (of
course, in the ideal case where the input and/or output do not bottleneck
the system), e.g. using the SeqGen source.

Thoughts?

Thanks

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Lior Zeno]

+1, tested simple scenarios on Windows 10 and Ubuntu 16.04.

On Thu, Oct 13, 2016 at 5:53 PM, Balazs Donat Bessenyei  wrote:

> Hi All,
>
> +1 for the RC2
>
> * md5s, sha1s  match
> * signatures are correct
> * mvn clean install -DskipTests runs successfully in the src ; RAT passes
> * binary works
> * LICENSE, NOTICE and README.md files are in the tar.gz-s
>
>
> Donat
>
> On Thu, Oct 13, 2016 at 3:10 PM, Denes Arvay  wrote:
> > Hi,
> >
> > +1 for the RC2
> > - checksums and signatures match
> > - source package successfully built (but skipped the tests), rat and
> > checkstyle pass
> > - I was able to start up flume extracted from the binary package with the
> > sample configuration
> >
> > Denes
> >
> > On Wed, Oct 12, 2016 at 9:29 PM Balazs Donat Bessenyei <
> bes...@cloudera.com>
> > wrote:
> >
> > Hi All,
> >
> >
> >
> > This is the tenth release for Apache Flume as a top-level project,
> >
> > version 1.7.0. We are voting on release candidate RC2.
> >
> >
> >
> > It fixes the following issues:
> >
> >   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
> >
> >
> >
> > *** Please cast your vote within the next 72 hours ***
> >
> >
> >
> > The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
> >
> > for the source and binary artifacts can be found here:
> >
> >   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
> >
> >
> >
> > Maven staging repo:
> >
> >   https://repository.apache.org/content/repositories/
> orgapacheflume-1020/
> >
> >
> >
> > The tag to be voted on:
> >
> >   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868
> >
> >
> >
> > Flume's KEYS file containing PGP keys we use to sign the release:
> >
> >   https://www.apache.org/dist/flume/KEYS
> >
> >
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Donat
>

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Balazs Donat Bessenyei]

Hi All,

+1 for the RC2

* md5s, sha1s  match
* signatures are correct
* mvn clean install -DskipTests runs successfully in the src ; RAT passes
* binary works
* LICENSE, NOTICE and README.md files are in the tar.gz-s


Donat

On Thu, Oct 13, 2016 at 3:10 PM, Denes Arvay  wrote:
> Hi,
>
> +1 for the RC2
> - checksums and signatures match
> - source package successfully built (but skipped the tests), rat and
> checkstyle pass
> - I was able to start up flume extracted from the binary package with the
> sample configuration
>
> Denes
>
> On Wed, Oct 12, 2016 at 9:29 PM Balazs Donat Bessenyei 
> wrote:
>
> Hi All,
>
>
>
> This is the tenth release for Apache Flume as a top-level project,
>
> version 1.7.0. We are voting on release candidate RC2.
>
>
>
> It fixes the following issues:
>
>   https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG
>
>
>
> *** Please cast your vote within the next 72 hours ***
>
>
>
> The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)
>
> for the source and binary artifacts can be found here:
>
>   http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/
>
>
>
> Maven staging repo:
>
>   https://repository.apache.org/content/repositories/orgapacheflume-1020/
>
>
>
> The tag to be voted on:
>
>   https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868
>
>
>
> Flume's KEYS file containing PGP keys we use to sign the release:
>
>   https://www.apache.org/dist/flume/KEYS
>
>
>
>
>
> Thank you,
>
>
>
> Donat

[jira] [Commented] (FLUME-3009) Update the website for 1.7 release

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLUME-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572056#comment-15572056
 ] 

Bessenyei Balázs Donát commented on FLUME-3009:
---

My bad.
I'm deleting this issue.

Thank you for notifying me, [~liorze]

> Update the website for 1.7 release
> --
>
> Key: FLUME-3009
> URL: https://issues.apache.org/jira/browse/FLUME-3009
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Bessenyei Balázs Donát
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Deleted] (FLUME-3009) Update the website for 1.7 release

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLUME-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát deleted FLUME-3009:
--


> Update the website for 1.7 release
> --
>
> Key: FLUME-3009
> URL: https://issues.apache.org/jira/browse/FLUME-3009
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Bessenyei Balázs Donát
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-3009) Update the website for 1.7 release

[Lior Zeno (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLUME-3009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15572050#comment-15572050
 ] 

Lior Zeno commented on FLUME-3009:
--

We already have a ticket for this - FLUME-2932. 

> Update the website for 1.7 release
> --
>
> Key: FLUME-3009
> URL: https://issues.apache.org/jira/browse/FLUME-3009
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Bessenyei Balázs Donát
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3009) Update the website for 1.7 release

[JIRA]

Bessenyei Balázs Donát created FLUME-3009:
-

 Summary: Update the website for 1.7 release
 Key: FLUME-3009
 URL: https://issues.apache.org/jira/browse/FLUME-3009
 Project: Flume
  Issue Type: Sub-task
Reporter: Bessenyei Balázs Donát






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-3007) Deploy Maven artifacts for 1.7 release

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLUME-3007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát updated FLUME-3007:
--
Assignee: (was: Bessenyei Balázs Donát)
 Summary: Deploy Maven artifacts for 1.7 release  (was: Deploy Maven 
artifacts)

> Deploy Maven artifacts for 1.7 release
> --
>
> Key: FLUME-3007
> URL: https://issues.apache.org/jira/browse/FLUME-3007
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Bessenyei Balázs Donát
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3008) Announce the release for 1.7

[JIRA]

Bessenyei Balázs Donát created FLUME-3008:
-

 Summary: Announce the release for 1.7
 Key: FLUME-3008
 URL: https://issues.apache.org/jira/browse/FLUME-3008
 Project: Flume
  Issue Type: Sub-task
Reporter: Bessenyei Balázs Donát






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLUME-3006) Upload the artifacts for 1.7 release

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLUME-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát updated FLUME-3006:
--
Assignee: (was: Bessenyei Balázs Donát)
 Summary: Upload the artifacts for 1.7 release  (was: Upload the artifacts)

> Upload the artifacts for 1.7 release
> 
>
> Key: FLUME-3006
> URL: https://issues.apache.org/jira/browse/FLUME-3006
> Project: Flume
>  Issue Type: Sub-task
>Reporter: Bessenyei Balázs Donát
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3006) Upload the artifacts

[JIRA]

Bessenyei Balázs Donát created FLUME-3006:
-

 Summary: Upload the artifacts
 Key: FLUME-3006
 URL: https://issues.apache.org/jira/browse/FLUME-3006
 Project: Flume
  Issue Type: Sub-task
Reporter: Bessenyei Balázs Donát
Assignee: Bessenyei Balázs Donát






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLUME-3007) Deploy Maven artifacts

[JIRA]

Bessenyei Balázs Donát created FLUME-3007:
-

 Summary: Deploy Maven artifacts
 Key: FLUME-3007
 URL: https://issues.apache.org/jira/browse/FLUME-3007
 Project: Flume
  Issue Type: Sub-task
Reporter: Bessenyei Balázs Donát
Assignee: Bessenyei Balázs Donát






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLUME-2929) Update License file for 1.7 release

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLUME-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571864#comment-15571864
 ] 

Bessenyei Balázs Donát commented on FLUME-2929:
---

[~sati]: thank you for the PR!

> Update License file for 1.7 release
> ---
>
> Key: FLUME-2929
> URL: https://issues.apache.org/jira/browse/FLUME-2929
> Project: Flume
>  Issue Type: Sub-task
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
>Assignee: Attila Simon
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLUME-2929) Update License file for 1.7 release

[JIRA]

 [ 
https://issues.apache.org/jira/browse/FLUME-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bessenyei Balázs Donát resolved FLUME-2929.
---
Resolution: Fixed

> Update License file for 1.7 release
> ---
>
> Key: FLUME-2929
> URL: https://issues.apache.org/jira/browse/FLUME-2929
> Project: Flume
>  Issue Type: Sub-task
>Affects Versions: v1.7.0
>Reporter: Lior Zeno
>Assignee: Attila Simon
> Fix For: v1.7.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Flume version 1.7.0 RC2

[Denes Arvay]

Hi,

+1 for the RC2
- checksums and signatures match
- source package successfully built (but skipped the tests), rat and
checkstyle pass
- I was able to start up flume extracted from the binary package with the
sample configuration

Denes

On Wed, Oct 12, 2016 at 9:29 PM Balazs Donat Bessenyei 
wrote:

Hi All,



This is the tenth release for Apache Flume as a top-level project,

version 1.7.0. We are voting on release candidate RC2.



It fixes the following issues:

  https://raw.githubusercontent.com/apache/flume/flume-1.7/CHANGELOG



*** Please cast your vote within the next 72 hours ***



The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1)

for the source and binary artifacts can be found here:

  http://people.apache.org/~bessbd/apache-flume-1.7.0-rc2/



Maven staging repo:

  https://repository.apache.org/content/repositories/orgapacheflume-1020/



The tag to be voted on:

  https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=511d868



Flume's KEYS file containing PGP keys we use to sign the release:

  https://www.apache.org/dist/flume/KEYS





Thank you,



Donat

[jira] [Commented] (FLUME-2997) Fix flaky junit test in SpillableMemoryChannel

[JIRA]

[ 
https://issues.apache.org/jira/browse/FLUME-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15571828#comment-15571828
 ] 

Bessenyei Balázs Donát commented on FLUME-2997:
---

[~sati]: thank you for the patch!

Can you please explain why you think it's better to remove the 
System.out.println-s we had in the test?

> Fix flaky junit test in SpillableMemoryChannel
> --
>
> Key: FLUME-2997
> URL: https://issues.apache.org/jira/browse/FLUME-2997
> Project: Flume
>  Issue Type: Test
>Affects Versions: v1.7.0
>Reporter: Attila Simon
>Assignee: Attila Simon
> Fix For: v1.8.0
>
> Attachments: FLUME-2997-1.patch, FLUME-2997.patch
>
>
> testParallelSingleSourceAndSink sometimes trigger an edge case scenario if 
> sinks are slower than sources. In such situations the channel can get full 
> thus uncaught ChannelFullException breaks the test. Since 
> testCapacityWithOverflow was designed to cover such edge-case scenario 
> already we can safely fix the test by increasing the channel capacity to make 
> sure it won't get full.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flume pull request #71: Flume 1.6

[cjn082030]

GitHub user cjn082030 opened a pull request:

https://github.com/apache/flume/pull/71

Flume 1.6

Use Flume-KafkaSink send messages to Kafka_0.10,Flume-server cpu up to 
99.*%.May be the method process() doesn't get the correct return value?Line 
127,if processedEvents <=0,the return status might should be Status.BACKOFF?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/flume flume-1.7

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flume/pull/71.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #71


commit 1f21df7e11f5fe8a3dcd0bd34116289c02943a64
Author: Hari Shreedharan 
Date:   2014-03-12T19:35:38Z

FLUME-2151. Windows: Update TestExecSource to use native commands on Windows

(Roshan Naik via Hari Shreedharan)

commit f4c6c7025e892ecf0d6fac1b72b5806fe7c9cd8d
Author: Hari Shreedharan 
Date:   2014-03-13T20:05:29Z

FLUME-2263. Bump Hadoop 2 version to 2.3.0

(Gopinathan A via Hari Shreedharan)

commit 1c8b8f5e9185d75c6b1cb06a617dfc08447935c9
Author: Hari Shreedharan 
Date:   2014-03-13T21:51:03Z

FLUME-2345. Update to Kite 0.12.0 dependency

(Ryan Blue via Hari Shreedharan)

commit e12f0a7a46752eff2ff3aaddfa0b93fb606074d7
Author: Hari Shreedharan 
Date:   2014-03-20T23:16:14Z

FLUME-2225. Elasticsearch Sink for ES HTTP API

(Pawel Rog via Edward Sargisson, Hari Shreedharan)

commit 09f0a513602c3b0bb2c9e19903ae6d5e052ab460
Author: Jarek Jarcec Cecho 
Date:   2014-03-24T19:55:35Z

FLUME-2347: Add FLUME_JAVA_OPTS which allows users to inject java 
properties from cmd line

(Brock Noland via Jarek Jarcec Cecho)

commit 61b9bcbb69ae3d19f72276b3aaa78ff3679cecfc
Author: Hari Shreedharan 
Date:   2014-03-27T19:30:29Z

FLUME-2309. Spooling directory should not always consume the oldest file 
first.

(Muhammad Ehsan ul Haque via Hari Shreedharan)

commit 62b383a00c3f678b0f504dc71bf36091ddd4067a
Author: Hari Shreedharan 
Date:   2014-03-28T22:55:25Z

FLUME-2350. Consume Order tests need to space out file creation.

(Muhammad Ehsan ul Haque via Hari Shreedharan)

commit a94594dd2c5cb980bc6f82b1fa606a922986569e
Author: Mike Percy 
Date:   2014-04-29T00:14:40Z

FLUME-2357. HDFS sink should retry closing files that previously had close 
errors

(Hari Shreedharan via Mike Percy)

commit 6115e7d6d611d2b82dc2583b95a13d4c0886a93f
Author: Brock Noland 
Date:   2014-05-02T14:32:33Z

FLUME-2181 - Optionally disable File Channel fsyncs (Hari via Brock)

commit 76b359b3fef1fade662388c8356e4a82c227c258
Author: Jarek Jarcec Cecho 
Date:   2014-05-04T04:43:16Z

FLUME-2381: Upgrade Hadoop version in Hadoop 2 profile to 2.4.0

(Hari Shreedharan via Jarek Jarcec Cecho)

commit 31d45f1b377826b8755d1479bda9e2d702206bf9
Author: Hari Shreedharan 
Date:   2014-05-04T05:23:56Z

FLUME-2368. Update CHANGELOG for 1.5 release

commit 45abf412c31f85e1b35135df529a36d94aac694c
Author: Hari Shreedharan 
Date:   2014-05-06T23:28:46Z

FLUME-2366. Update LICENSE

commit 63bade24bbabbe9eeb887e3294d4d3727cbac25d
Author: Hari Shreedharan 
Date:   2014-05-06T23:38:20Z

FLUME-2366. Preparing for Flume 1.5.0 release

commit 73fd7bddeff11ef7af91055db1148416dca9d0b1
Author: Hari Shreedharan 
Date:   2014-05-07T00:19:22Z

Adding default version for thrift

commit fd448caa8144ad3ea54a836d4ff6e944f36cfac6
Author: Hari Shreedharan 
Date:   2014-05-07T00:23:12Z

FLUME-2368. Update CHANGELOG.

commit f99113042dad2a67d24068e30ed37e4b18e62a72
Author: Hari Shreedharan 
Date:   2014-05-07T00:29:27Z

FLUME-2366. Update RELEASE NOTES for Flume 1.5

commit 33cdcf0d4e85e68e6df9e1ca4be729889d480246
Author: Hari Shreedharan 
Date:   2014-05-15T00:42:09Z

FLUME-2245. Pre-close flush failure can cause HDFS Sinks to not process 
events.

(Juhani Connolly, Brock Noland via Hari Shreedharan)

commit 09472ba12278a0d3696b9d2e26d6d1b0d361c830
Author: Juhani Connolly 
Date:   2014-06-03T02:33:54Z

FLUME-2273 - Add handling for header substitution in ElasticSearchSink

Satoshi Iijima via Juhani Connolly

commit 0cba73698dbba6b78d0a2cd7b469f4377723470a
Author: Jarek Jarcec Cecho 
Date:   2014-06-05T16:28:02Z

FLUME-2397: HBase-98 compatibility

(Hari Shreedharan via Jarek Jarcec Cecho)

commit 059795fb9aa890430555f3a4c02a73ec346e7092
Author: 

Re: how to make KafkaSource consume the existing messages

[Attila Simon]

Hi,

One more thing. If you switch to the new group.id and would like to
maintain the read from beginning behaviour every time flume restart
then you might try setting enable.auto.commit to false.
Again Kafka normally won't store the events indefinitely.

Cheers,
Attila


On Thu, Oct 13, 2016 at 11:45 AM, Attila Simon  wrote:
> for the records cc dev@
>
> On Thu, Oct 13, 2016 at 11:43 AM, Attila Simon  wrote:
>> Hi,
>>
>> auto.offset.reset aim to handle failure scenarios when Flume lost the
>> track of offsets. When Flume is able to successfully consume the
>> messages it also commits the last processed offset. When failure
>> happens and  was set resetting offset would use the last
>> committed value.
>> I don't think that always starting from "zero" offset would be
>> valuable (would result a lot of duplicates). So I assume you would
>> like to have a recovery scenario. What you can do is setting the
>> consumer group.id to something new so if kafka still has the messages
>> - you can check that with command line kafka consumer setting the
>> --from-beginning argument as kafka by default purges them periodically
>> - then flume would reset the offset to the effective beginning since
>> offsets are stored per group.id.
>>
>> Quoted from Kafka docs
>> (http://kafka.apache.org/documentation#newconsumerconfigs):
>> auto.offset.reset - What to do when there is no initial offset in
>> Kafka or if the current offset does not exist any more on the server
>> (e.g. because that data has been deleted):
>>
>> earliest: automatically reset the offset to the earliest offset
>> latest: automatically reset the offset to the latest offset
>> none: throw exception to the consumer if no previous offset is found
>> for the consumer's group
>> anything else: throw exception to the consumer.
>>
>> Cheers,
>> Attila
>>
>>
>> On Thu, Oct 13, 2016 at 10:00 AM, Ping PW Wang  wrote:
>>> Hi,
>>> I used KafkaSource to consume the messages from Kafka. I found only new
>>> messages were received while the old existing message not. I tried to use a
>>> new consumer group and update the parameter "auto.offset.reset = latest" to
>>> "earliest", but this does not work.
>>>
>>> tier2.sources.source1.kafka.consumer.group.id = test-consumer-group-new
>>> tier2.sources.source1.kafka.consumer.auto.offset.reset = earliest
>>>
>>> Anyone knows how to make KafkaSource consume the existing messages?
>>> Thanks a lot for any advice!
>>>
>>> Best Regards,
>>>
>>> Wang Ping (王苹)
>>> InfoSphere BigInsights, CDL
>>> Email: wpw...@cn.ibm.com Phone: (8610)82453448 Mobile: (86)17090815725
>>> Address: Ring Bldg.No.28 Building,ZhongGuanCun Software Park,No.8 Dong Bei
>>> Wang West Road, Haidian District Beijing P.R.China 100193
>>> 地址：北京市海淀区东北旺西路8号，中关村软件园28号楼 邮编：100193
>>>

Re: how to make KafkaSource consume the existing messages

[Attila Simon]

for the records cc dev@

On Thu, Oct 13, 2016 at 11:43 AM, Attila Simon  wrote:
> Hi,
>
> auto.offset.reset aim to handle failure scenarios when Flume lost the
> track of offsets. When Flume is able to successfully consume the
> messages it also commits the last processed offset. When failure
> happens and  was set resetting offset would use the last
> committed value.
> I don't think that always starting from "zero" offset would be
> valuable (would result a lot of duplicates). So I assume you would
> like to have a recovery scenario. What you can do is setting the
> consumer group.id to something new so if kafka still has the messages
> - you can check that with command line kafka consumer setting the
> --from-beginning argument as kafka by default purges them periodically
> - then flume would reset the offset to the effective beginning since
> offsets are stored per group.id.
>
> Quoted from Kafka docs
> (http://kafka.apache.org/documentation#newconsumerconfigs):
> auto.offset.reset - What to do when there is no initial offset in
> Kafka or if the current offset does not exist any more on the server
> (e.g. because that data has been deleted):
>
> earliest: automatically reset the offset to the earliest offset
> latest: automatically reset the offset to the latest offset
> none: throw exception to the consumer if no previous offset is found
> for the consumer's group
> anything else: throw exception to the consumer.
>
> Cheers,
> Attila
>
>
> On Thu, Oct 13, 2016 at 10:00 AM, Ping PW Wang  wrote:
>> Hi,
>> I used KafkaSource to consume the messages from Kafka. I found only new
>> messages were received while the old existing message not. I tried to use a
>> new consumer group and update the parameter "auto.offset.reset = latest" to
>> "earliest", but this does not work.
>>
>> tier2.sources.source1.kafka.consumer.group.id = test-consumer-group-new
>> tier2.sources.source1.kafka.consumer.auto.offset.reset = earliest
>>
>> Anyone knows how to make KafkaSource consume the existing messages?
>> Thanks a lot for any advice!
>>
>> Best Regards,
>>
>> Wang Ping (王苹)
>> InfoSphere BigInsights, CDL
>> Email: wpw...@cn.ibm.com Phone: (8610)82453448 Mobile: (86)17090815725
>> Address: Ring Bldg.No.28 Building,ZhongGuanCun Software Park,No.8 Dong Bei
>> Wang West Road, Haidian District Beijing P.R.China 100193
>> 地址：北京市海淀区东北旺西路8号，中关村软件园28号楼 邮编：100193
>>