Re: [VOTE] Release Apache Flume Spring Boot 2.0.0-rc2
+1 Build works fine and all tests pass on M1 On Mon, Mar 20, 2023 at 12:48 PM Ralph Goers wrote: > This is a vote to release Flume Spring Boot 2.0.0. Flume Spring Boot has > moved from the full Flume release to its own repo. Note that the staging > web site has been updated to reflect the fix for Bug #1 below. Also note > that this repo supports the use of GitHub Issues for bug tracking. > > RC2 adds a missing sentence to the NOTICE file. The web site has been > updated to include a section for sub projects and Flume Spring Boot has > been added there. > > Please download, test, and cast your votes on the Flume developers list. > [] +1, release the artifacts > [] -1, don't release because... > > The vote will remain open for 72 hours. All votes are welcome and we > encourage everyone to test the release, but only Flume PMC votes are > “officially” counted. As always, at least 3 +1 votes and more positive than > negative votes are required. > > Changes in this release include: > > ** Bug > • [#1] - Require Applications to define a configuration class containing > the appropriate ComponentScan declaration in spring.factories. > > > Tag: > a) for a new copy do "git clone > https://github.com/apache/flume-spring-boot.git and then "git checkout > tags/release-2.0.0-rc2” or just "git clone -b release-2.0.0-rc2 > https://github.com/apache/flume-spring-boot.git"; > b) for an existing working copy to “git pull” and then “git checkout > tags/release-2.0.00-rc2” > > Web Site: https://flume.staged.apache.org/. Specifically for Flume > Spring Boot - https://flume.staged.apache.org/flume-spring-boot/index.html > . > > Maven Artifacts: > https://repository.apache.org/content/repositories/orgapacheflume-1045. > > Distribution archives: > https://dist.apache.org/repos/dist/dev/flume/flume-spring-boot > > You may download all the Maven artifacts by executing: > wget -e robots=off --cut-dirs=7 -nH -r -p -np --no-check-certificate > https://repository.apache.org/content/repositories/orgapacheflume-1045/org/apache/flume/ > > Ralph -- thanks ashish
Re: [VOTE] Release Apache Flume 1.10.1-rc2
+1 (non-binding) Build looks good and all tests pass (validated on M1 Mac, JDK8) On Sat, Aug 13, 2022 at 2:40 PM Ralph Goers wrote: > This is a vote to release Flume 1.10.1, the next version of the Apache > Flume project. > > Please download, test, and cast your votes on the Flume developers list. > [] +1, release the artifacts > [] -1, don't release because... > > The vote will remain open for 72 hours. All votes are welcome and we > encourage everyone to test the release, but only Flume PMC votes are > “officially” counted. As always, at least 3 +1 votes and more positive than > negative votes are required. > > Changes in this release can be found at > https://flume.staged.apache.org/releases/1.10.1.html. > > Tag: > a) for a new copy do "git clone https://github.com/apache/flume.git and > then "git checkout tags/release-1.10.1-rc2” or just "git clone -b > release-1.10.1-r2 https://github.com/apache/flume.git"; > b) for an existing working copy to “git pull” and then “git checkout > tags/release.10.1-rc2” > > Web Site: https://flume.staged.apache.org/. > > Maven Artifacts: > https://repository.apache.org/content/repositories/orgapacheflume-1037. > > Distribution archives: https://dist.apache.org/repos/dist/dev/flume/ > > You may download all the Maven artifacts by executing: > wget -e robots=off --cut-dirs=7 -nH -r -p -np --no-check-certificate > https://repository.apache.org/content/repositories/orgapacheflume-1037/org/apache/flume/ > > Ralph -- thanks ashish
Re: [DISCUSS] Checkstyle maven plugin
+1 On Fri, Jun 24, 2016 at 2:24 PM, Mike Percy wrote: > Hey devs, > Code nitpicks have come up a bit lately (in code I'm the reviewer of). > Other Apache projects such as HBase and Kafka use checkstyle to do a > pre-commit check at build time. Rather than spend time going back and forth > on code style, how about we adopt the checkstyle plugin for Flume? > > I'd propose adopting the Google Java style. It's what the vast majority of > the Flume code uses today, and there is a config file shipped with > checkstyle for it. Here's a link to it: > https://google.github.io/styleguide/javaguide.html > > My goal is just to maintain a consistent style throughout the code base and > avoid the review noise. Please let me know whether or not this sounds > helpful. > > Thanks, > Mike -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Flume vs Apache Nifi
Just in case someone is still interested, here is a webinar from Streamsets that covers it http://go.streamsets.com/opensourceingest.html On Wed, Feb 11, 2015 at 8:32 PM, Ashish wrote: > Has anyone in the list compared Flume to incubating Apache Nifi? > > I have seen Nifi mentioned couple of time on the list. > > Just curious, is it going to compete with Flume? > > BTW, I like the UI feature > http://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#User_Interface > > Flume could adopt it in a diff way. We can use this tool to generate > deployment configurations and support templates. If someone is using > Zk based config management, it can be used easily to modify. > > If we take it further, we can integrate monitoring to show current > stats on the UI > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41702/#review125723 --- Ship it! Ship It! - Ashish Singh On March 28, 2016, 7:02 p.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 28, 2016, 7:02 p.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41702/#review125703 --- flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java (line 46) <https://reviews.apache.org/r/41702/#comment188571> No, we should never do this. Please use Runtime exception instead. - Ashish Singh On March 28, 2016, 5:42 p.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 28, 2016, 5:42 p.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41702/#review125702 --- flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java (line 45) <https://reviews.apache.org/r/41702/#comment188570> Sorry, I am probably not following this, but how do you stop test execution? Do you mean to add a Runtime exception here? - Ashish Singh On March 28, 2016, 5:42 p.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 28, 2016, 5:42 p.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
> On March 28, 2016, 4:26 p.m., Ashish Singh wrote: > > Ship It! Just one minor comment and we are good to go. Thanks for the awesome work Grigoriy! - Ashish --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41702/#review125668 --- On March 28, 2016, 9:38 a.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 28, 2016, 9:38 a.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41702/#review125668 --- Ship it! Ship It! - Ashish Singh On March 28, 2016, 9:38 a.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 28, 2016, 9:38 a.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41702/#review125667 --- flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java (line 45) <https://reviews.apache.org/r/41702/#comment188535> If we can not get host, I do not think we can continue with the tests. We should throw an exception here. - Ashish Singh On March 28, 2016, 9:38 a.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 28, 2016, 9:38 a.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41702/#review125458 --- flume-ng-doc/sphinx/FlumeUserGuide.rst (line 1207) <https://reviews.apache.org/r/41702/#comment188244> Typo in description? flume-ng-doc/sphinx/FlumeUserGuide.rst (lines 1257 - 1260) <https://reviews.apache.org/r/41702/#comment188245> Won't this require consumer group id? flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java (line 40) <https://reviews.apache.org/r/41702/#comment188249> localhost's ip can be obtained from InetAddress.getLocalHost().getHostAddress(). flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java (line 447) <https://reviews.apache.org/r/41702/#comment188251> Is it possible to avoid this type casting? flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java (line 474) <https://reviews.apache.org/r/41702/#comment188250> Maybe add a test to verify that regex is given priority over topics list for subscribing. - Ashish Singh On March 24, 2016, 6:26 p.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 24, 2016, 6:26 p.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Review Request 41702: Kafka Source support for new Consumer API FLUME-2821
flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java (line 114) <https://reviews.apache.org/r/41702/#comment187893> Same here. flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java (line 53) <https://reviews.apache.org/r/41702/#comment187894> Same comment for address. flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java (line 89) <https://reviews.apache.org/r/41702/#comment187895> Can we break this into multiple smaller tests? Would it make sense to do so? - Ashish Singh On March 23, 2016, 4:59 p.m., Grigoriy Rozhkov wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/41702/ > --- > > (Updated March 23, 2016, 4:59 p.m.) > > > Review request for Flume, Grant Henke, Jarek Cecho, and Jeff Holoman. > > > Repository: flume-git > > > Description > --- > > Kafka Source support for kafka 0.9 > > > Diffs > - > > flume-ng-doc/sphinx/FlumeUserGuide.rst > 423e0cf8a3fffbdab6363f4f71df7fd8c3c133a3 > flume-ng-sources/flume-kafka-source/pom.xml > 0f93476c61e0281d45a15426493c4c3579503cee > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java > fd1dd3c17bda875daaecef02cd7c537bb14242b8 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java > 911012cefcd656bac3308c3e02990a6ff42a0de5 > > flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceUtil.java > 4a4034bd826f67e790e16b67e5b52f469b182627 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java > 26c5c9d0aa29fbbc1d9eced70b26aa3f81855c26 > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedZookeeper.java > 1b8a27106e84b3ce1d8a100d31cf33d847c68f1b > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java > 8ec14cccf555ab38449b1bcb1e41a6ecbd19fe7c > > flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSourceUtil.java > 0cbb4b69eca3b052a4b74a52a71ea3e45ce5f814 > pom.xml 15c086b5090305a1a70185626452a05cb7c1941d > > Diff: https://reviews.apache.org/r/41702/diff/ > > > Testing > --- > > New features: > - multiple topic subscription > Style for configuring new Kafka Source is made following styles in these > patches. > https://reviews.apache.org/r/41626/ > https://reviews.apache.org/r/41629/ > > generate command: > git diff --no-prefix --full-index > FLUME-2821.patch > > apply command: > patch -p0 < FLUME-2821.patch > > > Thanks, > > Grigoriy Rozhkov > >
Re: Github integration
+1 for it. Let's try and see if it makes it easy to contribute. On Sat, Mar 19, 2016 at 12:14 PM, Hari Shreedharan wrote: > Hi, > > I have worked for a while on Spark recently, and like using github for > scm. While not the best tool for code reviews, it certainly is better > than using patches on jiras. We already get a lot of review requests > as Pull Requests anyway. I'd like some community feedback on this. > > I think we'd need a vote before we can get it done. > > Thanks! > Hari -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Flume HTTP source and CSRF vulnerability
IMHO, No. XSRF can happen with session based things where two parties are talking. In Flume's case we never talk, we just listen and pass on the data. Other than Http Ok, we send anything back. We don't provide "Identifier" to the client which can be used by anyone else. You must have encountered something interesting, would be good to know (on ML or off ML both works fine) On Fri, Jan 8, 2016 at 1:08 PM, Roshan Naik wrote: > My understanding is that Flume HTTP source does not have any protection > against Cross-Site Request Forgery (CSRF) attacks. Wanted to double check > with others if that is correct ? > > -roshan -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Resolved] (FLUME-2836) Flume and Cassandra sink
[
https://issues.apache.org/jira/browse/FLUME-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Paliwal resolved FLUME-2836.
---
Resolution: Won't Fix
ML question
> Flume and Cassandra sink
>
>
> Key: FLUME-2836
> URL: https://issues.apache.org/jira/browse/FLUME-2836
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Ubuntu.14.04,hadoop.2.2.0,flume.1.6.0,cassandra.2.1.11,
>Reporter: Veerendra Nath Jasthi
> Fix For: notrack
>
>
> This is My flume.conf file:
> # custom Cassandra sink
> cassandra_agent.sinks.sink3.type = com.veeru.flume.CustomCassandraSink
> cassandra_agent.sinks.sink3.channel = channel1
> cassandra_agent.sources = source1 source2 source3
> cassandra_agent.channels = channel1 channel2 channel3
> cassandra_agent.sinks = sink1 sink2 sink3
> cassandra_agent.sources.source1.type = exec
> cassandra_agent.sources.source1.command = tail -F /home/hduser/input1.log
> cassandra_agent.sources.source1.channels = channel1
> cassandra_agent.channels.channel1.type = memory
> cassandra_agent.sinks.sink1.type = avro
> cassandra_agent.sinks.sink1.hostname = localhost
> cassandra_agent.sinks.sink1.port = 9160
> cassandra_agent.sinks.sink1.channel = channel1
> cassandra_agent.sources.source2.type = exec
> cassandra_agent.sources.source2.command = tail -F /home/hduser/input2.log
> cassandra_agent.sources.source2.channels = channel2
> cassandra_agent.channels.channel2.type = memory
> cassandra_agent.sinks.sink2.type = avro
> cassandra_agent.sinks.sink2.hostname = localhost
> cassandra_agent.sinks.sink2.port = 9160
> cassandra_agent.sinks.sink2.channel = channel2
> cassandra_agent.sources.source3.type = avro
> cassandra_agent.sources.source3.bind = localhost
> cassandra_agent.sources.source3.port = 9160
> cassandra_agent.sources.source3.channels = channel3
> cassandra_agent.channels.channel3.type = memory
> cassandra_agent.channels.channel3.capacity = 1000
> cassandra_agent.channels.channel3.transactionCapacity = 100
> # custom Cassandra sink
> cassandra_agent.sinks.sink3.type = com.veeru.flume.CustomCassandraSink
> cassandra_agent.sinks.sink3.channel = channel3
>
> Whenever I am running the agent I could see this error:
> ERROR lifecycle.LifecycleSupervisor: Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@28c1b980 counterGroup:{
> name:null counters:{} } } - Exception follows.
> java.lang.IllegalArgumentException:
> Could someone help me out on this ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2836) Flume and Cassandra sink
[
https://issues.apache.org/jira/browse/FLUME-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14989607#comment-14989607
]
Ashish Paliwal commented on FLUME-2836:
---
Please ask questions on User Mailing lists
> Flume and Cassandra sink
>
>
> Key: FLUME-2836
> URL: https://issues.apache.org/jira/browse/FLUME-2836
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Ubuntu.14.04,hadoop.2.2.0,flume.1.6.0,cassandra.2.1.11,
>Reporter: Veerendra Nath Jasthi
> Fix For: notrack
>
>
> This is My flume.conf file:
> # custom Cassandra sink
> cassandra_agent.sinks.sink3.type = com.veeru.flume.CustomCassandraSink
> cassandra_agent.sinks.sink3.channel = channel1
> cassandra_agent.sources = source1 source2 source3
> cassandra_agent.channels = channel1 channel2 channel3
> cassandra_agent.sinks = sink1 sink2 sink3
> cassandra_agent.sources.source1.type = exec
> cassandra_agent.sources.source1.command = tail -F /home/hduser/input1.log
> cassandra_agent.sources.source1.channels = channel1
> cassandra_agent.channels.channel1.type = memory
> cassandra_agent.sinks.sink1.type = avro
> cassandra_agent.sinks.sink1.hostname = localhost
> cassandra_agent.sinks.sink1.port = 9160
> cassandra_agent.sinks.sink1.channel = channel1
> cassandra_agent.sources.source2.type = exec
> cassandra_agent.sources.source2.command = tail -F /home/hduser/input2.log
> cassandra_agent.sources.source2.channels = channel2
> cassandra_agent.channels.channel2.type = memory
> cassandra_agent.sinks.sink2.type = avro
> cassandra_agent.sinks.sink2.hostname = localhost
> cassandra_agent.sinks.sink2.port = 9160
> cassandra_agent.sinks.sink2.channel = channel2
> cassandra_agent.sources.source3.type = avro
> cassandra_agent.sources.source3.bind = localhost
> cassandra_agent.sources.source3.port = 9160
> cassandra_agent.sources.source3.channels = channel3
> cassandra_agent.channels.channel3.type = memory
> cassandra_agent.channels.channel3.capacity = 1000
> cassandra_agent.channels.channel3.transactionCapacity = 100
> # custom Cassandra sink
> cassandra_agent.sinks.sink3.type = com.veeru.flume.CustomCassandraSink
> cassandra_agent.sinks.sink3.channel = channel3
>
> Whenever I am running the agent I could see this error:
> ERROR lifecycle.LifecycleSupervisor: Unable to start SinkRunner: {
> policy:org.apache.flume.sink.DefaultSinkProcessor@28c1b980 counterGroup:{
> name:null counters:{} } } - Exception follows.
> java.lang.IllegalArgumentException:
> Could someone help me out on this ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [ANNOUNCE] Change of Apache Flume PMC Chair
Congrats Hari ! Arvind - Thanks for watching over and taking care of the community. Hope you would continue to do so in the future as well :) On Wed, Oct 21, 2015 at 5:50 PM, Arvind Prabhakar wrote: > Dear Flume Users and Developers, > > I have had the pleasure of serving as the PMC Chair of Apache Flume since > its graduation three years ago. I sincerely thank you and the Flume PMC for > this opportunity. However, I have decided to step down from this > responsibility due to personal reasons. > > I am very happy to announce that on the request of Flume PMC and with the > approval from the board of directors at The Apache Software Foundation, > Hari Shreedharan is hereby appointed as the new PMC Chair. I am confident > that Hari will do everything possible to help further grow the community > and adoption of Apache Flume. > > Please join me in congratulating Hari on his appointment and welcoming him > to this role. > > Regards, > Arvind Prabhakar -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Commented] (FLUME-2632) High CPU on KafkaSink
[ https://issues.apache.org/jira/browse/FLUME-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967982#comment-14967982 ] Ashish Paliwal commented on FLUME-2632: --- It's a minor fix. Since I authored it, I cannot commit the changes. If its working fine, lets get it out of way :) > High CPU on KafkaSink > - > > Key: FLUME-2632 > URL: https://issues.apache.org/jira/browse/FLUME-2632 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 >Reporter: Gwen Shapira >Assignee: Gwen Shapira > Labels: kafka > Fix For: v1.6.0 > > Attachments: FLUME-2632-0.patch > > > Reported here: https://github.com/harishreedharan/flume/issues/1 > "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu > utilization stay at 100% and never dropped down all the time even at the time > the channel is empty. > I looked into the source code and found that "process" function in KafkaSink > always return Status.READY even if no events available in channel. That > causes the SinkRunner keep running achieving event from channel and get > nothing. > Do we need to change to return Status.BACKOFF in "process" function in > KafkaSink when it notices that there is no events processed in current round? > So that the SinkRunner has a chance to take a rest when there is no event in > channel. If this proposal feasible, function "testEmptyChannel" in > TestKafkaSink also need to be changed. " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2632) High CPU on KafkaSink
[ https://issues.apache.org/jira/browse/FLUME-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14961473#comment-14961473 ] Ashish Paliwal commented on FLUME-2632: --- [~jrufus]/[~hshreedharan]/[~gwenshap] Can anyone of you please have a look and do the needful :) > High CPU on KafkaSink > - > > Key: FLUME-2632 > URL: https://issues.apache.org/jira/browse/FLUME-2632 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 >Reporter: Gwen Shapira >Assignee: Gwen Shapira > Labels: kafka > Fix For: v1.6.0 > > Attachments: FLUME-2632-0.patch > > > Reported here: https://github.com/harishreedharan/flume/issues/1 > "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu > utilization stay at 100% and never dropped down all the time even at the time > the channel is empty. > I looked into the source code and found that "process" function in KafkaSink > always return Status.READY even if no events available in channel. That > causes the SinkRunner keep running achieving event from channel and get > nothing. > Do we need to change to return Status.BACKOFF in "process" function in > KafkaSink when it notices that there is no events processed in current round? > So that the SinkRunner has a chance to take a rest when there is no event in > channel. If this proposal feasible, function "testEmptyChannel" in > TestKafkaSink also need to be changed. " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941465#comment-14941465 ] Ashish Paliwal commented on FLUME-2437: --- [~jrufus] The patch needs rebasing. Could you please look at it. [~otis] The Source does not cater to this use case of now. My take is to get it in trunk and refine it, else it would be left out in JIRA. > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Johny Rufus > Attachments: FLUME-2437-2.patch, FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940419#comment-14940419 ] Ashish Paliwal commented on FLUME-2437: --- Cool. [~hshreedharan] Do you have an concerns/suggestions before we proceed further? > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Johny Rufus > Attachments: FLUME-2437-2.patch, FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940288#comment-14940288 ] Ashish Paliwal commented on FLUME-2437: --- [~jrufus] Shall we push this in as an experimental source? > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Ashish Paliwal > Attachments: FLUME-2437-2.patch, FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2437: -- Assignee: Johny Rufus (was: Ashish Paliwal) > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Johny Rufus > Attachments: FLUME-2437-2.patch, FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Git diff does not pick new files
new files are a bit tricky, after git add use this >git diff HEAD Hari helped me with this on this JIRA https://issues.apache.org/jira/browse/FLUME-1501 On Mon, Sep 28, 2015 at 1:03 PM, Hari Shreedharan wrote: > You need to first do a git add for the new files > > > > > > Thanks, Hari > > On Mon, Sep 28, 2015 at 2:52 PM, IT CTO wrote: > >> Based on flume docs I tried using git diff to create the patch I want to >> submit. But I noticed the file only contains the updated files and no new >> files were added. >> Why? >> Also, if I use the --staged flag only new file are included. >> What am I missing? >> Eran >> -- >> Eran | "You don't need eyes to see, you need vision" (Faithless) -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to contribute + github
If you have the patch file uploaded, you can download and apply it to get your working version. Rebase is simple, git pull. If the patch is large I prefer to take a backup. But most part not worried much about it. Once I know how to solve the JIRA, its mostly easy to recreate the solution even from scratch. Have I ever lost my data, yes. But recovered a lot of it from IntelliJ local history. And sometimes re-wrote the complete solution. I prefer this coz I precisely know how much I can take and if lot of patches are pending, I know I have to stop and let the team catch-up. It's personal choice, after a few patches you will have your own :) On Mon, Sep 28, 2015 at 10:27 AM, IT CTO wrote: > Thanks for the reply. > So you build a local repo for every jira issue you work on... this make > sence but force you to rebase frequently especially if you update the doc > file which is one big file... > Also, in this scenario you don't have a backup on cloud for your source, > right? > > Again, thanks for sharing! > Eran > > בתאריך יום ב׳, 28 בספט׳ 2015, 20:10 מאת Ashish : > >> Flume doesn't accept PR's so fork or not is your choice. >> >> Simplest workflow could be >> keep an updated clone of repo >> Make the changes >> Create a patch and upload to JIRA >> >> I think your main concern is how to manage locally. I prefer to keep >> clone's specific to JIRA's which help me in tracking them easily. >> It's a little painful like this, but the simplicity makes my life >> easy. Since each one is different, helps me to keep track of it. In my >> case I usually work on lot of JIRA's in parallel. Some are simple and >> some takes few weeks to get in shape. >> >> You can choose whatever works for you. Hope it answers your question. >> >> On Mon, Sep 28, 2015 at 9:06 AM, IT CTO wrote: >> > Hi, >> > I read the >> > >> https://cwiki.apache.org/confluence/display/FLUME/Developers+Quick+Hack+Sheet >> > >> > If I write my code where my origin is the >> > origin https://github.com/apache/flume.git (fetch) >> > origin https://github.com/apache/flume.git (push) >> > How can I save my changes while they are being reviewed? >> > Shouldn't I fork the repo, checkout from my repo so I can save the >> changes? >> > >> > Can someone share his method of work? >> > Eran >> > -- >> > Eran | "You don't need eyes to see, you need vision" (Faithless) >> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> > -- > Eran | "You don't need eyes to see, you need vision" (Faithless) -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: How to contribute + github
Flume doesn't accept PR's so fork or not is your choice. Simplest workflow could be keep an updated clone of repo Make the changes Create a patch and upload to JIRA I think your main concern is how to manage locally. I prefer to keep clone's specific to JIRA's which help me in tracking them easily. It's a little painful like this, but the simplicity makes my life easy. Since each one is different, helps me to keep track of it. In my case I usually work on lot of JIRA's in parallel. Some are simple and some takes few weeks to get in shape. You can choose whatever works for you. Hope it answers your question. On Mon, Sep 28, 2015 at 9:06 AM, IT CTO wrote: > Hi, > I read the > https://cwiki.apache.org/confluence/display/FLUME/Developers+Quick+Hack+Sheet > > If I write my code where my origin is the > origin https://github.com/apache/flume.git (fetch) > origin https://github.com/apache/flume.git (push) > How can I save my changes while they are being reviewed? > Shouldn't I fork the repo, checkout from my repo so I can save the changes? > > Can someone share his method of work? > Eran > -- > Eran | "You don't need eyes to see, you need vision" (Faithless) -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Commented] (FLUME-2783) Update Website Team page with new Committer's
[ https://issues.apache.org/jira/browse/FLUME-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14790940#comment-14790940 ] Ashish Paliwal commented on FLUME-2783: --- +1 looks good. Pls go ahead and commit the changes > Update Website Team page with new Committer's > - > > Key: FLUME-2783 > URL: https://issues.apache.org/jira/browse/FLUME-2783 > Project: Flume > Issue Type: Documentation > Components: Docs >Reporter: Ashish Paliwal >Priority: Minor > Attachments: FLUME-2783.patch > > > Team page on website need to add 2 newly voted committers > Ashish Paliwal > Johny Rufus -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2783) Update Website Team page with new Committer's
Ashish Paliwal created FLUME-2783: - Summary: Update Website Team page with new Committer's Key: FLUME-2783 URL: https://issues.apache.org/jira/browse/FLUME-2783 Project: Flume Issue Type: Documentation Components: Docs Reporter: Ashish Paliwal Priority: Minor Team page on website need to add 2 newly voted committers Ashish Paliwal Johny Rufus -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2765) ThriftSource spaws too many threads
[
https://issues.apache.org/jira/browse/FLUME-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699731#comment-14699731
]
Ashish Paliwal commented on FLUME-2765:
---
Thanks ! I shall look into this whenever I get some free cycles.
Had a quick glance at the code, you can limit the number of Theads by using by
default it uses cached thread pool executor, which would create Threads as
needed.
> ThriftSource spaws too many threads
> ---
>
> Key: FLUME-2765
> URL: https://issues.apache.org/jira/browse/FLUME-2765
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6
>Reporter: Tobias Heintz
> Attachments: thread-dump-flume-1.6.txt
>
>
> We are in the process of migrating from the old Flume to version 1.6. We are
> using the ThriftSource with the new KafkaSink. Here's what our config looks
> like:
> {code}
> agent1.channels = ch1
> agent1.sources = thriftSrc
> agent1.sinks = kafka
> agent1.channels.ch1.type = memory
> agent1.channels.ch1.capacity = 1
> agent1.channels.ch1.transactionCapacity = 500
> # THRIFT
> agent1.sources.thriftSrc.type = thrift
> agent1.sources.thriftSrc.channels = ch1
> agent1.sources.thriftSrc.bind = 0.0.0.0
> agent1.sources.thriftSrc.port = 4042
> agent1.sources.thriftSrc.threads = 150 # if we don't set this option, the
> source keeps creating more and more threads until all heap memory is used up
> and then it crashes
> # KAFKA
> agent1.sinks.kafka.channel = ch1
> agent1.sinks.kafka.type = org.apache.flume.sink.kafka.KafkaSink
> agent1.sinks.kafka.batchSize = 50
> agent1.sinks.kafka.brokerList = broker.example.com:9092
> agent1.sinks.kafka.requiredAcks = 1
> agent1.sinks.kafka.topic = topic1
> {code}
> We have been noticing some bad behavior by the Thrift source/Thrift server
> using the JMX connection. If we don't restrict the number of threads, it
> spawns thousands of new threads, apparently one for every message it
> receives. These threads all have the name "Flume Thrift IPC Thread [number]"
> and according to the jvisualvm console they are always idle. At some point
> all of the JVM memory is used up through creating new threads and flume
> crashes with the following exception:
> {code}
> 12 Aug 2015 16:56:11,721 ERROR [Thread-1]
> (org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run:544) -
> run() exiting due to uncaught error
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
> at
> org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(TThreadedSelectorServer.java:310)
> at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:209)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:576)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:536)
> {code}
> When we set the option to restrict the number of threads, the server sticks
> to that number and runs smoothly, however it drops messages occasionally (may
> have a different cause).
> I am wondering whether this is a bug or in some way expected behavior? What
> are the best practices for using a ThriftSource? Are there further parameters
> to possibly tune (like channel.capacity)?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2757) Hive Sink Tests failing due to incorrect permission on hive scratch dir
[
https://issues.apache.org/jira/browse/FLUME-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697915#comment-14697915
]
Ashish Paliwal commented on FLUME-2757:
---
Is it possible to use this API TemporaryFolder(File parentFolder) instead of
TemporaryFolder(). We can try to pass a folder in user directory like "tmp"
instead of "/tmp". Can you give this a try?
> Hive Sink Tests failing due to incorrect permission on hive scratch dir
> ---
>
> Key: FLUME-2757
> URL: https://issues.apache.org/jira/browse/FLUME-2757
> Project: Flume
> Issue Type: Bug
> Components: Test
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2757.patch
>
>
> Error Message
> {code}
> java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be
> writable. Current permissions are: rwxr-xr-x
> {code}
> Stacktrace
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir:
> /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
> at org.apache.flume.sink.hive.TestHiveSink.(TestHiveSink.java:109)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on
> HDFS should be writable. Current permissions are: rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:529)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:478)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430)
> ... 29 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2765) ThriftSource spaws too many threads
[
https://issues.apache.org/jira/browse/FLUME-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14697396#comment-14697396
]
Ashish Paliwal commented on FLUME-2765:
---
A thread dump plus a simple test case that can help us re-create the scenario
would be a big plus. Patches are Welcome :)
> ThriftSource spaws too many threads
> ---
>
> Key: FLUME-2765
> URL: https://issues.apache.org/jira/browse/FLUME-2765
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6
>Reporter: Tobias Heintz
>
> We are in the process of migrating from the old Flume to version 1.6. We are
> using the ThriftSource with the new KafkaSink. Here's what our config looks
> like:
> {code}
> agent1.channels = ch1
> agent1.sources = thriftSrc
> agent1.sinks = kafka
> agent1.channels.ch1.type = memory
> agent1.channels.ch1.capacity = 1
> agent1.channels.ch1.transactionCapacity = 500
> # THRIFT
> agent1.sources.thriftSrc.type = thrift
> agent1.sources.thriftSrc.channels = ch1
> agent1.sources.thriftSrc.bind = 0.0.0.0
> agent1.sources.thriftSrc.port = 4042
> agent1.sources.thriftSrc.threads = 150 # if we don't set this option, the
> source keeps creating more and more threads until all heap memory is used up
> and then it crashes
> # KAFKA
> agent1.sinks.kafka.channel = ch1
> agent1.sinks.kafka.type = org.apache.flume.sink.kafka.KafkaSink
> agent1.sinks.kafka.batchSize = 50
> agent1.sinks.kafka.brokerList = broker.example.com:9092
> agent1.sinks.kafka.requiredAcks = 1
> agent1.sinks.kafka.topic = topic1
> {code}
> We have been noticing some bad behavior by the Thrift source/Thrift server
> using the JMX connection. If we don't restrict the number of threads, it
> spawns thousands of new threads, apparently one for every message it
> receives. These threads all have the name "Flume Thrift IPC Thread [number]"
> and according to the jvisualvm console they are always idle. At some point
> all of the JVM memory is used up through creating new threads and flume
> crashes with the following exception:
> {code}
> 12 Aug 2015 16:56:11,721 ERROR [Thread-1]
> (org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run:544) -
> run() exiting due to uncaught error
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
> at
> org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(TThreadedSelectorServer.java:310)
> at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:209)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:576)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:536)
> {code}
> When we set the option to restrict the number of threads, the server sticks
> to that number and runs smoothly, however it drops messages occasionally (may
> have a different cause).
> I am wondering whether this is a bug or in some way expected behavior? What
> are the best practices for using a ThriftSource? Are there further parameters
> to possibly tune (like channel.capacity)?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2498) Implement Taildir Source
[ https://issues.apache.org/jira/browse/FLUME-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663052#comment-14663052 ] Ashish Paliwal commented on FLUME-2498: --- +1, I think once we have this gets committed other user can provide patches. > Implement Taildir Source > > > Key: FLUME-2498 > URL: https://issues.apache.org/jira/browse/FLUME-2498 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Reporter: Satoshi Iijima > Fix For: v1.7.0 > > Attachments: FLUME-2498-2.patch, FLUME-2498-3.patch, FLUME-2498.patch > > > This is the proposal of implementing a new tailing source. > This source watches the specified files, and tails them in nearly real-time > once appends are detected to these files. > * This source is reliable and will not miss data even when the tailing files > rotate. > * It periodically writes the last read position of each file in a position > file using the JSON format. > * If Flume is stopped or down for some reason, it can restart tailing from > the position written on the existing position file. > * It can add event headers to each tailing file group. > A attached patch includes a config documentation of this. > This source requires Unix-style file system and Java 1.7 or later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2757) Hive Sink Tests failing due to incorrect permission on hive scratch dir
[
https://issues.apache.org/jira/browse/FLUME-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662170#comment-14662170
]
Ashish Paliwal commented on FLUME-2757:
---
Ok, a validation would be great.
> Hive Sink Tests failing due to incorrect permission on hive scratch dir
> ---
>
> Key: FLUME-2757
> URL: https://issues.apache.org/jira/browse/FLUME-2757
> Project: Flume
> Issue Type: Bug
> Components: Test
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2757.patch
>
>
> Error Message
> {code}
> java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be
> writable. Current permissions are: rwxr-xr-x
> {code}
> Stacktrace
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir:
> /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
> at org.apache.flume.sink.hive.TestHiveSink.(TestHiveSink.java:109)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on
> HDFS should be writable. Current permissions are: rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:529)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:478)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430)
> ... 29 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2757) Hive Sink Tests failing due to incorrect permission on hive scratch dir
[
https://issues.apache.org/jira/browse/FLUME-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662159#comment-14662159
]
Ashish Paliwal commented on FLUME-2757:
---
+1 Fix looks good to me. Do you want me to commit this?
> Hive Sink Tests failing due to incorrect permission on hive scratch dir
> ---
>
> Key: FLUME-2757
> URL: https://issues.apache.org/jira/browse/FLUME-2757
> Project: Flume
> Issue Type: Bug
> Components: Test
>Reporter: Roshan Naik
>Assignee: Roshan Naik
> Attachments: FLUME-2757.patch
>
>
> Error Message
> {code}
> java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be
> writable. Current permissions are: rwxr-xr-x
> {code}
> Stacktrace
> {code}
> java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir:
> /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
> at org.apache.flume.sink.hive.TestHiveSink.(TestHiveSink.java:109)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.junit.runners.BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:187)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.runReflectiveCall(BlockJUnit4ClassRunner.java:236)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.BlockJUnit4ClassRunner.methodBlock(BlockJUnit4ClassRunner.java:233)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on
> HDFS should be writable. Current permissions are: rwxr-xr-x
> at
> org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:529)
> at
> org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:478)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430)
> ... 29 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2753) Error when specifying empty replace string in Search and Replace Interceptor
[ https://issues.apache.org/jira/browse/FLUME-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660597#comment-14660597 ] Ashish Paliwal commented on FLUME-2753: --- +1 Running the tests before committing > Error when specifying empty replace string in Search and Replace Interceptor > > > Key: FLUME-2753 > URL: https://issues.apache.org/jira/browse/FLUME-2753 > Project: Flume > Issue Type: Bug >Reporter: Johny Rufus >Assignee: Johny Rufus > Attachments: FLUME-2753-1.patch, FLUME-2753.patch > > > The user guide specifies that the following configuration > a1.sources.avroSrc.interceptors.search-replace.replaceString = > will replace the search pattern with an empty string, but the following error > is observed and there is no way to specify an empty replace string > java.lang.NullPointerException: Must supply a replacement string > replaceString (empty is ok) > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108) > at > org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110) > We need to modify the SearchAndReplaceInterceptor to accept null values for > replacement string and use empty string in that case -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2753) Error when specifying empty replace string in Search and Replace Interceptor
[ https://issues.apache.org/jira/browse/FLUME-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654061#comment-14654061 ] Ashish Paliwal commented on FLUME-2753: --- Can we have a test case for this? > Error when specifying empty replace string in Search and Replace Interceptor > > > Key: FLUME-2753 > URL: https://issues.apache.org/jira/browse/FLUME-2753 > Project: Flume > Issue Type: Bug >Reporter: Johny Rufus >Assignee: Johny Rufus > Attachments: FLUME-2753.patch > > > The user guide specifies that the following configuration > a1.sources.avroSrc.interceptors.search-replace.replaceString = > will replace the search pattern with an empty string, but the following error > is observed and there is no way to specify an empty replace string > java.lang.NullPointerException: Must supply a replacement string > replaceString (empty is ok) > at > com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) > at > org.apache.flume.interceptor.SearchAndReplaceInterceptor$Builder.configure(SearchAndReplaceInterceptor.java:108) > at > org.apache.flume.channel.ChannelProcessor.configureInterceptors(ChannelProcessor.java:110) > We need to modify the SearchAndReplaceInterceptor to accept null values for > replacement string and use empty string in that case -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2660) Add documentation for EventValidator
[ https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647004#comment-14647004 ] Ashish Paliwal commented on FLUME-2660: --- The site is updated, already has this fix > Add documentation for EventValidator > > > Key: FLUME-2660 > URL: https://issues.apache.org/jira/browse/FLUME-2660 > Project: Flume > Issue Type: Bug >Affects Versions: v1.5.1 >Reporter: Hari Shreedharan >Assignee: Ashish Paliwal > Fix For: v1.7.0 > > Attachments: FLUME-2660-0.patch, FLUME-2660-1.patch > > > [~paliwalashish] - Assigning this to you. Please add docs for the > functionality you contributed in FLUME-2613 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2660) Add documentation for EventValidator
[ https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646791#comment-14646791 ] Ashish Paliwal commented on FLUME-2660: --- [~jrufus] Fixed the build issue with the patch. Would be great if you can commit the same. > Add documentation for EventValidator > > > Key: FLUME-2660 > URL: https://issues.apache.org/jira/browse/FLUME-2660 > Project: Flume > Issue Type: Bug >Affects Versions: v1.5.1 >Reporter: Hari Shreedharan >Assignee: Ashish Paliwal > Fix For: v1.7.0 > > Attachments: FLUME-2660-0.patch, FLUME-2660-1.patch > > > [~paliwalashish] - Assigning this to you. Please add docs for the > functionality you contributed in FLUME-2613 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2660) Add documentation for EventValidator
[ https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2660: -- Attachment: FLUME-2660-1.patch > Add documentation for EventValidator > > > Key: FLUME-2660 > URL: https://issues.apache.org/jira/browse/FLUME-2660 > Project: Flume > Issue Type: Bug >Affects Versions: v1.5.1 >Reporter: Hari Shreedharan >Assignee: Ashish Paliwal > Fix For: v1.7.0 > > Attachments: FLUME-2660-0.patch, FLUME-2660-1.patch > > > [~paliwalashish] - Assigning this to you. Please add docs for the > functionality you contributed in FLUME-2613 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (FLUME-2660) Add documentation for EventValidator
[ https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal reopened FLUME-2660: --- The patch has build issue, re opening > Add documentation for EventValidator > > > Key: FLUME-2660 > URL: https://issues.apache.org/jira/browse/FLUME-2660 > Project: Flume > Issue Type: Bug >Affects Versions: v1.5.1 >Reporter: Hari Shreedharan >Assignee: Ashish Paliwal > Fix For: v1.7.0 > > Attachments: FLUME-2660-0.patch > > > [~paliwalashish] - Assigning this to you. Please add docs for the > functionality you contributed in FLUME-2613 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2660) Add documentation for EventValidator
[ https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639063#comment-14639063 ] Ashish Paliwal commented on FLUME-2660: --- [~jrufus] Can you take a quick look at this? > Add documentation for EventValidator > > > Key: FLUME-2660 > URL: https://issues.apache.org/jira/browse/FLUME-2660 > Project: Flume > Issue Type: Bug >Affects Versions: v1.5.1 >Reporter: Hari Shreedharan >Assignee: Ashish Paliwal > Attachments: FLUME-2660-0.patch > > > [~paliwalashish] - Assigning this to you. Please add docs for the > functionality you contributed in FLUME-2613 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2746) How to include this Flume Patch in Flume 1.5.2 ?
[ https://issues.apache.org/jira/browse/FLUME-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14639052#comment-14639052 ] Ashish Paliwal commented on FLUME-2746: --- For #1 Download the source (not sure if you can download Hortonworks distro code, this would work for ASF code), apply the patch, resolve any conflicts. This should be easy as this code lives in its own package. Make a build, resolve any potential build issues and you have a binary build ready for use. For# 2, no ETA as of now, we shall try to pick this up soon. It's been on wanted list for long > How to include this Flume Patch in Flume 1.5.2 ? > > > Key: FLUME-2746 > URL: https://issues.apache.org/jira/browse/FLUME-2746 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1, v1.7.0 > Environment: 1. We run Hortontowns flume 1.5.2 but how can we include > this patch (sourceType) -- > https://issues.apache.org/jira/secure/attachment/12692432/FLUME-2498-2.patch > in Flume 1.5.2 ? > FLUME-2498-2.patch > 2. When is FLume 1.7 which has this sourceType (+Taildir Source) available > for Install ? >Reporter: sutanu das > Fix For: v1.7.0 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2053) TestExecSource.testMonitoredCounterGroup fails
[ https://issues.apache.org/jira/browse/FLUME-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633958#comment-14633958 ] Ashish Paliwal commented on FLUME-2053: --- [~jrufus] Please go ahead > TestExecSource.testMonitoredCounterGroup fails > -- > > Key: FLUME-2053 > URL: https://issues.apache.org/jira/browse/FLUME-2053 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: v1.4.0, v1.5.1 > Environment: Ubuntu 13.04 > Java(TM) SE Runtime Environment (build 1.7.0_13-b20) > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode) > Maven 3.0.5 > Reporter: Hans Uhlig >Assignee: Ashish Paliwal > > Failed tests: > testMonitoredCounterGroup(org.apache.flume.source.TestExecSource): Expected > Value: 5 expected:<[5]> but was:<[0]> -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2713) Document Fault Tolerant Config parameters in FlumeUserGuide
[ https://issues.apache.org/jira/browse/FLUME-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617900#comment-14617900 ] Ashish Paliwal commented on FLUME-2713: --- Thanks [~jrufus] Shall take care next time :) > Document Fault Tolerant Config parameters in FlumeUserGuide > --- > > Key: FLUME-2713 > URL: https://issues.apache.org/jira/browse/FLUME-2713 > Project: Flume > Issue Type: Documentation >Reporter: Johny Rufus >Assignee: Johny Rufus > Fix For: v1.7.0 > > Attachments: FLUME-2713.patch > > > The following FaultTolerance related parameters in MorphlineSolrSink need to > be documented in Flume user guide > FaultTolerance.IS_PRODUCTION_MODE > FaultTolerance.IS_IGNORING_RECOVERABLE_EXCEPTIONS > FaultTolerance.RECOVERABLE_EXCEPTION_CLASSES -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2713) Document Fault Tolerant Config parameters in FlumeUserGuide
[ https://issues.apache.org/jira/browse/FLUME-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617814#comment-14617814 ] Ashish Paliwal commented on FLUME-2713: --- [~hshreedharan] I committed on both trunk and 1.7 branch as per Committers guide. Let me know if this is correct or just commit in trunk. > Document Fault Tolerant Config parameters in FlumeUserGuide > --- > > Key: FLUME-2713 > URL: https://issues.apache.org/jira/browse/FLUME-2713 > Project: Flume > Issue Type: Documentation >Reporter: Johny Rufus >Assignee: Johny Rufus > Fix For: v1.7.0 > > Attachments: FLUME-2713.patch > > > The following FaultTolerance related parameters in MorphlineSolrSink need to > be documented in Flume user guide > FaultTolerance.IS_PRODUCTION_MODE > FaultTolerance.IS_IGNORING_RECOVERABLE_EXCEPTIONS > FaultTolerance.RECOVERABLE_EXCEPTION_CLASSES -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2713) Document Fault Tolerant Config parameters in FlumeUserGuide
[ https://issues.apache.org/jira/browse/FLUME-2713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617807#comment-14617807 ] Ashish Paliwal commented on FLUME-2713: --- Sure, missed that part. +1 committing > Document Fault Tolerant Config parameters in FlumeUserGuide > --- > > Key: FLUME-2713 > URL: https://issues.apache.org/jira/browse/FLUME-2713 > Project: Flume > Issue Type: Documentation >Reporter: Johny Rufus >Assignee: Johny Rufus > Fix For: v1.7.0 > > Attachments: FLUME-2713.patch > > > The following FaultTolerance related parameters in MorphlineSolrSink need to > be documented in Flume user guide > FaultTolerance.IS_PRODUCTION_MODE > FaultTolerance.IS_IGNORING_RECOVERABLE_EXCEPTIONS > FaultTolerance.RECOVERABLE_EXCEPTION_CLASSES -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2632) High CPU on KafkaSink
[ https://issues.apache.org/jira/browse/FLUME-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617692#comment-14617692 ] Ashish Paliwal commented on FLUME-2632: --- [~gwenshap] Can you have a look at it, minor fix. > High CPU on KafkaSink > - > > Key: FLUME-2632 > URL: https://issues.apache.org/jira/browse/FLUME-2632 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 >Reporter: Gwen Shapira >Assignee: Gwen Shapira > Labels: kafka > Fix For: 1.6 > > Attachments: FLUME-2632-0.patch > > > Reported here: https://github.com/harishreedharan/flume/issues/1 > "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu > utilization stay at 100% and never dropped down all the time even at the time > the channel is empty. > I looked into the source code and found that "process" function in KafkaSink > always return Status.READY even if no events available in channel. That > causes the SinkRunner keep running achieving event from channel and get > nothing. > Do we need to change to return Status.BACKOFF in "process" function in > KafkaSink when it notices that there is no events processed in current round? > So that the SinkRunner has a chance to take a rest when there is no event in > channel. If this proposal feasible, function "testEmptyChannel" in > TestKafkaSink also need to be changed. " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2632) High CPU on KafkaSink
[ https://issues.apache.org/jira/browse/FLUME-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617692#comment-14617692 ] Ashish Paliwal edited comment on FLUME-2632 at 7/8/15 12:02 AM: [~gwenshap] Can you please have a look at it, minor fix. was (Author: paliwalashish): [~gwenshap] Can you have a look at it, minor fix. > High CPU on KafkaSink > - > > Key: FLUME-2632 > URL: https://issues.apache.org/jira/browse/FLUME-2632 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 >Reporter: Gwen Shapira >Assignee: Gwen Shapira > Labels: kafka > Fix For: 1.6 > > Attachments: FLUME-2632-0.patch > > > Reported here: https://github.com/harishreedharan/flume/issues/1 > "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu > utilization stay at 100% and never dropped down all the time even at the time > the channel is empty. > I looked into the source code and found that "process" function in KafkaSink > always return Status.READY even if no events available in channel. That > causes the SinkRunner keep running achieving event from channel and get > nothing. > Do we need to change to return Status.BACKOFF in "process" function in > KafkaSink when it notices that there is no events processed in current round? > So that the SinkRunner has a chance to take a rest when there is no event in > channel. If this proposal feasible, function "testEmptyChannel" in > TestKafkaSink also need to be changed. " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2672) NPE in KafkaSourceCounter
[
https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617453#comment-14617453
]
Ashish Paliwal commented on FLUME-2672:
---
Getting an error while applying the patch.
{code}
bash-3.2$ git apply --check FLUME-2672-with-test.patch
error:
src/main/java/org/apache/flume/instrumentation/kafka/KafkaSourceCounter.java:
No such file or directory
{code}
> NPE in KafkaSourceCounter
> -
>
> Key: FLUME-2672
> URL: https://issues.apache.org/jira/browse/FLUME-2672
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.6.0
> Environment: Mac OS 10.10.3, Java 1.7.0_60
>Reporter: Rigo MacTaggart
>Priority: Trivial
> Labels: easyfix
> Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount()
> because it expects MonitoredCounterGroup.counterMap to contain key
> "source.kafka.empty.count". A patch is included which adds this key to the
> ATTRIBUTES string array, which is used to pre-populate keys with an initial
> value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2672) NPE in KafkaSourceCounter
[ https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615981#comment-14615981 ] Ashish Paliwal commented on FLUME-2672: --- Let me check and commit. > NPE in KafkaSourceCounter > - > > Key: FLUME-2672 > URL: https://issues.apache.org/jira/browse/FLUME-2672 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 > Environment: Mac OS 10.10.3, Java 1.7.0_60 >Reporter: Rigo MacTaggart >Priority: Trivial > Labels: easyfix > Attachments: FLUME-2672-with-test.patch, FLUME-2672.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() > because it expects MonitoredCounterGroup.counterMap to contain key > "source.kafka.empty.count". A patch is included which adds this key to the > ATTRIBUTES string array, which is used to pre-populate keys with an initial > value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2736) issue in exporting flipkart data from web url to HDFS
[ https://issues.apache.org/jira/browse/FLUME-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14615357#comment-14615357 ] Ashish Paliwal commented on FLUME-2736: --- HTTP Source is not a pull source, you need to push events to it via Get or put. It won't pull data from a URL. This what I could understood so far. As I said pls ask questions on User Mailing List not JIRA. I am closing this JIRA, please send email to ML. > issue in exporting flipkart data from web url to HDFS > - > > Key: FLUME-2736 > URL: https://issues.apache.org/jira/browse/FLUME-2736 > Project: Flume > Issue Type: Task > Components: Sinks+Sources >Affects Versions: v1.5.0 >Reporter: swayam > Original Estimate: 24h > Remaining Estimate: 24h > > Dear Team, > I am trying to export the flipkart's json formatted data from URL > > "https://affiliate-api.flipkart.net/affiliate/api/31873d12df9e46499e535333fdee9dad.json";. > Please help us . > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLUME-2736) issue in exporting flipkart data from web url to HDFS
[ https://issues.apache.org/jira/browse/FLUME-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal resolved FLUME-2736. --- Resolution: Invalid User ML Question > issue in exporting flipkart data from web url to HDFS > - > > Key: FLUME-2736 > URL: https://issues.apache.org/jira/browse/FLUME-2736 > Project: Flume > Issue Type: Task > Components: Sinks+Sources >Affects Versions: v1.5.0 >Reporter: swayam > Original Estimate: 24h > Remaining Estimate: 24h > > Dear Team, > I am trying to export the flipkart's json formatted data from URL > > "https://affiliate-api.flipkart.net/affiliate/api/31873d12df9e46499e535333fdee9dad.json";. > Please help us . > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2736) issue in exporting flipkart data from web url to HDFS
[ https://issues.apache.org/jira/browse/FLUME-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614464#comment-14614464 ] Ashish Paliwal commented on FLUME-2736: --- Not sure what are you trying to achieve here. Can you pls ask questions on User Mailing Lists http://flume.apache.org/mailinglists.html > issue in exporting flipkart data from web url to HDFS > - > > Key: FLUME-2736 > URL: https://issues.apache.org/jira/browse/FLUME-2736 > Project: Flume > Issue Type: Task > Components: Sinks+Sources >Affects Versions: v1.5.0 >Reporter: swayam > Original Estimate: 24h > Remaining Estimate: 24h > > Dear Team, > I am trying to export the flipkart's json formatted data from URL > > "https://affiliate-api.flipkart.net/affiliate/api/31873d12df9e46499e535333fdee9dad.json";. > Please help us . > Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2730) Flume execution time - configuration
[ https://issues.apache.org/jira/browse/FLUME-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614015#comment-14614015 ] Ashish Paliwal commented on FLUME-2730: --- There is nothing in configuration that shall allow you to achieve this. Have a script which kill the pid after the specified time. > Flume execution time - configuration > > > Key: FLUME-2730 > URL: https://issues.apache.org/jira/browse/FLUME-2730 > Project: Flume > Issue Type: Question >Reporter: Janarthanan >Priority: Critical > > I wanted the flume agent to exit after a specific time. I set the batch time > in the configuration file, but still the agent runs beyond the time limit set > in the configuration file. > Please suggest a way to terminate the agent after 10 minutes. > --configuration file content > # Name the components on this agent > a1.sources = r1 > a1.sinks = k1 > a1.channels = c1 > # Describe/configure the source > a1.sources.r1.type = spooldir > a1.sources.r1.spoolDir = /Users/administrator/temp > a1.sources.r1.bind = localhost > a1.sources.r1.port = 4 > a1.sources.r1.batchSize = 1 > a1.sources.r1.maxBatchDurationMillis = 200 > # Describe the sink > a1.sinks.k1.type = logger > # Use a channel which buffers events in memory > a1.channels.c1.type = memory > a1.channels.c1.capacity = 1000 > a1.channels.c1.transactionCapacity = 100 > # Bind the source and sink to the channel > a1.sources.r1.channels = c1 > a1.sinks.k1.channel = c1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2729) Allow pollableSource backoff times to be configurable
[ https://issues.apache.org/jira/browse/FLUME-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14610989#comment-14610989 ] Ashish Paliwal commented on FLUME-2729: --- Will be good to create a review request > Allow pollableSource backoff times to be configurable > -- > > Key: FLUME-2729 > URL: https://issues.apache.org/jira/browse/FLUME-2729 > Project: Flume > Issue Type: Improvement >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Attachments: FLUME-2729.patch.2, FLUME_2729.patch > > > There are use cases that require more control of the backoff times setting. > This jira will allow those times to be set through config of the source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2729) Allow pollableSource backoff times to be configurable
[ https://issues.apache.org/jira/browse/FLUME-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14602212#comment-14602212 ] Ashish Paliwal commented on FLUME-2729: --- It's just a suggestion :) I know Guava troubles so won't suggest it for now, but just that you know something exists :) > Allow pollableSource backoff times to be configurable > -- > > Key: FLUME-2729 > URL: https://issues.apache.org/jira/browse/FLUME-2729 > Project: Flume > Issue Type: Improvement >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Attachments: FLUME_2729.patch > > > There are use cases that require more control of the backoff times setting. > This jira will allow those times to be set through config of the source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2729) Allow pollableSource backoff times to be configurable
[ https://issues.apache.org/jira/browse/FLUME-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601601#comment-14601601 ] Ashish Paliwal commented on FLUME-2729: --- I didn't look much into the patch, but I did used Guava retrying lib for a use case and it had some wonderful constructs like simple time based back-off, exponential back-off etc. Would be great if we can use something similar here. Just a suggestion > Allow pollableSource backoff times to be configurable > -- > > Key: FLUME-2729 > URL: https://issues.apache.org/jira/browse/FLUME-2729 > Project: Flume > Issue Type: Improvement >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Attachments: FLUME_2729.patch > > > There are use cases that require more control of the backoff times setting. > This jira will allow those times to be set through config of the source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2726) Remove * imports from Kafka Channel
Ashish Paliwal created FLUME-2726: - Summary: Remove * imports from Kafka Channel Key: FLUME-2726 URL: https://issues.apache.org/jira/browse/FLUME-2726 Project: Flume Issue Type: Bug Components: Channel Affects Versions: v1.6.0 Reporter: Ashish Paliwal Priority: Trivial As per Flume dev guide, we prefer explicit imports. Need to remove * imports and replace with explicit imports -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [ANNOUNCE] New Flume committer - Johny Rufus
Yay... Congrats ! On Fri, Jun 19, 2015 at 2:09 PM, Roshan Naik wrote: > Johny, >Thanks for your help with Flume. Congratulations ! > -roshan > > From: Hari Shreedharan > mailto:hshreedha...@apache.org>> > Reply-To: "u...@flume.apache.org<mailto:u...@flume.apache.org>" > mailto:u...@flume.apache.org>> > Date: Friday, June 19, 2015 1:38 PM > To: "dev@flume.apache.org<mailto:dev@flume.apache.org>" > mailto:dev@flume.apache.org>>, > "u...@flume.apache.org<mailto:u...@flume.apache.org>" > mailto:u...@flume.apache.org>> > Subject: [ANNOUNCE] New Flume committer - Johny Rufus > > On behalf of the Apache Flume PMC, I am excited to welcome Johny Rufus as a > committer on the Apache Flume project. Johny has actively contributed several > patches to the Flume project, including bug fixes, authentication and other > new features. > > Congratulations and Welcome, Johny! > > > Cheers, > Hari Shreedharan -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Comment Edited] (FLUME-2720) HDFS Sink: Autogenerate LZO indexes while creating LZO files
[ https://issues.apache.org/jira/browse/FLUME-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588785#comment-14588785 ] Ashish Paliwal edited comment on FLUME-2720 at 6/16/15 9:13 PM: IMHO we should have atleast some basic tests, if possible. We may choose to create another JIRA to keep track of the test cases but we never know if someone shall pick it up for implementation (general concern, not specific to this) Since you mentioned a major refactoring, I am assuming it would help other test cases as well. So we can raise JIRA for the test cases. Also can we align the patch names as specified in our guide. was (Author: paliwalashish): IMHO we should have atleast some basic tests, if possible. We may choose to create another JIRA to keep track of the test cases but we never know if someone shall pick it up for implementation (general concern, not specific to this) Since you mentioned a major refactoring, I am assuming it would help other test cases as well. So we can raise JIRA for the test cases. Also can we align the patch names as specified in out guide. > HDFS Sink: Autogenerate LZO indexes while creating LZO files > > > Key: FLUME-2720 > URL: https://issues.apache.org/jira/browse/FLUME-2720 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Hari >Priority: Minor > Attachments: > 0001-FLUME-2720-Autogenerate-LZO-indexes-while-creating-L.patch > > > The LZO indexes are now generated offline using DistributedLZOIndexer. > It will be nice to auto generate these index files during the ingestion > itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2720) HDFS Sink: Autogenerate LZO indexes while creating LZO files
[ https://issues.apache.org/jira/browse/FLUME-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588785#comment-14588785 ] Ashish Paliwal commented on FLUME-2720: --- IMHO we should have atleast some basic tests, if possible. We may choose to create another JIRA to keep track of the test cases but we never know if someone shall pick it up for implementation (general concern, not specific to this) Since you mentioned a major refactoring, I am assuming it would help other test cases as well. So we can raise JIRA for the test cases. Also can we align the patch names as specified in out guide. > HDFS Sink: Autogenerate LZO indexes while creating LZO files > > > Key: FLUME-2720 > URL: https://issues.apache.org/jira/browse/FLUME-2720 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: Hari >Priority: Minor > Attachments: > 0001-FLUME-2720-Autogenerate-LZO-indexes-while-creating-L.patch > > > The LZO indexes are now generated offline using DistributedLZOIndexer. > It will be nice to auto generate these index files during the ingestion > itself. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582693#comment-14582693 ] Ashish Paliwal commented on FLUME-2437: --- I am tied up with my relocation. Would be great if someone else can look into this till i get free. > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Ashish Paliwal > Attachments: FLUME-2437-2.patch, FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release Apache Flume version 1.6.0 RC1
+1 Build works good, all test cases pass Randomly picked few JIRA's and validate the commits looks good Thank You Johny for all the hard work. On Tue, May 12, 2015 at 11:13 PM, Johny Rufus wrote: > Hi All, > > This is the ninth release for Apache Flume as a top-level project, > version 1.6.0. We are voting on release candidate RC1. > > It fixes the following issues: > > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=CHANGELOG;h=53ea45cbd496b89fcd84c89f2ebd8d51e5bb8016;hb=f7560038a25430378f09ea631b6e472979d7988c > > *** Please cast your vote within the next 72 hours *** > > The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) > for the source and binary artifacts can be found here: > > http://people.apache.org/~hshreedharan/apache-flume-1.6.0-rc1/ > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapacheflume-1016/ > > The tag to be voted on: > > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=f7560038a25430378f09ea631b6e472979d7988c > > > Flume's KEYS file containing PGP keys we use to sign the release: > http://www.apache.org/dist/flume/KEYS > > > Thanks, > Rufus -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [ANNOUNCE] New Flume Committer - Ashish Paliwal
Thanks Otis and Gwen ! On Tue, May 12, 2015 at 11:49 AM, Gwen Shapira wrote: > Congrats Ashish and thanks for all the reviews :) > > On Fri, May 8, 2015 at 8:42 PM, Hari Shreedharan > wrote: > >> On behalf of the Apache Flume PMC, I am excited to welcome Ashish Paliwal as >> a committer on the Apache Flume project. Ashish has actively contributed >> several patches to the Flume project, including bug fixes, configuration >> improvements and other new features. >> >> Congratulations and Welcome, Ashish! >> >> >> Cheers, >> Hari Shreedharan >> -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [VOTE] Release Apache Flume version 1.6.0 RC1
The build is working good :) I shall do some more checks and send my vote later today. On Wed, May 13, 2015 at 4:13 AM, Johny Rufus wrote: > Hi All, > > This is the ninth release for Apache Flume as a top-level project, > version 1.6.0. We are voting on release candidate RC1. This will be the > last release built for Java 6. > > It fixes the following issues: > > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=CHANGELOG;h=53ea45cbd496b89fcd84c89f2ebd8d51e5bb8016;hb=2561a23240a71ba20bf288c7c2cda88f443c2080 > > *** Please cast your vote within the next 72 hours *** > > The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) > for the source and binary artifacts can be found here: > > http://people.apache.org/~hshreedharan/apache-flume-1.6.0-rc1/ > > Maven staging repo: > > https://repository.apache.org/content/repositories/orgapacheflume-1016/ > > The tag to be voted on: > > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=tag;h=24ae0e7409f057667a9756ec3d7d494a58d8ed1d > > > Flume's KEYS file containing PGP keys we use to sign the release: > http://www.apache.org/dist/flume/KEYS > > > Thanks, > Rufus -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [DISCUSS] Drop Hadoop 1 support
+1 (speaking as dev, it keeps life a bit simple) Would wait for the users to give +1's as they get directly affected and possibly a few may have to upgrade Data Infrastructure. On Wed, May 13, 2015 at 4:00 AM, Hari Shreedharan wrote: > Flume devs, > > With Flume moving away from Java 6 in Flume 1.7, I would like to start a > discussion on whether we should completely drop Hadoop-1 support. I am not > sure we can even link against Hadoop-1 if we build with Java 7 as the minimum > requirement. Even if we can, I think it might be a good idea to drop Hadoop 1 > support at this point. > > Comments? > > Thanks, > Hari Shreedharan > > > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [ANNOUNCE] New Flume Committer - Ashish Paliwal
Thanks Folks ! On Sat, May 9, 2015 at 3:09 AM, iain wright wrote: > As a long time user, thanks and congrats from all of us at Telescope! > > -- > Iain Wright > > This email message is confidential, intended only for the recipient(s) named > above and may contain information that is privileged, exempt from disclosure > under applicable law. If you are not the intended recipient, do not disclose > or disseminate the message to anyone except the intended recipient. If you > have received this message in error, or are not the named recipient(s), > please immediately notify the sender by return email, and delete all copies > of this message. > > On Fri, May 8, 2015 at 2:29 PM, Roshan Naik wrote: >> >> Congratulations Ashish ! >> >> From: Hari Shreedharan >> Reply-To: "u...@flume.apache.org" >> Date: Friday, May 8, 2015 10:42 AM >> To: "dev@flume.apache.org" , "u...@flume.apache.org" >> >> Subject: [ANNOUNCE] New Flume Committer - Ashish Paliwal >> >> On behalf of the Apache Flume PMC, I am excited to welcome Ashish Paliwal >> as a committer on the Apache Flume project. Ashish has actively contributed >> several patches to the Flume project, including bug fixes, configuration >> improvements and other new features. >> >> Congratulations and Welcome, Ashish! >> >> >> Cheers, >> Hari Shreedharan > > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [VOTE] Release Apache Flume version 1.6.0 RC0
by.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) ... 38 more Caused by: ERROR XBM0A: The database directory '/Users/ashishpaliwal/work/trash/apache-flume-1.6.0-src/flume-ng-sinks/flume-hive-sink/metastore_db' exists. However, it does not contain the expected 'service.properties' file. Perhaps Derby was brought down in the middle of creating this database. You may want to delete this directory and try creating the database again. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.services.monitor.StorageFactoryService.vetService(Unknown Source) at org.apache.derby.impl.services.monitor.StorageFactoryService.access$600(Unknown Source) at org.apache.derby.impl.services.monitor.StorageFactoryService$9.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source) ... 38 more On Sat, May 2, 2015 at 8:26 AM, Ashish wrote: > No luck with Java 7 either. > > With Java 6 and running into this error (running on Macbook Air with 4G RAM) > > org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to > zookeeper server within timeout: 1000 > > at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880) > > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:98) > > at org.I0Itec.zkclient.ZkClient.(ZkClient.java:84) > > at > kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:156) > > at > kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:114) > > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:65) > > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:67) > > at > kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100) > > at > kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala) > > at > org.apache.flume.sink.kafka.util.KafkaConsumer.(KafkaConsumer.java:51) > > at > org.apache.flume.sink.kafka.util.TestUtil.getKafkaConsumer(TestUtil.java:120) > > at org.apache.flume.sink.kafka.util.TestUtil.prepare(TestUtil.java:145) > > at org.apache.flume.sink.kafka.TestKafkaSink.setup(TestKafkaSink.java:51) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) > > at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) > > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) > > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) > > at > org.apache.maven.surefire.booter.ForkedBoot
Re: [VOTE] Release Apache Flume version 1.6.0 RC0
No luck with Java 7 either. With Java 6 and running into this error (running on Macbook Air with 4G RAM) org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 1000 at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880) at org.I0Itec.zkclient.ZkClient.(ZkClient.java:98) at org.I0Itec.zkclient.ZkClient.(ZkClient.java:84) at kafka.consumer.ZookeeperConsumerConnector.connectZk(ZookeeperConsumerConnector.scala:156) at kafka.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:114) at kafka.javaapi.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:65) at kafka.javaapi.consumer.ZookeeperConsumerConnector.(ZookeeperConsumerConnector.scala:67) at kafka.consumer.Consumer$.createJavaConsumerConnector(ConsumerConnector.scala:100) at kafka.consumer.Consumer.createJavaConsumerConnector(ConsumerConnector.scala) at org.apache.flume.sink.kafka.util.KafkaConsumer.(KafkaConsumer.java:51) at org.apache.flume.sink.kafka.util.TestUtil.getKafkaConsumer(TestUtil.java:120) at org.apache.flume.sink.kafka.util.TestUtil.prepare(TestUtil.java:145) at org.apache.flume.sink.kafka.TestKafkaSink.setup(TestKafkaSink.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Results : Tests in error: org.apache.flume.sink.kafka.TestKafkaSink: Unable to connect to zookeeper server within timeout: 1000 On Thu, Apr 30, 2015 at 10:11 PM, Hari Shreedharan wrote: > We don't really support or test against Java 8. I think there are several > jiras open to fix Flume to work properly against Java 8, though I don't > think all of them have any work on it yet. Can you test against Java 7? > > On Thu, Apr 30, 2015 at 5:11 AM, Ashish wrote: > >> Have trouble with the build. I don't think it's specific to this >> release. I have seen this in past. >> >> Building with jdk1.8.0_25 >> Encountering the following error (tried 4-5 times), so far haven't >> been able to get a working build >> >> log4j:WARN No appenders could be found for logger >> (org.apache.flume.channel.file.FileChannel). >> log4j:WARN Please initialize the log4j system properly. >> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >> for more info. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> Attempting to shutdown background worker. >> src : [ 1106 ms ]. >> sink : [ 2232 ms ]. >>
Re: [VOTE] Release Apache Flume version 1.6.0 RC0
eMemoryTransaction.putCommit(SpillableMemoryChannel.java:480) at org.apache.flume.channel.SpillableMemoryChannel$SpillableMemoryTransaction.doCommit(SpillableMemoryChannel.java:401) at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) at org.apache.flume.channel.TestSpillableMemoryChannel.transactionalPutN(TestSpillableMemoryChannel.java:157) at org.apache.flume.channel.TestSpillableMemoryChannel.access$000(TestSpillableMemoryChannel.java:43) at org.apache.flume.channel.TestSpillableMemoryChannel$1.run(TestSpillableMemoryChannel.java:230) On Thu, Apr 30, 2015 at 1:56 AM, Johny Rufus wrote: > Hi All, > > This is the ninth release for Apache Flume as a top-level project, > version 1.6.0. We are voting on release candidate RC0. > > It fixes the following issues: > > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=CHANGELOG;h=774aced731de1e49043c179a722e55feb69f1b29;hb=493976e20dfe14b0b611c92f3e160d4336d10af2 > > *** Please cast your vote within the next 72 hours *** > > The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) > for the source and binary artifacts can be found here: > > http://people.apache.org/~hshreedharan/apache-flume-1.6.0-rc0/ > > Maven staging repo: > > *https://repository.apache.org/content/repositories/orgapacheflume-1013/ > <https://repository.apache.org/content/repositories/orgapacheflume-1013/>* > > The tag to be voted on: > > https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=30af6e90603d476aa058d2736c2ae154acff4af9 > > > Flume's KEYS file containing PGP keys we use to sign the release: > http://www.apache.org/dist/flume/KEYS > > > Thanks, > Rufus -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Commented] (FLUME-2661) Support timeout in ES Sink
[ https://issues.apache.org/jira/browse/FLUME-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505027#comment-14505027 ] Ashish Paliwal commented on FLUME-2661: --- The problem is valid so let's keep this open. There might be an alternate way to fix this. The deprecated API might have been replaced with some other API, we need to see if we can support something similar. > Support timeout in ES Sink > -- > > Key: FLUME-2661 > URL: https://issues.apache.org/jira/browse/FLUME-2661 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 >Reporter: Ashish Paliwal > > Credits to Otis, From ML > Hi, > I don't have a very good bug report, but I want to share one problem with > Flume we observed the other day. Specifically, with Elasticsearch Sink and > AWS ELB, but others may have this problem, too. > In short, ELB has this notion of a Spillover where it can reject requests > when it can't handle the load. Unfortunately, it doesn't actually signal > this by closing the connection, so Flume's connection remains open, I > believe in some java.net... readSocket() method. > To work around this we had to add a socket timeout in our custom version of > Flume's Elasticsearch Sink. > I think it's worth fixing this in Flume and checking if other Sinks need > this sort of timeout. > HTH > Otis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Fix version updates?
Thanks Johny ! Got it. Somehow I had a feeling that Kafka stuff went into a release. Might be some race condition in my head :) thanks ashish On Fri, Apr 17, 2015 at 10:18 PM, Johny Rufus wrote: > Hi Ashish, > > FLUME-2251 is not in 1.5.1, if you look at the commits for tag > release-1.5.1 (Correct me if am wrong) > Even I initially assumed to go by resolved date (and had a set of Jiras > that got resolved between 1.5.0 release and 1.5.1 release, but still did > not have a Fix Version, change to 1.5.1), but Hari clarified that > 1.5.1/1.5.2 had very specific commits, and not everything that was in > resolved state at the time of 1.5.1/1.5.2 went into those releases. Based > on this, I had to re-change what I changed from no-fix-version to 1.5.1, > to 1.6. > > I ended up following the query in the JIRA cleanup section - > https://cwiki.apache.org/confluence/display/FLUME/How+to+Release#HowtoRelease-JIRAcleanup > to update the Fix version. > > Thanks, > Rufus > > On Thu, Apr 16, 2015 at 10:16 PM, Ashish wrote: > >> I am seeing a lot of email with fix version being updated from 1.5.1 >> to 1.6.0, like https://issues.apache.org/jira/browse/FLUME-2251 >> >> I believe this went into 1.5.1. >> >> Just curious on how JIRA's version update is to be done for a release >> (for my benefit) >> >> thanks >> ashish >> -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Fix version updates?
I am seeing a lot of email with fix version being updated from 1.5.1 to 1.6.0, like https://issues.apache.org/jira/browse/FLUME-2251 I believe this went into 1.5.1. Just curious on how JIRA's version update is to be done for a release (for my benefit) thanks ashish
Re: Flume 1.6 jiras
Is it possible to push the pending JIRA's in trunk now? Keen on getting S3 source and tail dir in the code base. The idea is to enable faster collaboration on them, S3 source needs a lot of love. Once in trunk, lot many people may submit patches to get it working. Same holds true for tail dir. I shall also try to get the cadahale patch working with trunk and get it in. It would need a serious look as it might bring in some dependencies. Thoughts? On Thu, Apr 16, 2015 at 3:49 AM, Hari Shreedharan wrote: > Once we branch for 1.6 release (making trunk Java 7+ only), I will review the > tail source > > > > > Thanks, Hari > > On Wed, Apr 15, 2015 at 11:14 AM, Johny Rufus wrote: > >> For tail source we definitely have to go with Java 7, so +1, for having a >> 1.7 release soon, with tail and s3 sources. >> Thanks, >> Rufus >> On Tue, Apr 14, 2015 at 4:27 PM, Hari Shreedharan >> wrote: >>> I realize that a lot of you wanted the tail source and s3 source. I really >>> don’t have the bandwidth to get it reviewed right now and I would like to >>> get the Flume 1.6 release out soon. We can come back and do a 1.7 release >>> with Java 7 as minimum required version in a month or something, and we can >>> get both these in by then? >>> >>> >>> >>> >>> Thanks, Hari >>> >>> On Sun, Apr 12, 2015 at 10:11 AM, Hari Shreedharan >>> wrote: >>> >>> > I think Johnny is working on it now, so I'd expect an RC put in the next >>> few days! >>> > Thanks, Hari >>> > On Sun, Apr 12, 2015 at 3:43 AM, 李响 wrote: >>> >> Hi Johny and harry, >>> >> I think it is a must to include FLUME-2575 >>> >> <https://issues.apache.org/jira/browse/FLUME-2575>. Do you agree ? >>> >> On Sat, Apr 11, 2015 at 11:21 AM, Ashish >>> wrote: >>> >>> This jira FLUME-2632 - High CPU on KafkaSink, might also be worth >>> adding >>> >>> to 1.6 >>> >>> It's a simple fix. >>> >>> >>> >>> On Tue, Apr 7, 2015 at 7:21 AM, Ashish >>> wrote: >>> >>> > If I am not late for the party, here are some JIRA's needing some >>> >>> attention >>> >>> > >>> >>> > FLUME-2613 - Tool/script for deleting individual message from queue >>> >>> > FLUME-2589 - Add Related Projects Page >>> >>> > FLUME-2413 - Add a new metric for the number of bytes transferred on >>> a >>> >>> > flume channel >>> >>> > >>> >>> > On Tue, Apr 7, 2015 at 2:43 AM, Hari Shreedharan >>> >>> > wrote: >>> >>> >> I am committing those one by one (you will see a few today). Sorry >>> for >>> >>> the delay! >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> Thanks, Hari >>> >>> >> >>> >>> >> On Mon, Apr 6, 2015 at 2:11 PM, Roshan Naik >> > >>> >>> >> wrote: >>> >>> >> >>> >>> >>> Johny, >>> >>> >>> Earlier in this email thread you will find multiple responses >>> to the >>> >>> >>> same question posed by Hari. >>> >>> >>> -roshan >>> >>> >>> On 4/6/15 2:05 PM, "Johny Rufus" wrote: >>> >>> >>>>Folks, >>> >>> >>>> >>> >>> >>>>Am starting the release work for 1.6, can you reply back if there >>> any >>> >>> >>>>pending patches that need to be committed for 1.6. >>> >>> >>>> >>> >>> >>>>Thanks >>> >>> >>>>Rufus >>> >>> >>>> >>> >>> >>>> >>> >>> >>>>On Mon, Apr 6, 2015 at 10:55 AM, Hari Shreedharan >>> >>> >>>>>> >>> >>>>> wrote: >>> >>> >>>> >>> >>> >>>>> I am all in for including it, but it looks like it is Java 7+ - >>> so we >>> >>> >>>>>have >>> >>> >>>>> to wait for Flume 1.7 to actually do it, since 1.6 will be the >>> last >>> >>> >>>>>release >>> >>> >>>>> we will be supporting Java 6. I am ok with doing a 1.7 release >>> right >>> >>> >>>>>after. >>> >>> >>>>> >>> >>> > >>> >>> > >>> >>> > >>> >>> > -- >>> >>> > thanks >>> >>> > ashish >>> >>> > >>> >>> > Blog: http://www.ashishpaliwal.com/blog >>> >>> > My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> thanks >>> >>> ashish >>> >>> >>> >>> Blog: http://www.ashishpaliwal.com/blog >>> >>> My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> >>> >>> >> -- >>> >>李响 >>> >> 手机 cellphone ：+86-1368-113-8972 >>> >> E-mail ：wate...@gmail.com >>> >> MSN ：wate...@hotmail.com >>> -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Commented] (FLUME-2672) NPE in KafkaSourceCounter
[ https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496248#comment-14496248 ] Ashish Paliwal commented on FLUME-2672: --- +1, Patch looks good. May be we should have some test cases around counters as well. > NPE in KafkaSourceCounter > - > > Key: FLUME-2672 > URL: https://issues.apache.org/jira/browse/FLUME-2672 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 > Environment: Mac OS 10.10.3, Java 1.7.0_60 >Reporter: Rigo MacTaggart >Priority: Trivial > Labels: easyfix > Fix For: v1.6.0 > > Attachments: FLUME-2672.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() > because it expects MonitoredCounterGroup.counterMap to contain key > "source.kafka.empty.count". A patch is included which adds this key to the > ATTRIBUTES string array, which is used to pre-populate keys with an initial > value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2672) NPE in KafkaSourceCounter
[ https://issues.apache.org/jira/browse/FLUME-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2672: -- Summary: NPE in KafkaSourceCounter (was: NPE in KafkaSinkCounter) > NPE in KafkaSourceCounter > - > > Key: FLUME-2672 > URL: https://issues.apache.org/jira/browse/FLUME-2672 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 > Environment: Mac OS 10.10.3, Java 1.7.0_60 >Reporter: Rigo MacTaggart >Priority: Trivial > Labels: easyfix > Fix For: v1.6.0 > > Attachments: FLUME-2672.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > A NPE is thrown when KafkaSource calls counter.incrementKafkaEmptyCount() > because it expects MonitoredCounterGroup.counterMap to contain key > "source.kafka.empty.count". A patch is included which adds this key to the > ATTRIBUTES string array, which is used to pre-populate keys with an initial > value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Flume 1.6 jiras
This jira FLUME-2632 - High CPU on KafkaSink, might also be worth adding to 1.6 It's a simple fix. On Tue, Apr 7, 2015 at 7:21 AM, Ashish wrote: > If I am not late for the party, here are some JIRA's needing some attention > > FLUME-2613 - Tool/script for deleting individual message from queue > FLUME-2589 - Add Related Projects Page > FLUME-2413 - Add a new metric for the number of bytes transferred on a > flume channel > > On Tue, Apr 7, 2015 at 2:43 AM, Hari Shreedharan > wrote: >> I am committing those one by one (you will see a few today). Sorry for the >> delay! >> >> >> >> Thanks, Hari >> >> On Mon, Apr 6, 2015 at 2:11 PM, Roshan Naik >> wrote: >> >>> Johny, >>>Earlier in this email thread you will find multiple responses to the >>> same question posed by Hari. >>> -roshan >>> On 4/6/15 2:05 PM, "Johny Rufus" wrote: >>>>Folks, >>>> >>>>Am starting the release work for 1.6, can you reply back if there any >>>>pending patches that need to be committed for 1.6. >>>> >>>>Thanks >>>>Rufus >>>> >>>> >>>>On Mon, Apr 6, 2015 at 10:55 AM, Hari Shreedharan >>>>>>>> wrote: >>>> >>>>> I am all in for including it, but it looks like it is Java 7+ - so we >>>>>have >>>>> to wait for Flume 1.7 to actually do it, since 1.6 will be the last >>>>>release >>>>> we will be supporting Java 6. I am ok with doing a 1.7 release right >>>>>after. >>>>> > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Commented] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14487369#comment-14487369 ] Ashish Paliwal commented on FLUME-2437: --- I was thinking if we can push this as experimental in 1.6 and keep refining it. Can discuss this with Hari and plan accordingly. > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Ashish Paliwal > Attachments: FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2437) S3 Source
[ https://issues.apache.org/jira/browse/FLUME-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486860#comment-14486860 ] Ashish Paliwal commented on FLUME-2437: --- [~jrufus] Did you get time to update the code based on review comments? I think we can try to push it in 1.6 :) > S3 Source > - > > Key: FLUME-2437 > URL: https://issues.apache.org/jira/browse/FLUME-2437 > Project: Flume > Issue Type: New Feature >Reporter: Jonathan Natkins >Assignee: Ashish Paliwal > Attachments: FLUME-2437.patch > > > There have been multiple requests on the mailing list for an S3 source -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Issue : Flume performance
Missed this thread. No direct solution comes to my mind for this Couple of things, since the degradation is occurring only with few instances. 1. Find out what is different on the machines seeing degradation from the others (including OS configurations) 2. Is it the degradation random or occurring on specific hosts Put me in this situation and I would do following 1. Try to look out for a Pattern of degradation, similar hosts, some event storm or could be anything 2. If you have monitoring system, see what's happening around during the degradation, heavy GC activity. Once you get a similar pattern, perhaps debugging and solving it would be easier. On Mon, Apr 6, 2015 at 12:20 AM, ajeet singh wrote: > -- Forwarded message -- > From: ajeet singh > Date: Mon, Apr 6, 2015 at 12:03 AM > Subject: Issue : Flume performance > To: u...@flume.apache.org > > > I am using flume for logging events from aws servers to KAFKA. I am using > logback as flume source, and kafka-sink as sink. In servers we are getting > 3000 events/ per seconds, and there are almost 30 servers running and one > central kafka. At each server flume process is running which will be > writing to central kafka. > I am getting some strange situations like at some servers > flume is able for process the event with rate in which it's getting the > event but on some servers some time it get slow ( like it's getting event > with rate 3k / sec but transferring events to kafka with rate 1Kv per sec). > This way 2K /per sec events are getting added to flume channel, and after > 3-4 hours channel overflow starts. I compared both type of servers where > flume is working fine, and where flume is slow, all parameters ( load, cpu > utilization, memory usages and disk writes) are almost same. What can be > reason for this strange behavior ?? > > Thanks > Ajeet > > > > -- > AJEET PRATAP SINGH -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: I want to be a contributor for apache flume-ng and I have a source here
How is it different from http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source? More details on Getting involved here http://flume.apache.org/getinvolved.html On Thu, Apr 9, 2015 at 6:46 AM, 青年才俊 <719259...@qq.com> wrote: > Hi,dev team > > as the topic of the mail > > I write a source——file monitor source > > it monitors a file and check the modified to get a file stream to record all > the modified content,and it will record the position in disk file,although > the flume application has been killed,OK,it record the position,so it knows > which line should be read next time when it starts. > > I have a test on it ,it can sink all the checked contents without lost. > > thank you! > > here is my github repo: > https://github.com/cwtree/flume-filemonitor-source -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Review Request 32963: Patch for FLUME-2660 - Add documentation for EventValidator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32963/ --- Review request for Flume. Bugs: FLUME-2660 https://issues.apache.org/jira/browse/FLUME-2660 Repository: flume-git Description --- Added documentation for Event Validator implementation Diffs - flume-ng-doc/sphinx/FlumeUserGuide.rst 576cba1 Diff: https://reviews.apache.org/r/32963/diff/ Testing --- NA doc patch Thanks, Ashish Paliwal
[jira] [Updated] (FLUME-2660) Add documentation for EventValidator
[ https://issues.apache.org/jira/browse/FLUME-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2660: -- Attachment: FLUME-2660-0.patch > Add documentation for EventValidator > > > Key: FLUME-2660 > URL: https://issues.apache.org/jira/browse/FLUME-2660 > Project: Flume > Issue Type: Bug >Affects Versions: v1.5.1 >Reporter: Hari Shreedharan >Assignee: Ashish Paliwal > Fix For: v1.6.0 > > Attachments: FLUME-2660-0.patch > > > [~paliwalashish] - Assigning this to you. Please add docs for the > functionality you contributed in FLUME-2613 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Sinks without timeout and ELB Spillover - hangs
Before the thread fades in-memory, here is the JIRA https://issues.apache.org/jira/browse/FLUME-2661 On Thu, Apr 2, 2015 at 6:26 AM, Ashish wrote: > Is it possible to share the method and class name? It would be > somewhere in client package I guess. > I shall do the rest, else would need to re-invent the wheel :) > > On Wed, Apr 1, 2015 at 3:30 AM, Otis Gospodnetic > wrote: >> Hi, >> >> Unfortunately don't have a patch, though I think this is just 1 method that >> needs to be added. Don't think ELB provides the info needed, so one needs >> to just have timeout. >> >> Otis >> -- >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> Solr & Elasticsearch Support * http://sematext.com/ >> >> >> On Tue, Mar 31, 2015 at 4:25 PM, Hari Shreedharan >> wrote: >> >>> I am not very familiar with ELB, but if you do have a patch that will make >>> Flume work better - I’ll be glad to review it. I think the HDFS, HBase, >>> Morphline, Solr, Thrift, and Avro sinks will need it (similar logic, but >>> different impl). >>> >>> >>> >>> >>> Thanks, Hari >>> >>> On Tue, Mar 31, 2015 at 9:27 AM, Ashish wrote: >>> >>> > Does ELB provide any kind of status code when it rejects the request >>> > or it just doesn't send any response? >>> > Perhaps adding a JIRA would be great. Doesn't ELB times out connection >>> > which are Idle for more than 60 sec? >>> > thanks >>> > ashish >>> > On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic >>> > wrote: >>> >> Hi, >>> >> >>> >> I don't have a very good bug report, but I want to share one problem >>> with >>> >> Flume we observed the other day. Specifically, with Elasticsearch Sink >>> and >>> >> AWS ELB, but others may have this problem, too. >>> >> >>> >> In short, ELB has this notion of a Spillover where it can reject >>> requests >>> >> when it can't handle the load. Unfortunately, it doesn't actually >>> signal >>> >> this by closing the connection, so Flume's connection remains open, I >>> >> believe in some java.net... readSocket() method. >>> >> >>> >> To work around this we had to add a socket timeout in our custom >>> version of >>> >> Flume's Elasticsearch Sink. >>> >> >>> >> I think it's worth fixing this in Flume and checking if other Sinks need >>> >> this sort of timeout. >>> >> >>> >> HTH >>> >> >>> >> Otis >>> >> -- >>> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >>> >> Solr & Elasticsearch Support * http://sematext.com/ >>> > -- >>> > thanks >>> > ashish >>> > Blog: http://www.ashishpaliwal.com/blog >>> > My Photo Galleries: http://www.pbase.com/ashishpaliwal >>> > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Created] (FLUME-2661) Support timeout in ES Sink
Ashish Paliwal created FLUME-2661: - Summary: Support timeout in ES Sink Key: FLUME-2661 URL: https://issues.apache.org/jira/browse/FLUME-2661 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.5.1 Reporter: Ashish Paliwal Credits to Otis, From ML Hi, I don't have a very good bug report, but I want to share one problem with Flume we observed the other day. Specifically, with Elasticsearch Sink and AWS ELB, but others may have this problem, too. In short, ELB has this notion of a Spillover where it can reject requests when it can't handle the load. Unfortunately, it doesn't actually signal this by closing the connection, so Flume's connection remains open, I believe in some java.net... readSocket() method. To work around this we had to add a socket timeout in our custom version of Flume's Elasticsearch Sink. I think it's worth fixing this in Flume and checking if other Sinks need this sort of timeout. HTH Otis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2655) Monitoring should start before all other components
[ https://issues.apache.org/jira/browse/FLUME-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14482603#comment-14482603 ] Ashish Paliwal commented on FLUME-2655: --- IMHO, this is the correct behaviour, unless I am missing something. Agent is considered started only when all the components have started. Assume if we start the monitoring system before and the monitoring system adds the Agent to processing pool, like, to an ELB for processing. There is a related JIRA for exposing Agent state, may be that is better way of handling the scenario. > Monitoring should start before all other components > --- > > Key: FLUME-2655 > URL: https://issues.apache.org/jira/browse/FLUME-2655 > Project: Flume > Issue Type: Bug >Reporter: Hari Shreedharan >Assignee: Hari Shreedharan > > Some monitoring systems look at the web server to figure out if the flume > agent has started or not. But if the channel takes a while to start, the > tools think the flume agent have not started yet. So we must start the > monitoring server before the components are started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Flume 1.6 jiras
If I am not late for the party, here are some JIRA's needing some attention FLUME-2613 - Tool/script for deleting individual message from queue FLUME-2589 - Add Related Projects Page FLUME-2413 - Add a new metric for the number of bytes transferred on a flume channel On Tue, Apr 7, 2015 at 2:43 AM, Hari Shreedharan wrote: > I am committing those one by one (you will see a few today). Sorry for the > delay! > > > > Thanks, Hari > > On Mon, Apr 6, 2015 at 2:11 PM, Roshan Naik > wrote: > >> Johny, >>Earlier in this email thread you will find multiple responses to the >> same question posed by Hari. >> -roshan >> On 4/6/15 2:05 PM, "Johny Rufus" wrote: >>>Folks, >>> >>>Am starting the release work for 1.6, can you reply back if there any >>>pending patches that need to be committed for 1.6. >>> >>>Thanks >>>Rufus >>> >>> >>>On Mon, Apr 6, 2015 at 10:55 AM, Hari Shreedharan >>>>>> wrote: >>> >>>> I am all in for including it, but it looks like it is Java 7+ - so we >>>>have >>>> to wait for Flume 1.7 to actually do it, since 1.6 will be the last >>>>release >>>> we will be supporting Java 6. I am ok with doing a 1.7 release right >>>>after. >>>> -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Updated] (FLUME-2635) Documentation for zookeeper dynamic configuration in flume
[ https://issues.apache.org/jira/browse/FLUME-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2635: -- Attachment: FLUME-2635-1.patch Updated as per review comments > Documentation for zookeeper dynamic configuration in flume > -- > > Key: FLUME-2635 > URL: https://issues.apache.org/jira/browse/FLUME-2635 > Project: Flume > Issue Type: Documentation > Components: Configuration >Affects Versions: v1.5.1 >Reporter: Simeon Zaharici >Assignee: Ashish Paliwal >Priority: Minor > Attachments: FLUME-2635-0.patch, FLUME-2635-1.patch > > > Currently the new feature that enables managing the flume configuration via > zookeeper introduced by https://issues.apache.org/jira/browse/FLUME-1491 is > not documented. > It would be very helpful to have this feature documented in the Flume user > guide. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Sinks without timeout and ELB Spillover - hangs
Is it possible to share the method and class name? It would be somewhere in client package I guess. I shall do the rest, else would need to re-invent the wheel :) On Wed, Apr 1, 2015 at 3:30 AM, Otis Gospodnetic wrote: > Hi, > > Unfortunately don't have a patch, though I think this is just 1 method that > needs to be added. Don't think ELB provides the info needed, so one needs > to just have timeout. > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Tue, Mar 31, 2015 at 4:25 PM, Hari Shreedharan > wrote: > >> I am not very familiar with ELB, but if you do have a patch that will make >> Flume work better - I’ll be glad to review it. I think the HDFS, HBase, >> Morphline, Solr, Thrift, and Avro sinks will need it (similar logic, but >> different impl). >> >> >> >> >> Thanks, Hari >> >> On Tue, Mar 31, 2015 at 9:27 AM, Ashish wrote: >> >> > Does ELB provide any kind of status code when it rejects the request >> > or it just doesn't send any response? >> > Perhaps adding a JIRA would be great. Doesn't ELB times out connection >> > which are Idle for more than 60 sec? >> > thanks >> > ashish >> > On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic >> > wrote: >> >> Hi, >> >> >> >> I don't have a very good bug report, but I want to share one problem >> with >> >> Flume we observed the other day. Specifically, with Elasticsearch Sink >> and >> >> AWS ELB, but others may have this problem, too. >> >> >> >> In short, ELB has this notion of a Spillover where it can reject >> requests >> >> when it can't handle the load. Unfortunately, it doesn't actually >> signal >> >> this by closing the connection, so Flume's connection remains open, I >> >> believe in some java.net... readSocket() method. >> >> >> >> To work around this we had to add a socket timeout in our custom >> version of >> >> Flume's Elasticsearch Sink. >> >> >> >> I think it's worth fixing this in Flume and checking if other Sinks need >> >> this sort of timeout. >> >> >> >> HTH >> >> >> >> Otis >> >> -- >> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> >> Solr & Elasticsearch Support * http://sematext.com/ >> > -- >> > thanks >> > ashish >> > Blog: http://www.ashishpaliwal.com/blog >> > My Photo Galleries: http://www.pbase.com/ashishpaliwal >> -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Sinks without timeout and ELB Spillover - hangs
Does ELB provide any kind of status code when it rejects the request or it just doesn't send any response? Perhaps adding a JIRA would be great. Doesn't ELB times out connection which are Idle for more than 60 sec? thanks ashish On Tue, Mar 31, 2015 at 7:50 PM, Otis Gospodnetic wrote: > Hi, > > I don't have a very good bug report, but I want to share one problem with > Flume we observed the other day. Specifically, with Elasticsearch Sink and > AWS ELB, but others may have this problem, too. > > In short, ELB has this notion of a Spillover where it can reject requests > when it can't handle the load. Unfortunately, it doesn't actually signal > this by closing the connection, so Flume's connection remains open, I > believe in some java.net... readSocket() method. > > To work around this we had to add a socket timeout in our custom version of > Flume's Elasticsearch Sink. > > I think it's worth fixing this in Flume and checking if other Sinks need > this sort of timeout. > > HTH > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Flume 1.6 jiras
I can work on this, but not before 3rd April. We also have dis-advantage of being remote :( But open to suggestions. Regarding S3 Source, it would need a lot more love in review + rework and as I see it won't be complete in given timelines unless we put in a very dedicated effort on it. Once I am back, I can fork Johny repo and send PR's with changes. There are lot of JIRA's with patch available status, let's scan them and see if we can push in the changes. thanks ashish On Tue, Mar 31, 2015 at 5:52 AM, Hari Shreedharan wrote: > Flume devs, > > > > > I am going to start working on the release tomorrow, as Johny mentioned that > he is sick and is out for a week or so. I don’t want to delay this release > any further. > > > > > If anyone else can start the work in the next day or so, please let me know - > I will be happy to help. Else, I will do it myself. > > > > > Any more jiras other than the S3 source that needs love? > > > > > Thanks, Hari > > On Thu, Mar 19, 2015 at 12:01 PM, Ashish wrote: > >> Not to push on the release. Please take your time. >> ML was so silent, it was killing me :) >> On Thu, Mar 19, 2015 at 11:51 PM, Johny Rufus wrote: >>> Hari/Ashish, >>> >>> We can begin the release work starting from Monday. >>> >>> Thanks, >>> Rufus >>> >>> On Thu, Mar 19, 2015 at 11:16 AM, Hari Shreedharan < >>> hshreedha...@cloudera.com> wrote: >>> >>>> Johny - any update? >>>> >>>> >>>> Ashish wrote: >>>> >>>>> >>>>> Folks, >>>>> >>>>> what's the plan for the release? Just wanted to checking before >>>>> getting back on hunting more JIRA's :) >>>>> >>>>> thanks >>>>> ashish >>>>> >>>>> On Tue, Feb 24, 2015 at 1:29 AM, Johny Rufus wrote: >>>>> >>>>>> >>>>>> Hi Hari, >>>>>> >>>>>> Yes, definitely, and thanks for the opportunity to get involved in the >>>>>> release process. >>>>>> >>>>>> Thanks, >>>>>> Rufus >>>>>> >>>>>> On Mon, Feb 23, 2015 at 11:39 AM, Hari Shreedharan< >>>>>> hshreedha...@cloudera.com> wrote: >>>>>> >>>>>> >>>>>>> Hey folks, >>>>>>> >>>>>>> >>>>>>> I am planning to merge as many patches into Flume 1.6 as possible in the >>>>>>> next few days so we can start working on a release. >>>>>>> >>>>>>> >>>>>>> Johny - You mentioned that you were interested in doing the release >>>>>>> management work for this. Are you still interested? It would take quite >>>>>>> a >>>>>>> bit of time and effort for this, especially considering the fact that a >>>>>>> lot >>>>>>> of new integration work has been done, pulling in a lot of a new >>>>>>> dependencies. >>>>>>> >>>>>>> >>>>>>> Due credit to the Sqoop project, where the idea of non-committers being >>>>>>> RMs is pretty regular (and from where I copied the idea). >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, Hari >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >> -- >> thanks >> ashish >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Flume 1.6 jiras
Hi Otis, S3 Source is on my priority list. I shall spend some time with it as soon as I am back from vacation. thanks ashish On Mon, Mar 30, 2015 at 9:32 AM, Otis Gospodnetic wrote: > Hi Ashish, > > I think there is that S3 Source JIRA that's waiting for review love in > ReviewBoard that would be good to have in 1.6. > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Thu, Mar 19, 2015 at 1:15 PM, Ashish wrote: > >> Folks, >> >> what's the plan for the release? Just wanted to checking before >> getting back on hunting more JIRA's :) >> >> thanks >> ashish >> >> On Tue, Feb 24, 2015 at 1:29 AM, Johny Rufus wrote: >> > Hi Hari, >> > >> > Yes, definitely, and thanks for the opportunity to get involved in the >> > release process. >> > >> > Thanks, >> > Rufus >> > >> > On Mon, Feb 23, 2015 at 11:39 AM, Hari Shreedharan < >> > hshreedha...@cloudera.com> wrote: >> > >> >> Hey folks, >> >> >> >> >> >> I am planning to merge as many patches into Flume 1.6 as possible in the >> >> next few days so we can start working on a release. >> >> >> >> >> >> Johny - You mentioned that you were interested in doing the release >> >> management work for this. Are you still interested? It would take quite >> a >> >> bit of time and effort for this, especially considering the fact that a >> lot >> >> of new integration work has been done, pulling in a lot of a new >> >> dependencies. >> >> >> >> >> >> Due credit to the Sqoop project, where the idea of non-committers being >> >> RMs is pretty regular (and from where I copied the idea). >> >> >> >> >> >> >> >> Thanks, Hari >> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Commented] (FLUME-2635) Documentation for zookeeper dynamic configuration in flume
[ https://issues.apache.org/jira/browse/FLUME-2635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377606#comment-14377606 ] Ashish Paliwal commented on FLUME-2635: --- [~roshan_naik] Welcome back :) Thanks for feedback. On vacation for next few days, shall see if I can send a patch before that, else it would be definitely in by 1st week of April. > Documentation for zookeeper dynamic configuration in flume > -- > > Key: FLUME-2635 > URL: https://issues.apache.org/jira/browse/FLUME-2635 > Project: Flume > Issue Type: Documentation > Components: Configuration >Affects Versions: v1.5.1 >Reporter: Simeon Zaharici >Assignee: Ashish Paliwal >Priority: Minor > Attachments: FLUME-2635-0.patch > > > Currently the new feature that enables managing the flume configuration via > zookeeper introduced by https://issues.apache.org/jira/browse/FLUME-1491 is > not documented. > It would be very helpful to have this feature documented in the Flume user > guide. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Flume 1.6 jiras
Not to push on the release. Please take your time. ML was so silent, it was killing me :) On Thu, Mar 19, 2015 at 11:51 PM, Johny Rufus wrote: > Hari/Ashish, > > We can begin the release work starting from Monday. > > Thanks, > Rufus > > On Thu, Mar 19, 2015 at 11:16 AM, Hari Shreedharan < > hshreedha...@cloudera.com> wrote: > >> Johny - any update? >> >> >> Ashish wrote: >> >>> >>> Folks, >>> >>> what's the plan for the release? Just wanted to checking before >>> getting back on hunting more JIRA's :) >>> >>> thanks >>> ashish >>> >>> On Tue, Feb 24, 2015 at 1:29 AM, Johny Rufus wrote: >>> >>>> >>>> Hi Hari, >>>> >>>> Yes, definitely, and thanks for the opportunity to get involved in the >>>> release process. >>>> >>>> Thanks, >>>> Rufus >>>> >>>> On Mon, Feb 23, 2015 at 11:39 AM, Hari Shreedharan< >>>> hshreedha...@cloudera.com> wrote: >>>> >>>> >>>>> Hey folks, >>>>> >>>>> >>>>> I am planning to merge as many patches into Flume 1.6 as possible in the >>>>> next few days so we can start working on a release. >>>>> >>>>> >>>>> Johny - You mentioned that you were interested in doing the release >>>>> management work for this. Are you still interested? It would take quite >>>>> a >>>>> bit of time and effort for this, especially considering the fact that a >>>>> lot >>>>> of new integration work has been done, pulling in a lot of a new >>>>> dependencies. >>>>> >>>>> >>>>> Due credit to the Sqoop project, where the idea of non-committers being >>>>> RMs is pretty regular (and from where I copied the idea). >>>>> >>>>> >>>>> >>>>> Thanks, Hari >>>>> >>>> >>>> >>> >>> >>> -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Flume 1.6 jiras
Folks, what's the plan for the release? Just wanted to checking before getting back on hunting more JIRA's :) thanks ashish On Tue, Feb 24, 2015 at 1:29 AM, Johny Rufus wrote: > Hi Hari, > > Yes, definitely, and thanks for the opportunity to get involved in the > release process. > > Thanks, > Rufus > > On Mon, Feb 23, 2015 at 11:39 AM, Hari Shreedharan < > hshreedha...@cloudera.com> wrote: > >> Hey folks, >> >> >> I am planning to merge as many patches into Flume 1.6 as possible in the >> next few days so we can start working on a release. >> >> >> Johny - You mentioned that you were interested in doing the release >> management work for this. Are you still interested? It would take quite a >> bit of time and effort for this, especially considering the fact that a lot >> of new integration work has been done, pulling in a lot of a new >> dependencies. >> >> >> Due credit to the Sqoop project, where the idea of non-committers being >> RMs is pretty regular (and from where I copied the idea). >> >> >> >> Thanks, Hari -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
[jira] [Updated] (FLUME-2613) Tool/script for deleting individual message from queue
[ https://issues.apache.org/jira/browse/FLUME-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2613: -- Attachment: FLUME-2613-3.patch > Tool/script for deleting individual message from queue > -- > > Key: FLUME-2613 > URL: https://issues.apache.org/jira/browse/FLUME-2613 > Project: Flume > Issue Type: Improvement >Reporter: Charles McLaughlin > Assignee: Ashish Paliwal >Priority: Minor > Attachments: FLUME-2613-0.patch, FLUME-2613-1.patch, > FLUME-2613-2.patch, FLUME-2613-3.patch > > > We had a situation where one of our Flume agents got stuck on a message due > to unexpected format. To get things moving again, I stopped the Flume agent, > moved the file backed channel data out of the way and re-started the Flume > agent. I'd like to pop the bad message from the queue data on disk and > ideally there would be a recommended tool/script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2613) Tool/script for deleting individual message from queue
[ https://issues.apache.org/jira/browse/FLUME-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363305#comment-14363305 ] Ashish Paliwal commented on FLUME-2613: --- Handling for command line parameters added. Parameter for Builder can be passed using -D option and they are passed on EventValidator Builder. > Tool/script for deleting individual message from queue > -- > > Key: FLUME-2613 > URL: https://issues.apache.org/jira/browse/FLUME-2613 > Project: Flume > Issue Type: Improvement >Reporter: Charles McLaughlin >Assignee: Ashish Paliwal >Priority: Minor > Attachments: FLUME-2613-0.patch, FLUME-2613-1.patch, > FLUME-2613-2.patch, FLUME-2613-3.patch > > > We had a situation where one of our Flume agents got stuck on a message due > to unexpected format. To get things moving again, I stopped the Flume agent, > moved the file backed channel data out of the way and re-started the Flume > agent. I'd like to pop the bad message from the queue data on disk and > ideally there would be a recommended tool/script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31932: Fix for FLUME-2613 Tool/script for deleting individual message from queue
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31932/ --- (Updated March 16, 2015, 3 p.m.) Review request for Flume. Changes --- Added code to accept command line options for EventValidator Bugs: FLUME-2613 https://issues.apache.org/jira/browse/FLUME-2613 Repository: flume-git Description --- Fix for Tool/script for deleting individual message from queue New tool added to validate events and drop onces that do not meet App criteria Diffs (updated) - flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/EventUtils.java PRE-CREATION flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestEventUtils.java PRE-CREATION flume-tools/src/main/java/org/apache/flume/tools/EventValidater.java PRE-CREATION flume-tools/src/main/java/org/apache/flume/tools/FileChannelIntegrityTool.java 1030442 flume-tools/src/test/java/org/apache/flume/tools/TestFileChannelIntegrityTool.java f24ae56 Diff: https://reviews.apache.org/r/31932/diff/ Testing --- Test Cases Added Thanks, Ashish Paliwal
[jira] [Commented] (FLUME-2613) Tool/script for deleting individual message from queue
[ https://issues.apache.org/jira/browse/FLUME-2613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14362702#comment-14362702 ] Ashish Paliwal commented on FLUME-2613: --- Writing in customized format was in context of a JIRA where it was needed to extract log lines from event, writing back in channel is tricky. It's to get the data and write the Event payload like log line to a file. Command line options was the way I wanted to go, but it was making things a bit messy so let go of it. Shall work on it. > Tool/script for deleting individual message from queue > -- > > Key: FLUME-2613 > URL: https://issues.apache.org/jira/browse/FLUME-2613 > Project: Flume > Issue Type: Improvement >Reporter: Charles McLaughlin >Assignee: Ashish Paliwal >Priority: Minor > Attachments: FLUME-2613-0.patch, FLUME-2613-1.patch, > FLUME-2613-2.patch > > > We had a situation where one of our Flume agents got stuck on a message due > to unexpected format. To get things moving again, I stopped the Flume agent, > moved the file backed channel data out of the way and re-started the Flume > agent. I'd like to pop the bad message from the queue data on disk and > ideally there would be a recommended tool/script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2632) High CPU on KafkaSink
[ https://issues.apache.org/jira/browse/FLUME-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Paliwal updated FLUME-2632: -- Component/s: Sinks+Sources > High CPU on KafkaSink > - > > Key: FLUME-2632 > URL: https://issues.apache.org/jira/browse/FLUME-2632 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.5.1 >Reporter: Gwen Shapira >Assignee: Gwen Shapira > Labels: kafka > Fix For: 1.6 > > Attachments: FLUME-2632-0.patch > > > Reported here: https://github.com/harishreedharan/flume/issues/1 > "I tried flume-ng-kafka-sink and it worked fine. But I noticed that the cpu > utilization stay at 100% and never dropped down all the time even at the time > the channel is empty. > I looked into the source code and found that "process" function in KafkaSink > always return Status.READY even if no events available in channel. That > causes the SinkRunner keep running achieving event from channel and get > nothing. > Do we need to change to return Status.BACKOFF in "process" function in > KafkaSink when it notices that there is no events processed in current round? > So that the SinkRunner has a chance to take a rest when there is no event in > channel. If this proposal feasible, function "testEmptyChannel" in > TestKafkaSink also need to be changed. " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
