[GitHub] flume pull request #255: Remove outdated reference to 0.9.x and 1.4.x line f...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/255 Remove outdated reference to 0.9.x and 1.4.x line from the User Guide Removed the reference to the "0.9.x line" and two other outdated paragraphs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume userguide-outdated-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/255.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #255 commit efffc0cd854f2ee9ed21db83e3675ccd92100c4d Author: Denes Arvay Date: 2018-12-05T11:33:20Z Remove outdated reference to 0.9.x and 1.4.x line from the User Guide commit fb491dfd6002705f5307e8903a9d40d31f81d743 Author: Denes Arvay Date: 2018-12-05T12:11:37Z Fix "Title overline too short." error in FlumeUserGuide.rst ---
[GitHub] flume pull request #195: FLUME-2957. Remove Guava from our public API
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/195 FLUME-2957. Remove Guava from our public API The only Guava class used in public API is the `ImmutableMap` in the `org.apache.flume.Context`, `org.apache.flume.node.MaterializedConfiguration` and `org.apache.flume.node.SimpleMaterializedConfiguration` classes. This commit replaces these with `java.util.Map`. Note: this is a breaking change as it modifies the return type of a method on a public interface. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume FLUME-2957 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/195.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #195 commit 3232832bbfc18aa1baefa1bcd995f2dea75e857f Author: Denes Arvay Date: 2018-01-26T16:07:44Z FLUME-2957. Remove Guava from our public API The only Guava class used in public API is the ImmutableMap in the org.apache.flume.Context, org.apache.flume.node.MaterializedConfiguration and org.apache.flume.node.SimpleMaterializedConfiguration classes. This commit replaces these with java.util.Map. ---
[GitHub] flume pull request #175: Fix source assembly
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/175 Fix source assembly flume-checkstyle breaks the assembly because its parent is not the flume-parent Removing the moduleSets definition from the src assembly solved the issue. Files are added based on fileSets, the resulting tarball's content equals to the result of the dev-support/generate-source-release.sh in a clean working directory. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume fix-src-assembly Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/175.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #175 commit 03f726bdb52083acd73402876ab799f54b85075c Author: Denes Arvay Date: 2017-09-15T11:33:20Z Fix source assembly flume-checkstyle breaks the assembly because its parent is not the flume-parent Removing the moduleSets definition from the src assembly solved the issue. Files are added based on fileSets, the resulting tarball's content equals to the result of the dev-support/generate-source-release.sh in a clean working directory. ---
[GitHub] flume pull request #174: Change the checkstyle project's parent to flume-par...
Github user adenes closed the pull request at: https://github.com/apache/flume/pull/174 ---
[GitHub] flume pull request #174: Change the checkstyle project's parent to flume-par...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/174 Change the checkstyle project's parent to flume-parent - Change the checkstyle project's parent to flume-parent - Set flume-checkstyle's packaging to "pom" This fixes among others the packaging issue when the modules were missing from the source tarball. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume fix-checkstyle-parent Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/174.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #174 commit 8fd53d7225f693ef05ea9a46a546e5a6aa301723 Author: Denes Arvay Date: 2017-09-14T22:20:33Z Change the checkstyle project's parent to flume-parent - Change the checkstyle project's parent to flume-parent - Set flume-checkstyle's packaging to "pom" This fixes among others the packaging issue when the modules were missing from the source tarball. ---
[GitHub] flume pull request #171: Set the copyright date dynamically in documentation...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/171 Set the copyright date dynamically in documentation footer Display "2009-current year" instead of the hardcoded "2009-2012". You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume fix-copyright-year Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/171.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #171 commit e9ba7c30543cf578673fbbd631f750b778dcf990 Author: Denes Arvay Date: 2017-09-12T15:44:13Z Set the copyright date dynamically in documentation footer Display "2009-current year" instead of the hardcoded "2009-2012". ---
[GitHub] flume pull request #170: Fix NetCat UDP Source table in FlumeUserGuide.rst
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/170 Fix NetCat UDP Source table in FlumeUserGuide.rst Netcat UDP source's table in the user guide was broken, rendering failed with _Malformed table. Text in column margin at line offset 7_ error. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume fix-netcat-udp-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #170 commit 91cd238f91498193bbc92928c4968cd10b266131 Author: Denes Arvay Date: 2017-09-12T12:49:13Z Fix NetCat UDP Source table in FlumeUserGuide.rst ---
[GitHub] flume pull request #168: FLUME-3175. Turn off doclint for javadoc generation...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/168 FLUME-3175. Turn off doclint for javadoc generation. With Java 8's new doclint feature the javadoc generation breaks due to the strict checking so adding -Xdoclint:none to the maven-javadoc-plugin's configuration to turn off the check. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume fix-javadoc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/168.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #168 commit efa17764ca422aabb5de3ffa1999e6e6389bca42 Author: Denes Arvay Date: 2017-09-11T19:58:14Z FLUME-3175. Turn off doclint for javadoc generation. With Java 8's new doclint feature the javadoc generation breaks due to the strict checking so adding -Xdoclint:none to the maven-javadoc-plugin's configuration. ---
[GitHub] flume pull request #151: FLUME-3144. Improve Log4jAppender's performance by ...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/151 FLUME-3144. Improve Log4jAppender's performance by allowing logging collection of messages `Log4jAppender` treats `Collection` messages as a special case making it possible to log collection of events in one Log4j log call. The appender sends these events to the receiving Flume instance as one batch with the `rpcClient.appendBatch()` method. New tests added: - logging of a collection of multiple strings, expect one `appendBatch()` call. - logging one string, expect one `append()` call. - logging of a singleton collection of a string, expect one `append()` call. - logging of an empty collection, expect that neither `append()` nor `appendBatch()` was called. - logging of a `String` and an arbitrary object in one collection with `AvroReflectionEnabled` set to `true` - logging of a `String` and an arbitrary object in one collection with `AvroReflectionEnabled` set to `false` You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume FLUME-3144 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #151 commit ba4a9a9793104be715d451cc06c42ac73a77e00e Author: Denes Arvay Date: 2017-08-11T14:22:04Z FLUME-3144. Improve Log4jAppender's performance by allowing logging collection of messages Log4jAppender treats Collection messages as a special case making it possible to log Collection of events in one Log4j log call. The appender sends these events to the receiving Flume instance as one batch with the rpcClient.appendBatch() method. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #129: FLUME-3085: HDFS Sink can skip flushing some Bucket...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/129 FLUME-3085: HDFS Sink can skip flushing some BucketWriters, might lead to data loss This commit fixes the issue when in `HDFSEventSink.process()` a `BucketWriter.append()` call threw a `BucketClosedException` then the newly created `BucketWriter` wasn't flushed after the processing loop. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume FLUME-3085 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/129.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #129 commit f775a629c40bf8373cf3c0a991ea8738e2989c39 Author: Denes Arvay Date: 2017-04-20T13:58:47Z FLUME-3085: HDFS Sink can skip flushing some BucketWriters, might lead to data loss This commit fixes the issue when in HDFSEventSink.process() a BucketWriter.append() call threw a BucketClosedException then the newly created BucketWriter wasn't flushed after the processing loop. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #127: FLUME-3080. Call DistributedFileSystem.recoverLease...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/127 FLUME-3080. Call DistributedFileSystem.recoverLease() if close() fails to avoid lease leak If the HDFS Sink tries to close a file but it fails (e.g. due to timeout) the last block might not end up in COMPLETE state. In this case block recovery should happen but as the lease is still held by Flume the NameNode will start the recovery process only after the hard limit of 1 hour expires. This change adds an explicit recoverLease() call in case of close failure. For more details see https://issues.apache.org/jira/browse/FLUME-3080 You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume FLUME-3080 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/127.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #127 commit 8cc9082c69ad0aea2e8cfa20e906261a6a417245 Author: Denes Arvay Date: 2017-04-03T15:27:19Z FLUME-3080. call DistributedFileSystem.recoverLease() if close() fails to avoid lease leak If the HDFS Sink tries to close a file but it fails (e.g. due to timeout) the last block might not end up in COMPLETE state. In this case block recovery should happen but as the lease is still held by Flume the NameNode will start the recovery process only after the hard limit of 1 hour expires. This change adds an explicit recoverLease() call in case of close failure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #115: Doc fix: Size of files created by HDFS Sink won't n...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/115 Doc fix: Size of files created by HDFS Sink won't necessarily match the configured rollSize value Updated the description of the HDFS Sink's `hdfs.rollSize` property to reflect the actual behaviour: it keeps track of the size of the incoming events' body thus the size of the created files won't necessarily match the configured value. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume hdfs-rollsize-doc-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/115.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #115 commit 8fc1f538010b54e9bc46b0ea0029789aff0d2552 Author: Denes Arvay Date: 2017-03-08T15:40:46Z Doc fix: Size of files created by HDFS Sink won't necessarily match the configured rollSize value --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #106: FLUME-3049: Remove exception wrapping in UGIExecuto...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/106 FLUME-3049: Remove exception wrapping in UGIExecutor.execute() You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume FLUME-3049 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/106.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #106 commit a4286afc5eb28cb7d89137a53bfe47c365e5a170 Author: Denes Arvay Date: 2017-01-27T10:43:27Z FLUME-3049: Remove exception wrapping in UGIExecutor.execute() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #103: Add BucketWriterOptions to simplify the BucketWrite...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/103 Add BucketWriterOptions to simplify the BucketWriter's constructor The goal if this pull request is to get rid of the 24 parameters of the `BucketWriter`'s constructor by wrapping them to the builder-like `BucketWriterOptions` class. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume bucketwriter-options Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/103.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #103 commit 01c243e605f1a233a55612f3c7b602ac4ea3a076 Author: Denes Arvay Date: 2016-10-24T12:49:03Z Add BucketWriterOptions to simplify BucketWriter's constructor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #85: FLUME-3025: Expose FileChannel.open JMX
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/85 FLUME-3025: Expose FileChannel.open JMX This patch exposes the `FileChannel`'s `open` flag on JMX to make it possible to detect when it wasn't able to start up (thus its `open` flag is `false`). You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume expose-filechannel-open-jmx Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/85.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #85 commit ec250a672523ef36a72b6ebbcdd9c7ecfab39447 Author: Denes Arvay Date: 2016-11-15T16:21:12Z Expose FileChannel.open on JMX commit b6330e7366079f581e6d4d1ade9a97c1067f0bb1 Author: Denes Arvay Date: 2016-11-16T15:44:06Z Add test for FileChannelCounter.isOpen() --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #83: FLUME-2812: Exception in thread "SinkRunner-PollingR...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/83 FLUME-2812: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded Fix semaphore leak causing `java.lang.Error: Maximum permit count exceeded` in `MemoryChannel` ```java Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded at java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192) at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341) at java.util.concurrent.Semaphore.release(Semaphore.java:609) at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147) at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) at org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:745) ``` for more details and explanation see my comment on [FLUME-2812](https://issues.apache.org/jira/browse/FLUME-2812) You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume FLUME-2812 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/83.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #83 commit 538837d5c035461c9b3073a2ce89da375aa139f3 Author: Denes Arvay Date: 2016-10-28T15:38:33Z FLUME-2812: Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded Change-Id: Ib6761364abefca91ca50d80c073637766bfd93ca --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #82: [WIP] Proof of concept for a generic password provid...
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/82 [WIP] Proof of concept for a generic password provider One of the main security concerns regarding Flume is that currently passwords can only be set in plain text in the config file. I have a proof-of-concept to overcome this limitation with an extensible password provider. The core of the solution is the `PasswordProvider` interface which has a default implementation (`PlainTextPasswordProvider`) which returns the value of the given key, thus taking care of backwards compatibility. The other implementation is the `ExternalProcessPasswordProvider` which executes the configured command and returns its output. Usage example can be seen in the `AvroSource` (see the 2nd commit of this PR): ```diff - keystorePassword = context.getString(KEYSTORE_PASSWORD_KEY); + keystorePassword = PasswordConfigurator.getPassword(context, KEYSTORE_PASSWORD_KEY); ``` **Example configuration to use the `ExternalProcessPasswordProvider`:** ``` ... a.sources.avro.keystore-password.passwordProviderClass=org.apache.flume.conf.ExternalProcessPasswordProvider a.sources.avro.keystore-password.command=get_avro_keystore_password.sh ... ``` **Example configuration with no `passwordProviderClass` set:** ``` ... a.sources.avro.keystore-password=SecretPassword ... ``` As no `passwordProviderClass` is set in this example the default `PlainTextPasswordProvider` is used which returns the value of `a.sources.avro.keystore-password`. --- _Note: this is still a work in progress, I wanted to sketch up my idea. Any questions/comments/suggestions are more than welcome._ You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume password-provider Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/82.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #82 commit 85f96a6e7a660e14b2fe1d2aaceb6021395249df Author: Denes Arvay Date: 2016-10-25T12:15:11Z add generic PasswordProvider Change-Id: I28a11ccbeb3bd19f8de14c66d4cf47dc6e1c7969 commit b64a7ddca1b00df303c64a237adbb0a1008cf739 Author: Denes Arvay Date: 2016-10-25T13:01:54Z use PasswordConfigurator.getPassword in AvroSource Change-Id: I0dadcb576c9b9eefb557cb8e111cb899cd415c0e --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flume pull request #67: Add flume-shared module to assembly xmls
GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/67 Add flume-shared module to assembly xmls Fixes the issue that the assembly plugin created the source package without the `flume-shared` project in it so maven compile failed on that. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume flume-1.7 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/67.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #67 commit 45f56414a1f8903cdcf268bc1aa54445f589e033 Author: Denes Arvay Date: 2016-10-11T14:23:51Z Add flume-shared module to assembly xmls Change-Id: I1d1d4d3ad6edff297a26dfa901d41f4143124fdb --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
