[jira] [Created] (FLUME-2544) Windows: Incorrect Path Separator used in HDFS path (HDFS Sink)
Roshan Naik created FLUME-2544: -- Summary: Windows: Incorrect Path Separator used in HDFS path (HDFS Sink) Key: FLUME-2544 URL: https://issues.apache.org/jira/browse/FLUME-2544 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.5.0.1 Reporter: Roshan Naik Assignee: Roshan Naik Need to use / for HDFS paths as separator instead of system specific System.getProperty(file.separator) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2544) Windows: Incorrect Path Separator used in HDFS path (HDFS Sink)
[ https://issues.apache.org/jira/browse/FLUME-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated FLUME-2544: --- Attachment: FLUME-2544.patch Uploading patch Windows: Incorrect Path Separator used in HDFS path (HDFS Sink) --- Key: FLUME-2544 URL: https://issues.apache.org/jira/browse/FLUME-2544 Project: Flume Issue Type: Bug Components: Sinks+Sources Affects Versions: v1.5.0.1 Reporter: Roshan Naik Assignee: Roshan Naik Labels: windows Attachments: FLUME-2544.patch Need to use / for HDFS paths as separator instead of system specific System.getProperty(file.separator) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release Apache Flume 1.5.1 RC1
It does not look like we ever actually included the dev-support directory in the source tarball (I checked 1.3.1,1.4.0 and 1.5.0.1). If we need a re-spin for another reason, I will try to fix the release process to pull this in and remove the iml files. Arvind - does that sound good to you? Otherwise I will spin another RC. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari
Re: [VOTE] Release Apache Flume 1.5.1 RC1
Roshan, The Avro Source does make it configurable - https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java;h=59ee43a8e1b758ca3d98ba572a885ee2f01b7bed;hb=HEAD#l185 But the HTTPSource disables it completely (it is not a configurable option). Should we remove the option from the Avro Source (or add these two to the list of excluded protocols?). I believe it is best to not allow the protocols to be used at all, so it must be included anyway - any additional ones should just be added to these. I think we can add the configurable option for HTTP Source in a later release. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari
[jira] [Commented] (FLUME-2307) Remove Log writetimeout
[
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205212#comment-14205212
]
Hari Shreedharan commented on FLUME-2307:
-
Which version of Flume? OS details and also stacktrace/logs?
Remove Log writetimeout
---
Key: FLUME-2307
URL: https://issues.apache.org/jira/browse/FLUME-2307
Project: Flume
Issue Type: Bug
Components: Channel
Affects Versions: v1.4.0
Reporter: Steve Zesch
Assignee: Hari Shreedharan
Fix For: v1.5.0
Attachments: FLUME-2307-1.patch, FLUME-2307.patch
I've observed Flume failing to clean up old log data in FileChannels. The
amount of old log data can range anywhere from tens to hundreds of GB. I was
able to confirm that the channels were in fact empty. This behavior always
occurs after lock timeouts when attempting to put, take, rollback, or commit
to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old
files. I was able to confirm that the Log's writeCheckpoint method was still
being called and successfully obtaining a lock from tryLockExclusive(), but I
was not able to confirm removeOldLogs being called. The application log did
not include Removing old file: log-xyz for the old files which the Log
class would output if they were correctly being removed. I suspect the lock
timeouts were due to high I/O load at the time.
Some stack traces:
{code}
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
at
dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:619)
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
at
org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at
dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
at org.apache.avro.ipc.Responder.respond(Responder.java:151)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
[jira] [Commented] (FLUME-2307) Remove Log writetimeout
[
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205225#comment-14205225
]
Hari Shreedharan commented on FLUME-2307:
-
Your checkpoint interval is set to 27+ hours (that value is in seconds). Flume
deletes files only on every checkpoint. In fact, once the events in a file are
removed, it will remove the file in the 2nd checkpoint after that one (not the
one immediately after). So in your case, it needs to wait for 54 hours before
deleting the file. Even after a replay it waits for another checkpoint to
delete the file. Did you wait for that long and verify?
Remove Log writetimeout
---
Key: FLUME-2307
URL: https://issues.apache.org/jira/browse/FLUME-2307
Project: Flume
Issue Type: Bug
Components: Channel
Affects Versions: v1.4.0
Reporter: Steve Zesch
Assignee: Hari Shreedharan
Fix For: v1.5.0
Attachments: FLUME-2307-1.patch, FLUME-2307.patch
I've observed Flume failing to clean up old log data in FileChannels. The
amount of old log data can range anywhere from tens to hundreds of GB. I was
able to confirm that the channels were in fact empty. This behavior always
occurs after lock timeouts when attempting to put, take, rollback, or commit
to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old
files. I was able to confirm that the Log's writeCheckpoint method was still
being called and successfully obtaining a lock from tryLockExclusive(), but I
was not able to confirm removeOldLogs being called. The application log did
not include Removing old file: log-xyz for the old files which the Log
class would output if they were correctly being removed. I suspect the lock
timeouts were due to high I/O load at the time.
Some stack traces:
{code}
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
at
dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:619)
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
at
org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at
dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
at org.apache.avro.ipc.Responder.respond(Responder.java:151)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
[jira] [Commented] (FLUME-2307) Remove Log writetimeout
[
https://issues.apache.org/jira/browse/FLUME-2307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205230#comment-14205230
]
Hari Shreedharan commented on FLUME-2307:
-
Sorry - my mistake. That is in millis. But you should still account for the 2
checkpoints.
Remove Log writetimeout
---
Key: FLUME-2307
URL: https://issues.apache.org/jira/browse/FLUME-2307
Project: Flume
Issue Type: Bug
Components: Channel
Affects Versions: v1.4.0
Reporter: Steve Zesch
Assignee: Hari Shreedharan
Fix For: v1.5.0
Attachments: FLUME-2307-1.patch, FLUME-2307.patch
I've observed Flume failing to clean up old log data in FileChannels. The
amount of old log data can range anywhere from tens to hundreds of GB. I was
able to confirm that the channels were in fact empty. This behavior always
occurs after lock timeouts when attempting to put, take, rollback, or commit
to a FileChannel. Once the timeout occurs, Flume stops cleaning up the old
files. I was able to confirm that the Log's writeCheckpoint method was still
being called and successfully obtaining a lock from tryLockExclusive(), but I
was not able to confirm removeOldLogs being called. The application log did
not include Removing old file: log-xyz for the old files which the Log
class would output if they were correctly being removed. I suspect the lock
timeouts were due to high I/O load at the time.
Some stack traces:
{code}
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:478)
at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doCommit(FileChannel.java:594)
at
org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
at
dataxu.flume.plugins.avro.AsyncAvroSink.process(AsyncAvroSink.java:548)
at
dataxu.flume.plugins.ClassLoaderFlumeSink.process(ClassLoaderFlumeSink.java:33)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:619)
org.apache.flume.ChannelException: Failed to obtain lock for writing to the
log. Try increasing the log write timeout value. [channel=fileChannel]
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621)
at
org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194)
at
dataxu.flume.plugins.avro.AvroSource.appendBatch(AvroSource.java:209)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:91)
at org.apache.avro.ipc.Responder.respond(Responder.java:151)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)
at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:321)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:303)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:220)
at
Re: [VOTE] Release Apache Flume 1.5.1 RC1
Looks like most major companies/products are moving away from SSLv2 3 very quickly. I am ok with disabling it completely and allowing user to add more protocols to disable list. Not a security expert not sure how much of a backward compat issue this implies. I am fine with supporting the hard coded ban on the protocols in Avro source with additional ban as per user config. Also fine with adding the same behavior later to a later release. i think its good to keep the same strategy for both Avro and HTTPS. If the intent is to add the configurable option to HTTPS in a later release, then please drop the setting from the doc too. We can track the pending work for HTTPS on another jira. -roshan On Mon, Nov 10, 2014 at 11:15 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: Roshan, The Avro Source does make it configurable - https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java;h=59ee43a8e1b758ca3d98ba572a885ee2f01b7bed;hb=HEAD#l185 But the HTTPSource disables it completely (it is not a configurable option). Should we remove the option from the Avro Source (or add these two to the list of excluded protocols?). I believe it is best to not allow the protocols to be used at all, so it must be included anyway - any additional ones should just be added to these. I think we can add the configurable option for HTTP Source in a later release. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Flume 1.5.1 RC1
That sounds good to me. Thanks for working on this release Hari. Regards, Arvind Prabhakar On Mon, Nov 10, 2014 at 11:10 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: It does not look like we ever actually included the dev-support directory in the source tarball (I checked 1.3.1,1.4.0 and 1.5.0.1). If we need a re-spin for another reason, I will try to fix the release process to pull this in and remove the iml files. Arvind - does that sound good to you? Otherwise I will spin another RC. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari
Re: [VOTE] Release Apache Flume 1.5.1 RC1
The HTTP Source does not have any config parameter in the user guide. Only for the Avro Source, is there configuration mentioned in the user guide (though for now, it is possible to use SSLv3, but we can disable that in a future release - right now we provide the option, so I guess that would be enough). So I guess we are ok? Thanks, Hari On Mon, Nov 10, 2014 at 2:51 PM, Roshan Naik ros...@hortonworks.com wrote: Looks like most major companies/products are moving away from SSLv2 3 very quickly. I am ok with disabling it completely and allowing user to add more protocols to disable list. Not a security expert not sure how much of a backward compat issue this implies. I am fine with supporting the hard coded ban on the protocols in Avro source with additional ban as per user config. Also fine with adding the same behavior later to a later release. i think its good to keep the same strategy for both Avro and HTTPS. If the intent is to add the configurable option to HTTPS in a later release, then please drop the setting from the doc too. We can track the pending work for HTTPS on another jira. -roshan On Mon, Nov 10, 2014 at 11:15 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: Roshan, The Avro Source does make it configurable - https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java;h=59ee43a8e1b758ca3d98ba572a885ee2f01b7bed;hb=HEAD#l185 But the HTTPSource disables it completely (it is not a configurable option). Should we remove the option from the Avro Source (or add these two to the list of excluded protocols?). I believe it is best to not allow the protocols to be used at all, so it must be included anyway - any additional ones should just be added to these. I think we can add the configurable option for HTTP Source in a later release. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Kafka Channel vs Kafka Sink?
Yes, you can. I came up with 2 use-cases where the Kafka channel is useful (in addition to the HA aspect of the channel). 1. Receive data from various sources (even Kafka itself) - and modify it using interceptors and write out to Kafka. This would be lower latency than using a channel + sink - and this could be HA if you have multiple Flume agents receiving the data, so a dead Flume agent would not delay your data. 2. Send data from Kafka to HDFS/HBase at low latency. This again, gives the advantage of dead Flume agents not delaying data delivery. One agent dies, another picks up the slack sending data to HDFS/HBase etc. I think the Storm Spout is really not required to write to HDFS unless you have more complex processing required on the events. Thanks, Hari On Fri, Nov 7, 2014 at 8:20 PM, Ashish paliwalash...@gmail.com wrote: Just wondering, can I use Kafka Channel instead of Kafka Sink? Essentially the flow is like. Things are coming from working on https://issues.apache.org/jira/browse/FLUME-1286) Source - Channel - Kafka Sink - Kafka - kafka-Storm spout To me it seems like we can use an Agent with Kafka Channel and without a Sink. Just trying to find out Pro's and Con's of this. I am not using it, just curious after reviewing the patch for Kafka Channel documentation. One thing that I could think of was not being able to use Multiple Sinks to drain events faster. Comments/Suggestions? thanks ashish
Re: Using CodaHale metrics for monitoring
Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc). Thanks, Hari On Sat, Nov 8, 2014 at 8:21 PM, Ashish paliwalash...@gmail.com wrote: Hi, Have hacked a bit into our existing instrumentation package and piggy backed cohahale metrics package. Here is one sample for Spooled Directory source (with instrumentation only for Source and Channel ), using console reporter -- Gauges -- org.apache.flume.instrumentation.ChannelCounter.channel.current.size value = 200 org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage value = 2.0 org.apache.flume.instrumentation.SourceCounter.src.open-connection.count value = 0 -- Counters org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt count = 1138800 org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success count = 1138800 org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt count = 1138601 org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success count = 1138600 org.apache.flume.instrumentation.SourceCounter.src.events.accepted count = 1138800 org.apache.flume.instrumentation.SourceCounter.src.events.received count = 1138800 src.append-batch.accepted count = 11388 src.append-batch.received count = 11388 src.append.accepted count = 0 src.append.received count = 0 -- Meters -- eventAcceptedMeter count = 1138800 mean rate = 106478.56 events/second 1-minute rate = 93420.18 events/second 5-minute rate = 91389.24 events/second 15-minute rate = 91037.40 events/second eventReceivedMeter count = 1138800 mean rate = 106462.14 events/second 1-minute rate = 93420.18 events/second 5-minute rate = 91389.24 events/second 15-minute rate = 91037.40 events/second If there is interest in the community, can raise a jira and continue to work on it. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [ANNOUNCE] New Flume PMC Member - Roshan Naik
Welcome Roshan, and congrats. Mike On Wed, Nov 5, 2014 at 5:50 PM, Gwen Shapira gshap...@cloudera.com wrote: Congrats, Roshan :) Very much deserved. On Tue, Nov 4, 2014 at 2:12 PM, Arvind Prabhakar arv...@apache.org wrote: On behalf of Apache Flume PMC, it is my pleasure to announce that Roshan Naik has been elected to the Flume Project Management Committee. Roshan has been active with the project for many years and has been a committer on the project since September of 2013. Please join me in congratulating Roshan and welcoming him to the Flume PMC. Regards, Arvind Prabhakar
Re: Kafka Channel vs Kafka Sink?
Thanks! Storm Spout was more to connect Flume to Storm, rather than writing to HDFS. What I meant was, may be don't need Storm Sink anymore. On Tue, Nov 11, 2014 at 4:45 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: Yes, you can. I came up with 2 use-cases where the Kafka channel is useful (in addition to the HA aspect of the channel). 1. Receive data from various sources (even Kafka itself) - and modify it using interceptors and write out to Kafka. This would be lower latency than using a channel + sink - and this could be HA if you have multiple Flume agents receiving the data, so a dead Flume agent would not delay your data. 2. Send data from Kafka to HDFS/HBase at low latency. This again, gives the advantage of dead Flume agents not delaying data delivery. One agent dies, another picks up the slack sending data to HDFS/HBase etc. I think the Storm Spout is really not required to write to HDFS unless you have more complex processing required on the events. Thanks, Hari On Fri, Nov 7, 2014 at 8:20 PM, Ashish paliwalash...@gmail.com wrote: Just wondering, can I use Kafka Channel instead of Kafka Sink? Essentially the flow is like. Things are coming from working on https://issues.apache.org/jira/browse/FLUME-1286) Source - Channel - Kafka Sink - Kafka - kafka-Storm spout To me it seems like we can use an Agent with Kafka Channel and without a Sink. Just trying to find out Pro's and Con's of this. I am not using it, just curious after reviewing the patch for Kafka Channel documentation. One thing that I could think of was not being able to use Multiple Sinks to drain events faster. Comments/Suggestions? thanks ashish -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Codahale does have lot more features like the rate/sec one, which is most needed metrics. It provides a lot of reporters out of box (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to enable it, rather than writing custom components. As of now we have to hide it within MonitoredCounterGroup and its sub-classes, that's what I did for running it. It was running parallel to existing system. Let me work more on it and see if I can simplify the same. On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc). Thanks, Hari On Sat, Nov 8, 2014 at 8:21 PM, Ashish paliwalash...@gmail.com wrote: Hi, Have hacked a bit into our existing instrumentation package and piggy backed cohahale metrics package. Here is one sample for Spooled Directory source (with instrumentation only for Source and Channel ), using console reporter -- Gauges -- org.apache.flume.instrumentation.ChannelCounter.channel.current.size value = 200 org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage value = 2.0 org.apache.flume.instrumentation.SourceCounter.src.open-connection.count value = 0 -- Counters org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt count = 1138800 org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success count = 1138800 org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt count = 1138601 org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success count = 1138600 org.apache.flume.instrumentation.SourceCounter.src.events.accepted count = 1138800 org.apache.flume.instrumentation.SourceCounter.src.events.received count = 1138800 src.append-batch.accepted count = 11388 src.append-batch.received count = 11388 src.append.accepted count = 0 src.append.received count = 0 -- Meters -- eventAcceptedMeter count = 1138800 mean rate = 106478.56 events/second 1-minute rate = 93420.18 events/second 5-minute rate = 91389.24 events/second 15-minute rate = 91037.40 events/second eventReceivedMeter count = 1138800 mean rate = 106462.14 events/second 1-minute rate = 93420.18 events/second 5-minute rate = 91389.24 events/second 15-minute rate = 91037.40 events/second If there is interest in the community, can raise a jira and continue to work on it. -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: [VOTE] Release Apache Flume 1.5.1 RC1
Wouldnt we want to document in the http source section that SSL is being blocked ? -roshan On Mon, Nov 10, 2014 at 2:56 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: The HTTP Source does not have any config parameter in the user guide. Only for the Avro Source, is there configuration mentioned in the user guide (though for now, it is possible to use SSLv3, but we can disable that in a future release - right now we provide the option, so I guess that would be enough). So I guess we are ok? Thanks, Hari On Mon, Nov 10, 2014 at 2:51 PM, Roshan Naik ros...@hortonworks.com wrote: Looks like most major companies/products are moving away from SSLv2 3 very quickly. I am ok with disabling it completely and allowing user to add more protocols to disable list. Not a security expert not sure how much of a backward compat issue this implies. I am fine with supporting the hard coded ban on the protocols in Avro source with additional ban as per user config. Also fine with adding the same behavior later to a later release. i think its good to keep the same strategy for both Avro and HTTPS. If the intent is to add the configurable option to HTTPS in a later release, then please drop the setting from the doc too. We can track the pending work for HTTPS on another jira. -roshan On Mon, Nov 10, 2014 at 11:15 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: Roshan, The Avro Source does make it configurable - https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java;h=59ee43a8e1b758ca3d98ba572a885ee2f01b7bed;hb=HEAD#l185 But the HTTPSource disables it completely (it is not a configurable option). Should we remove the option from the Avro Source (or add these two to the list of excluded protocols?). I believe it is best to not allow the protocols to be used at all, so it must be included anyway - any additional ones should just be added to these. I think we can add the configurable option for HTTP Source in a later release. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure
[jira] [Commented] (FLUME-2497) TCP and UDP syslog sources parsing the timestamp incorrectly
[ https://issues.apache.org/jira/browse/FLUME-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205834#comment-14205834 ] Mike Percy commented on FLUME-2497: --- +1 lgtm. I have a couple minor nits (add a couple comments, formatting) that I'll just address on commit. TCP and UDP syslog sources parsing the timestamp incorrectly Key: FLUME-2497 URL: https://issues.apache.org/jira/browse/FLUME-2497 Project: Flume Issue Type: Bug Affects Versions: v1.5.0.1 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: v1.6.0 Attachments: FLUME-2497.patch TCP and UDP syslog sources parse the timestamp incorrectly while using Syslogutils extractEvent and buildEvent methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2497) TCP and UDP syslog sources parsing the timestamp incorrectly
[ https://issues.apache.org/jira/browse/FLUME-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205864#comment-14205864 ] ASF subversion and git services commented on FLUME-2497: Commit 534fe268d1d0ad197f6e4a867ab0ae0155d1a927 in flume's branch refs/heads/trunk from [~mpercy] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=534fe26 ] FLUME-2497. Support fractional seconds in Syslog timestamps This fixes a bug in the SyslogTcpSource and SyslogUdpSource where fractional timestamps fail to parse. (Johny Rufus via Mike Percy) TCP and UDP syslog sources parsing the timestamp incorrectly Key: FLUME-2497 URL: https://issues.apache.org/jira/browse/FLUME-2497 Project: Flume Issue Type: Bug Affects Versions: v1.5.0.1 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: v1.6.0 Attachments: FLUME-2497.patch TCP and UDP syslog sources parse the timestamp incorrectly while using Syslogutils extractEvent and buildEvent methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2497) TCP and UDP syslog sources parsing the timestamp incorrectly
[ https://issues.apache.org/jira/browse/FLUME-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205865#comment-14205865 ] ASF subversion and git services commented on FLUME-2497: Commit dcc1ce066853a9594508f21a6bd0d7382a4c240e in flume's branch refs/heads/flume-1.6 from [~mpercy] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=dcc1ce0 ] FLUME-2497. Support fractional seconds in Syslog timestamps This fixes a bug in the SyslogTcpSource and SyslogUdpSource where fractional timestamps fail to parse. (Johny Rufus via Mike Percy) TCP and UDP syslog sources parsing the timestamp incorrectly Key: FLUME-2497 URL: https://issues.apache.org/jira/browse/FLUME-2497 Project: Flume Issue Type: Bug Affects Versions: v1.5.0.1 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: v1.6.0 Attachments: FLUME-2497.patch TCP and UDP syslog sources parse the timestamp incorrectly while using Syslogutils extractEvent and buildEvent methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2497) TCP and UDP syslog sources parsing the timestamp incorrectly
[ https://issues.apache.org/jira/browse/FLUME-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205890#comment-14205890 ] Hudson commented on FLUME-2497: --- SUCCESS: Integrated in flume-trunk #691 (See [https://builds.apache.org/job/flume-trunk/691/]) FLUME-2497. Support fractional seconds in Syslog timestamps (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=534fe268d1d0ad197f6e4a867ab0ae0155d1a927) * flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java * flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java TCP and UDP syslog sources parsing the timestamp incorrectly Key: FLUME-2497 URL: https://issues.apache.org/jira/browse/FLUME-2497 Project: Flume Issue Type: Bug Affects Versions: v1.5.0.1 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: v1.6.0 Attachments: FLUME-2497.patch TCP and UDP syslog sources parse the timestamp incorrectly while using Syslogutils extractEvent and buildEvent methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2385) Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level
[ https://issues.apache.org/jira/browse/FLUME-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205891#comment-14205891 ] Ye Xianjin commented on FLUME-2385: --- hi, [~scaph01], I think (according to my colleague) the more reasonable change is to set the log level to debug. Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level Key: FLUME-2385 URL: https://issues.apache.org/jira/browse/FLUME-2385 Project: Flume Issue Type: Improvement Affects Versions: v1.4.0 Reporter: Justin Hayes Assignee: Phil Scala Priority: Minor Fix For: v1.6.0 Attachments: FLUME-2385-0.patch When I start an agent with the following config, the spooling directory source emits 14/05/14 22:36:12 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. messages twice a second. Pretty innocuous but it will fill up the file system needlessly and get in the way of other INFO messages. cis.sources = httpd cis.sinks = loggerSink cis.channels = mem2logger cis.sources.httpd.type = spooldir cis.sources.httpd.spoolDir = /var/log/httpd cis.sources.httpd.trackerDir = /var/lib/flume-ng/tracker/httpd cis.sources.httpd.channels = mem2logger cis.sinks.loggerSink.type = logger cis.sinks.loggerSink.channel = mem2logger cis.channels.mem2logger.type = memory cis.channels.mem2logger.capacity = 1 cis.channels.mem2logger.transactionCapacity = 1000 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2497) TCP and UDP syslog sources parsing the timestamp incorrectly
[ https://issues.apache.org/jira/browse/FLUME-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205898#comment-14205898 ] Hudson commented on FLUME-2497: --- SUCCESS: Integrated in Flume-trunk-hbase-98 #48 (See [https://builds.apache.org/job/Flume-trunk-hbase-98/48/]) FLUME-2497. Support fractional seconds in Syslog timestamps (mpercy: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=534fe268d1d0ad197f6e4a867ab0ae0155d1a927) * flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUtils.java * flume-ng-core/src/main/java/org/apache/flume/source/SyslogUtils.java TCP and UDP syslog sources parsing the timestamp incorrectly Key: FLUME-2497 URL: https://issues.apache.org/jira/browse/FLUME-2497 Project: Flume Issue Type: Bug Affects Versions: v1.5.0.1 Reporter: Johny Rufus Assignee: Johny Rufus Fix For: v1.6.0 Attachments: FLUME-2497.patch TCP and UDP syslog sources parse the timestamp incorrectly while using Syslogutils extractEvent and buildEvent methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release Apache Flume 1.5.1 RC1
I will update the documentation before I post it to the website — that should be good. Thanks, Hari On Mon, Nov 10, 2014 at 6:18 PM, Roshan Naik ros...@hortonworks.com wrote: Wouldnt we want to document in the http source section that SSL is being blocked ? -roshan On Mon, Nov 10, 2014 at 2:56 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: The HTTP Source does not have any config parameter in the user guide. Only for the Avro Source, is there configuration mentioned in the user guide (though for now, it is possible to use SSLv3, but we can disable that in a future release - right now we provide the option, so I guess that would be enough). So I guess we are ok? Thanks, Hari On Mon, Nov 10, 2014 at 2:51 PM, Roshan Naik ros...@hortonworks.com wrote: Looks like most major companies/products are moving away from SSLv2 3 very quickly. I am ok with disabling it completely and allowing user to add more protocols to disable list. Not a security expert not sure how much of a backward compat issue this implies. I am fine with supporting the hard coded ban on the protocols in Avro source with additional ban as per user config. Also fine with adding the same behavior later to a later release. i think its good to keep the same strategy for both Avro and HTTPS. If the intent is to add the configurable option to HTTPS in a later release, then please drop the setting from the doc too. We can track the pending work for HTTPS on another jira. -roshan On Mon, Nov 10, 2014 at 11:15 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: Roshan, The Avro Source does make it configurable - https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java;h=59ee43a8e1b758ca3d98ba572a885ee2f01b7bed;hb=HEAD#l185 But the HTTPSource disables it completely (it is not a configurable option). Should we remove the option from the Avro Source (or add these two to the list of excluded protocols?). I believe it is best to not allow the protocols to be used at all, so it must be included anyway - any additional ones should just be added to these. I think we can add the configurable option for HTTP Source in a later release. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE
Re: [VOTE] Release Apache Flume 1.5.1 RC1
My vote: +1. * Verified checksums * Verified signatures * Checked NOTICE, CHANGELOG, README, RELEASE-NOTES etc. Thanks, Hari On Mon, Nov 10, 2014 at 6:18 PM, Roshan Naik ros...@hortonworks.com wrote: Wouldnt we want to document in the http source section that SSL is being blocked ? -roshan On Mon, Nov 10, 2014 at 2:56 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: The HTTP Source does not have any config parameter in the user guide. Only for the Avro Source, is there configuration mentioned in the user guide (though for now, it is possible to use SSLv3, but we can disable that in a future release - right now we provide the option, so I guess that would be enough). So I guess we are ok? Thanks, Hari On Mon, Nov 10, 2014 at 2:51 PM, Roshan Naik ros...@hortonworks.com wrote: Looks like most major companies/products are moving away from SSLv2 3 very quickly. I am ok with disabling it completely and allowing user to add more protocols to disable list. Not a security expert not sure how much of a backward compat issue this implies. I am fine with supporting the hard coded ban on the protocols in Avro source with additional ban as per user config. Also fine with adding the same behavior later to a later release. i think its good to keep the same strategy for both Avro and HTTPS. If the intent is to add the configurable option to HTTPS in a later release, then please drop the setting from the doc too. We can track the pending work for HTTPS on another jira. -roshan On Mon, Nov 10, 2014 at 11:15 AM, Hari Shreedharan hshreedha...@cloudera.com wrote: Roshan, The Avro Source does make it configurable - https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob;f=flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java;h=59ee43a8e1b758ca3d98ba572a885ee2f01b7bed;hb=HEAD#l185 But the HTTPSource disables it completely (it is not a configurable option). Should we remove the option from the Avro Source (or add these two to the list of excluded protocols?). I believe it is best to not allow the protocols to be used at all, so it must be included anyway - any additional ones should just be added to these. I think we can add the configurable option for HTTP Source in a later release. Thanks, Hari On Sun, Nov 9, 2014 at 8:47 PM, Arvind Prabhakar arv...@apache.org wrote: +1 * Verified signatures * Verified checksums * Verified the tag (minor issues noted below - would be good to address if there is RC2) * Builds correctly * All tests run with default profile and avro version set to 1.7.5 (to avoid an issue with snappy on Mac OS) Nits: * The tag and sources match except that the src tarball contains the iml files and does not contain the dev-support directory. Since both the iml files and dev-support files are not related to product functionality, it is OK for the tarball to not include them. However, if there is a respin it would be good to address that. * It is time we updated the avro version in the system to a newer release, which among other things will allow people to build on Mac OS without running into the JDK7+Snappy 1.0.4 problem where tests because native library does not load. Regards, Arvind On Thu, Nov 6, 2014 at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Flume 1.5.1 RC1
+1 * Verified checksums and signature files * Verified that each jar in binary tarball is in the license * Checked top level files (NOTICE, ...) * Run tests Jarcec On Nov 6, 2014, at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari
[jira] [Commented] (FLUME-2385) Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level
[ https://issues.apache.org/jira/browse/FLUME-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14205980#comment-14205980 ] Hari Shreedharan commented on FLUME-2385: - +1. Running tests and committing. Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level Key: FLUME-2385 URL: https://issues.apache.org/jira/browse/FLUME-2385 Project: Flume Issue Type: Improvement Affects Versions: v1.4.0 Reporter: Justin Hayes Assignee: Phil Scala Priority: Minor Fix For: v1.6.0 Attachments: FLUME-2385-0.patch When I start an agent with the following config, the spooling directory source emits 14/05/14 22:36:12 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. messages twice a second. Pretty innocuous but it will fill up the file system needlessly and get in the way of other INFO messages. cis.sources = httpd cis.sinks = loggerSink cis.channels = mem2logger cis.sources.httpd.type = spooldir cis.sources.httpd.spoolDir = /var/log/httpd cis.sources.httpd.trackerDir = /var/lib/flume-ng/tracker/httpd cis.sources.httpd.channels = mem2logger cis.sinks.loggerSink.type = logger cis.sinks.loggerSink.channel = mem2logger cis.channels.mem2logger.type = memory cis.channels.mem2logger.capacity = 1 cis.channels.mem2logger.transactionCapacity = 1000 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2385) Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level
[ https://issues.apache.org/jira/browse/FLUME-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206031#comment-14206031 ] ASF subversion and git services commented on FLUME-2385: Commit c1512241c0104daa80f58c6146ecb759ef6417da in flume's branch refs/heads/flume-1.6 from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=c151224 ] FLUME-2385. Remove incorrect log message at INFO level in Spool Directory Source. (Phil Scala via Hari Shreedharan) Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level Key: FLUME-2385 URL: https://issues.apache.org/jira/browse/FLUME-2385 Project: Flume Issue Type: Improvement Affects Versions: v1.4.0 Reporter: Justin Hayes Assignee: Phil Scala Priority: Minor Fix For: v1.6.0 Attachments: FLUME-2385-0.patch When I start an agent with the following config, the spooling directory source emits 14/05/14 22:36:12 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. messages twice a second. Pretty innocuous but it will fill up the file system needlessly and get in the way of other INFO messages. cis.sources = httpd cis.sinks = loggerSink cis.channels = mem2logger cis.sources.httpd.type = spooldir cis.sources.httpd.spoolDir = /var/log/httpd cis.sources.httpd.trackerDir = /var/lib/flume-ng/tracker/httpd cis.sources.httpd.channels = mem2logger cis.sinks.loggerSink.type = logger cis.sinks.loggerSink.channel = mem2logger cis.channels.mem2logger.type = memory cis.channels.mem2logger.capacity = 1 cis.channels.mem2logger.transactionCapacity = 1000 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2385) Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level
[ https://issues.apache.org/jira/browse/FLUME-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206030#comment-14206030 ] ASF subversion and git services commented on FLUME-2385: Commit 2c18533253be786b9c60bf687cdf38d2384d2625 in flume's branch refs/heads/trunk from [~hshreedharan] [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=2c18533 ] FLUME-2385. Remove incorrect log message at INFO level in Spool Directory Source. (Phil Scala via Hari Shreedharan) Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level Key: FLUME-2385 URL: https://issues.apache.org/jira/browse/FLUME-2385 Project: Flume Issue Type: Improvement Affects Versions: v1.4.0 Reporter: Justin Hayes Assignee: Phil Scala Priority: Minor Fix For: v1.6.0 Attachments: FLUME-2385-0.patch When I start an agent with the following config, the spooling directory source emits 14/05/14 22:36:12 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. messages twice a second. Pretty innocuous but it will fill up the file system needlessly and get in the way of other INFO messages. cis.sources = httpd cis.sinks = loggerSink cis.channels = mem2logger cis.sources.httpd.type = spooldir cis.sources.httpd.spoolDir = /var/log/httpd cis.sources.httpd.trackerDir = /var/lib/flume-ng/tracker/httpd cis.sources.httpd.channels = mem2logger cis.sinks.loggerSink.type = logger cis.sinks.loggerSink.channel = mem2logger cis.channels.mem2logger.type = memory cis.channels.mem2logger.capacity = 1 cis.channels.mem2logger.transactionCapacity = 1000 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [VOTE] Release Apache Flume 1.5.1 RC1
This vote is now closed. This vote has passed with 4 +1 binding votes: - Arvind Prabhakar - Roshan Naik - Hari Shreedharan - Jarcec Non-binding +1 votes: - Ashish Paliwal No +0 or -1 votes were received. As a result, this RC has passed and will be promoted to the Apache Flume 1.5.1 release. I will send out an email announcing the release soon! Thanks, Hari On Mon, Nov 10, 2014 at 9:33 PM, Jarek Jarcec Cecho jar...@apache.org wrote: +1 * Verified checksums and signature files * Verified that each jar in binary tarball is in the license * Checked top level files (NOTICE, ...) * Run tests Jarcec On Nov 6, 2014, at 3:17 PM, Hari Shreedharan hshreedha...@cloudera.com wrote: This is the seventh release for Apache Flume as a top-level project, version 1.5.1. We are voting on release candidate RC1. It fixes the following issues: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=blob_plain;f=CHANGELOG;hb=c74804226bcee59823c0cbc09cdf803a3d9e6920 *** Please cast your vote within the next 72 hours *** The tarball (*.tar.gz), signature (*.asc), and checksums (*.md5, *.sha1) for the source and binary artifacts can be found here: https://people.apache.org/~hshreedharan/apache-flume-1.5.1-rc1/ Maven staging repo: https://repository.apache.org/content/repositories/orgapacheflume-1006/ The tag to be voted on: https://git-wip-us.apache.org/repos/asf?p=flume.git;a=commit;h=c74804226bcee59823c0cbc09cdf803a3d9e6920 Flume's KEYS file containing PGP keys we use to sign the release: http://www.apache.org/dist/flume/KEYS Thanks, Hari
Jenkins build became unstable: Flume-trunk-hbase-98 #49
See https://builds.apache.org/job/Flume-trunk-hbase-98/49/changes
[jira] [Commented] (FLUME-2385) Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level
[ https://issues.apache.org/jira/browse/FLUME-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14206087#comment-14206087 ] Hudson commented on FLUME-2385: --- UNSTABLE: Integrated in Flume-trunk-hbase-98 #49 (See [https://builds.apache.org/job/Flume-trunk-hbase-98/49/]) FLUME-2385. Remove incorrect log message at INFO level in Spool Directory Source. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.gita=commith=2c18533253be786b9c60bf687cdf38d2384d2625) * flume-ng-core/src/main/java/org/apache/flume/source/SpoolDirectorySource.java Flume spans log file with Spooling Directory Source runner has shutdown messages at INFO level Key: FLUME-2385 URL: https://issues.apache.org/jira/browse/FLUME-2385 Project: Flume Issue Type: Improvement Affects Versions: v1.4.0 Reporter: Justin Hayes Assignee: Phil Scala Priority: Minor Fix For: v1.6.0 Attachments: FLUME-2385-0.patch When I start an agent with the following config, the spooling directory source emits 14/05/14 22:36:12 INFO source.SpoolDirectorySource: Spooling Directory Source runner has shutdown. messages twice a second. Pretty innocuous but it will fill up the file system needlessly and get in the way of other INFO messages. cis.sources = httpd cis.sinks = loggerSink cis.channels = mem2logger cis.sources.httpd.type = spooldir cis.sources.httpd.spoolDir = /var/log/httpd cis.sources.httpd.trackerDir = /var/lib/flume-ng/tracker/httpd cis.sources.httpd.channels = mem2logger cis.sinks.loggerSink.type = logger cis.sinks.loggerSink.channel = mem2logger cis.channels.mem2logger.type = memory cis.channels.mem2logger.capacity = 1 cis.channels.mem2logger.transactionCapacity = 1000 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
