[jira] [Assigned] (FLUME-2725) HDFS Sink does not use configured timezone for rounding
[ https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay reassigned FLUME-2725: -- Assignee: Denes Arvay > HDFS Sink does not use configured timezone for rounding > --- > > Key: FLUME-2725 > URL: https://issues.apache.org/jira/browse/FLUME-2725 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Eric Czech >Assignee: Denes Arvay >Priority: Minor > > When a BucketPath used by an HDFS sink is configured to run with some > roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by > BucketPath does not actually round the date correctly. > That function calls TimestampRoundDownUtil which creates a Calendar instance > using the *local* timezone to truncate a unix timestamp rather than the > TimeZone that the sink was configured to convert dates to paths with (and > that timezone is already available in the BucketPath class but it just isn't > passed to TimestampRoundDownUtil). > The net effect of this is that if a flume jvm is running on a system with an > EST clock while trying to write, say, 6 hour directories in UTC time, the > directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, > 18 like you would expect. > I found a workaround for this by passing > "-Duser.timezone=" as a system property, but I wanted to > create a ticket for this since it seems like it would be very minimal effort > to carry that configured timezone down into the rounding utility as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2870) Flume agent throw NULLPOINTEXCEPTION when use kakfa-source
[
https://issues.apache.org/jira/browse/FLUME-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay reassigned FLUME-2870:
--
Assignee: Denes Arvay
> Flume agent throw NULLPOINTEXCEPTION when use kakfa-source
> --
>
> Key: FLUME-2870
> URL: https://issues.apache.org/jira/browse/FLUME-2870
> Project: Flume
> Issue Type: Bug
> Components: Build
>Affects Versions: v1.6.0
> Environment: kafka source,es sink
>Reporter: justdebugit
>Assignee: Denes Arvay
> Labels: kafka-source
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> CONFIG:
> a1.sources = s1
> a1.channels = c1
> a1.sinks = k1
> a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
> a1.sources.s1.channels = c1
> a1.sources.s1.zookeeperConnect =
> 10.xxx.xxx.160:5181,10.2x.xxx.203:5181/xxx/charge
> a1.sources.s1.topic = charge.log
> a1.sources.s1.groupId = flume
> #a1.channels.c1.type = memory
> #a1.channels.c1.capacity = 5
> #a1.channels.c1.transactionCapacity = 100
> a1.channels.c1.type = file
> a1.channels.c1.checkpointDir = /home//app/flume/checkpoint
> a1.channels.c1.dataDirs = /home/x/app/flume/data
> a1.channels.c1.capacity = 500
> a1.sinks.k1.type = com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink
> a1.sinks.k1.hostNames = 127.0.0.1:9300
> a1.sinks.k1.indexName = xxx_charge_log
> a1.sinks.k1.indexType = logs
> a1.sinks.k1.clusterName = elasticsearch
> a1.sinks.k1.batchSize = 500
> a1.sinks.k1.ttl = 3d
> a1.sinks.k1.serializer =
> com.frontier45.flume.sink.elasticsearch2.ElasticSearchLogStashEventSerializer
> a1.sinks.k1.channel = c1
> EXCEPTION:
> 28 Jan 2016 11:09:03,830 ERROR [PollableSourceRunner-KafkaSource-s1]
> (org.apache.flume.source.kafka.KafkaSource.process:153) - KafkaSource
> EXCEPTION, {}
> java.lang.NullPointerException
> at
> org.apache.flume.instrumentation.MonitoredCounterGroup.increment(MonitoredCounterGroup.java:261)
> at
> org.apache.flume.instrumentation.kafka.KafkaSourceCounter.incrementKafkaEmptyCount(KafkaSourceCounter.java:49)
> at
> org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:146)
> at
> org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2870) Flume agent throw NULLPOINTEXCEPTION when use kakfa-source
[
https://issues.apache.org/jira/browse/FLUME-2870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356714#comment-15356714
]
Denes Arvay commented on FLUME-2870:
This one seems to be a duplicate of FLUME-2672, so if there are no objections
I'd close it as duplicate.
> Flume agent throw NULLPOINTEXCEPTION when use kakfa-source
> --
>
> Key: FLUME-2870
> URL: https://issues.apache.org/jira/browse/FLUME-2870
> Project: Flume
> Issue Type: Bug
> Components: Build
>Affects Versions: v1.6.0
> Environment: kafka source,es sink
>Reporter: justdebugit
>Assignee: Denes Arvay
> Labels: kafka-source
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> CONFIG:
> a1.sources = s1
> a1.channels = c1
> a1.sinks = k1
> a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
> a1.sources.s1.channels = c1
> a1.sources.s1.zookeeperConnect =
> 10.xxx.xxx.160:5181,10.2x.xxx.203:5181/xxx/charge
> a1.sources.s1.topic = charge.log
> a1.sources.s1.groupId = flume
> #a1.channels.c1.type = memory
> #a1.channels.c1.capacity = 5
> #a1.channels.c1.transactionCapacity = 100
> a1.channels.c1.type = file
> a1.channels.c1.checkpointDir = /home//app/flume/checkpoint
> a1.channels.c1.dataDirs = /home/x/app/flume/data
> a1.channels.c1.capacity = 500
> a1.sinks.k1.type = com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink
> a1.sinks.k1.hostNames = 127.0.0.1:9300
> a1.sinks.k1.indexName = xxx_charge_log
> a1.sinks.k1.indexType = logs
> a1.sinks.k1.clusterName = elasticsearch
> a1.sinks.k1.batchSize = 500
> a1.sinks.k1.ttl = 3d
> a1.sinks.k1.serializer =
> com.frontier45.flume.sink.elasticsearch2.ElasticSearchLogStashEventSerializer
> a1.sinks.k1.channel = c1
> EXCEPTION:
> 28 Jan 2016 11:09:03,830 ERROR [PollableSourceRunner-KafkaSource-s1]
> (org.apache.flume.source.kafka.KafkaSource.process:153) - KafkaSource
> EXCEPTION, {}
> java.lang.NullPointerException
> at
> org.apache.flume.instrumentation.MonitoredCounterGroup.increment(MonitoredCounterGroup.java:261)
> at
> org.apache.flume.instrumentation.kafka.KafkaSourceCounter.incrementKafkaEmptyCount(KafkaSourceCounter.java:49)
> at
> org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:146)
> at
> org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2725) HDFS Sink does not use configured timezone for rounding
[ https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2725: --- Attachment: FLUME-2725.patch > HDFS Sink does not use configured timezone for rounding > --- > > Key: FLUME-2725 > URL: https://issues.apache.org/jira/browse/FLUME-2725 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Eric Czech >Assignee: Denes Arvay >Priority: Minor > Attachments: FLUME-2725.patch > > > When a BucketPath used by an HDFS sink is configured to run with some > roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by > BucketPath does not actually round the date correctly. > That function calls TimestampRoundDownUtil which creates a Calendar instance > using the *local* timezone to truncate a unix timestamp rather than the > TimeZone that the sink was configured to convert dates to paths with (and > that timezone is already available in the BucketPath class but it just isn't > passed to TimestampRoundDownUtil). > The net effect of this is that if a flume jvm is running on a system with an > EST clock while trying to write, say, 6 hour directories in UTC time, the > directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, > 18 like you would expect. > I found a workaround for this by passing > "-Duser.timezone=" as a system property, but I wanted to > create a ticket for this since it seems like it would be very minimal effort > to carry that configured timezone down into the rounding utility as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2725) HDFS Sink does not use configured timezone for rounding
[ https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358826#comment-15358826 ] Denes Arvay commented on FLUME-2725: I've attached a patch to use the configured time zone for rounding but I'm a bit concerned about this change as it might break stuff at users who already adapted to the current behavior. Shouldn't we put this behind a configuration parameter, wdyt? > HDFS Sink does not use configured timezone for rounding > --- > > Key: FLUME-2725 > URL: https://issues.apache.org/jira/browse/FLUME-2725 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Eric Czech >Assignee: Denes Arvay >Priority: Minor > Attachments: FLUME-2725.patch > > > When a BucketPath used by an HDFS sink is configured to run with some > roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by > BucketPath does not actually round the date correctly. > That function calls TimestampRoundDownUtil which creates a Calendar instance > using the *local* timezone to truncate a unix timestamp rather than the > TimeZone that the sink was configured to convert dates to paths with (and > that timezone is already available in the BucketPath class but it just isn't > passed to TimestampRoundDownUtil). > The net effect of this is that if a flume jvm is running on a system with an > EST clock while trying to write, say, 6 hour directories in UTC time, the > directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, > 18 like you would expect. > I found a workaround for this by passing > "-Duser.timezone=" as a system property, but I wanted to > create a ticket for this since it seems like it would be very minimal effort > to carry that configured timezone down into the rounding utility as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2725) HDFS Sink does not use configured timezone for rounding
[ https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2725: --- Attachment: FLUME-2725-2.patch rebased on trunk + fixed checkstyle errors > HDFS Sink does not use configured timezone for rounding > --- > > Key: FLUME-2725 > URL: https://issues.apache.org/jira/browse/FLUME-2725 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Eric Czech >Assignee: Denes Arvay >Priority: Minor > Attachments: FLUME-2725-2.patch, FLUME-2725.patch > > > When a BucketPath used by an HDFS sink is configured to run with some > roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by > BucketPath does not actually round the date correctly. > That function calls TimestampRoundDownUtil which creates a Calendar instance > using the *local* timezone to truncate a unix timestamp rather than the > TimeZone that the sink was configured to convert dates to paths with (and > that timezone is already available in the BucketPath class but it just isn't > passed to TimestampRoundDownUtil). > The net effect of this is that if a flume jvm is running on a system with an > EST clock while trying to write, say, 6 hour directories in UTC time, the > directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, > 18 like you would expect. > I found a workaround for this by passing > "-Duser.timezone=" as a system property, but I wanted to > create a ticket for this since it seems like it would be very minimal effort > to carry that configured timezone down into the rounding utility as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (FLUME-2725) HDFS Sink does not use configured timezone for rounding
[ https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15372535#comment-15372535 ] Denes Arvay edited comment on FLUME-2725 at 7/12/16 8:40 AM: - new patch: rebased on trunk + fixed checkstyle errors was (Author: denes): rebased on trunk + fixed checkstyle errors > HDFS Sink does not use configured timezone for rounding > --- > > Key: FLUME-2725 > URL: https://issues.apache.org/jira/browse/FLUME-2725 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Eric Czech >Assignee: Denes Arvay >Priority: Minor > Attachments: FLUME-2725-2.patch, FLUME-2725.patch > > > When a BucketPath used by an HDFS sink is configured to run with some > roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by > BucketPath does not actually round the date correctly. > That function calls TimestampRoundDownUtil which creates a Calendar instance > using the *local* timezone to truncate a unix timestamp rather than the > TimeZone that the sink was configured to convert dates to paths with (and > that timezone is already available in the BucketPath class but it just isn't > passed to TimestampRoundDownUtil). > The net effect of this is that if a flume jvm is running on a system with an > EST clock while trying to write, say, 6 hour directories in UTC time, the > directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, > 18 like you would expect. > I found a workaround for this by passing > "-Duser.timezone=" as a system property, but I wanted to > create a ticket for this since it seems like it would be very minimal effort > to carry that configured timezone down into the rounding utility as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2725) HDFS Sink does not use configured timezone for rounding
[ https://issues.apache.org/jira/browse/FLUME-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2725: --- Attachment: FLUME-2725-3.patch > HDFS Sink does not use configured timezone for rounding > --- > > Key: FLUME-2725 > URL: https://issues.apache.org/jira/browse/FLUME-2725 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Reporter: Eric Czech >Assignee: Denes Arvay >Priority: Minor > Attachments: FLUME-2725-2.patch, FLUME-2725-3.patch, FLUME-2725.patch > > > When a BucketPath used by an HDFS sink is configured to run with some > roundUnit and roundValue > 1 (e.g. 6 hours), the "roundDown" function used by > BucketPath does not actually round the date correctly. > That function calls TimestampRoundDownUtil which creates a Calendar instance > using the *local* timezone to truncate a unix timestamp rather than the > TimeZone that the sink was configured to convert dates to paths with (and > that timezone is already available in the BucketPath class but it just isn't > passed to TimestampRoundDownUtil). > The net effect of this is that if a flume jvm is running on a system with an > EST clock while trying to write, say, 6 hour directories in UTC time, the > directories are written with the hours 04, 10, 16, 22 rather than 00, 06, 12, > 18 like you would expect. > I found a workaround for this by passing > "-Duser.timezone=" as a system property, but I wanted to > create a ticket for this since it seems like it would be very minimal effort > to carry that configured timezone down into the rounding utility as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2951) Exec Source generate massive logfile in File Channel data dirs
[ https://issues.apache.org/jira/browse/FLUME-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381824#comment-15381824 ] Denes Arvay commented on FLUME-2951: hi [~lolee_k], could you please create a review request on https://reviews.apache.org for this issue? Thanks. > Exec Source generate massive logfile in File Channel data dirs > -- > > Key: FLUME-2951 > URL: https://issues.apache.org/jira/browse/FLUME-2951 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: dengkai > Attachments: FLUME-2951.patch > > > When the file channel is full, exec source always get a filechannel > exception.Every time the main thread start to roll, and then the > timedFlushService flush the events again.So an exception follows : > 07 Jul 2016 17:09:38,789 ERROR [timedFlushExecService63-0] > (org.apache.flume.source.ExecSource$ExecRunnable$1.run:328) - Exception > occured when processing event batch > org.apache.flume.ChannelException: Commit failed due to IO error > [channel=channel-4] > at > org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:621) > at > org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168) > at > org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:194) > at > org.apache.flume.source.ExecSource$ExecRunnable.flushEventBatch(ExecSource.java:382) > at > org.apache.flume.source.ExecSource$ExecRunnable.access$100(ExecSource.java:255) > at > org.apache.flume.source.ExecSource$ExecRunnable$1.run(ExecSource.java:324) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.nio.channels.ClosedByInterruptException > at > java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202) > at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:380) > at > org.apache.flume.channel.file.LogFileV3.writeDelimitedTo(LogFileV3.java:148) > at > org.apache.flume.channel.file.LogFileV3$Writer.(LogFileV3.java:209) > at > org.apache.flume.channel.file.LogFileFactory.getWriter(LogFileFactory.java:77) > at org.apache.flume.channel.file.Log.roll(Log.java:964) > at org.apache.flume.channel.file.Log.roll(Log.java:933) > at org.apache.flume.channel.file.Log.rollback(Log.java:740) > at > org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:619) > ... 12 more > When this exceptions happens, file channel start another roll, and the ID of > the log file keep increasing. > If the exec source is configured as restart = true and file channel is still > full, this problem always be there and will generate massive log-ID size > about 1.0MB and log-ID.meta size about 47bytes in the channel's data dir(in > my try, I even got thousands of files and got too many files error when exec > source started tail -F command). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2955) Add file path to the header in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15381991#comment-15381991 ] Denes Arvay commented on FLUME-2955: hi [~wenqiao], could you please create a review request for this issue on https://reviews.apache.org for us to be able to review? Thanks. > Add file path to the header in TaildirSource > > > Key: FLUME-2955 > URL: https://issues.apache.org/jira/browse/FLUME-2955 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao > Labels: file, path, taildirsource > Fix For: v1.7.0 > > Attachments: FLUME-2955.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > After using TaildirSource to collect log data, I found that there is no file > path in the flume event. > Though TaildirSource supports add customed header value like this : > a1.sources.r1.headers.f1.headerKey1 = value1, > we can't add file path to the header when the filegroup path contains regex > expression.File path is necessary to locate the log data. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2514) Some TestFileChannelRestart tests are extremely slow
[ https://issues.apache.org/jira/browse/FLUME-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2514: --- Attachment: FLUME-2215-1.patch > Some TestFileChannelRestart tests are extremely slow > > > Key: FLUME-2514 > URL: https://issues.apache.org/jira/browse/FLUME-2514 > Project: Flume > Issue Type: Bug > Components: File Channel >Reporter: Santiago M. Mola >Assignee: Santiago M. Mola > Fix For: v1.7.0 > > Attachments: FLUME-2215-0.patch, FLUME-2215-1.patch > > > TestFileChannelRestart tests are really slow. For example, > testToggleCheckpointCompressionFromFalseToTrue and > testToggleCheckpointCompressionFromTrueToFalse take ~4 minutes each. > Some of them could be made faster by using channels with lower capacity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2807) Add a simple split interceptor
[ https://issues.apache.org/jira/browse/FLUME-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2807: --- Labels: features patch reviewboard-missing (was: features patch) > Add a simple split interceptor > --- > > Key: FLUME-2807 > URL: https://issues.apache.org/jira/browse/FLUME-2807 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0, v1.7.0 >Reporter: seekerak > Labels: features, patch, reviewboard-missing, unit-test-missing > Fix For: v1.7.0 > > Attachments: FLUME-2807.patch > > > a simple split interceptor , aims to deal with the situation as follows: > the source data like this: > “ > 1,tom,boy,13 > 2,lili,girl,14 > 3,jack,boy,10 > ... > ” > and i hope to sink source data into two different hdfs directories named by > boy and girl like this: > “ > hdfs:///sink/boy/ > hdfs:///sink/girl/ > ” > we can use this interceptor to accomplish this goal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2807) Add a simple split interceptor
[ https://issues.apache.org/jira/browse/FLUME-2807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2807: --- Labels: features patch reviewboard-missing unit-test-missing (was: features patch reviewboard-missing) > Add a simple split interceptor > --- > > Key: FLUME-2807 > URL: https://issues.apache.org/jira/browse/FLUME-2807 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.6.0, v1.7.0 >Reporter: seekerak > Labels: features, patch, reviewboard-missing, unit-test-missing > Fix For: v1.7.0 > > Attachments: FLUME-2807.patch > > > a simple split interceptor , aims to deal with the situation as follows: > the source data like this: > “ > 1,tom,boy,13 > 2,lili,girl,14 > 3,jack,boy,10 > ... > ” > and i hope to sink source data into two different hdfs directories named by > boy and girl like this: > “ > hdfs:///sink/boy/ > hdfs:///sink/girl/ > ” > we can use this interceptor to accomplish this goal. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2716) File Channel cannot handle capacity Integer.MAX_VALUE
[ https://issues.apache.org/jira/browse/FLUME-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2716: --- Labels: unit-test-missing (was: ) > File Channel cannot handle capacity Integer.MAX_VALUE > - > > Key: FLUME-2716 > URL: https://issues.apache.org/jira/browse/FLUME-2716 > Project: Flume > Issue Type: Bug > Components: Channel, File Channel >Affects Versions: v1.6.0, v1.7.0 >Reporter: Dong Zhao > Labels: unit-test-missing > Fix For: v1.7.0 > > Attachments: FLUME-2716.patch > > > if capacity is set to Integer.MAX_VALUE(2147483647), checkpoint file size is > calculated wrongly to 8224. The calculation should first cast int to long, > then calculate the totalBytes. See the patch for details. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2802) Folder name interceptor
[ https://issues.apache.org/jira/browse/FLUME-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2802: --- Labels: reviewboard-missing (was: ) > Folder name interceptor > --- > > Key: FLUME-2802 > URL: https://issues.apache.org/jira/browse/FLUME-2802 > Project: Flume > Issue Type: New Feature >Reporter: Eran W >Assignee: Eran W > Labels: reviewboard-missing > Fix For: notrack, v1.7.0 > > Attachments: FLUME-2802.patch, flume-2802.patch > > > This interceptor retrieve the last folder name from the > SpoolDir.fileHeaderKey and set it to the given folderKey. > This is allow users to set the target hdfs directory based on the source > directory and not the whole path or file name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389658#comment-15389658 ] Denes Arvay commented on FLUME-2797: Hi [~otis], thanks for the patch. It's quite an old issue but I've just bumped into a problem which could be solved by this change so I'd be happy to review it. Could you please submit your patch to the reviewboard? Thank you. > SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source > --- > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2844) ChannelCounter of SpillableMemoryChannel doesn't register actually.
[ https://issues.apache.org/jira/browse/FLUME-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389667#comment-15389667 ] Denes Arvay commented on FLUME-2844: Hi [~sandywei], thanks for the patch. Could you please submit it to reviewboard.apache.org? I'd be happy to review it there. Thank you. > ChannelCounter of SpillableMemoryChannel doesn't register actually. > --- > > Key: FLUME-2844 > URL: https://issues.apache.org/jira/browse/FLUME-2844 > Project: Flume > Issue Type: Bug > Components: Channel >Affects Versions: v1.6.0 >Reporter: Daniel Wen > Attachments: FLUME-2844..patch > > > When using SpillableMemoryChannel, the values of all metrics of channel > component int monitoring system is zero.It's caused by a bug belowed. > SpillableMemoryChannel extends FileChannel,and each of them have a > ChannelCounter-Type field named of 'channelCounter'. FileChannel's > channelCounter filed is private, so it can not be override by > SpillableMemoryChannel. SpillableMemoryChannel's channelCounter is > initialized in configure method(Line 600 at SpillableMemoryChannel.java), > then super.configure() is called(Line 727 at SpillableMemoryChannel.java). So > FileChannel's channelCounter is also initialized(Line 267 at > SpillableMemoryChannel.java). > When SpillableMemoryChannel start(call start() method), it should and want to > register ChannelCounter. In start() method, SpillableMemoryChannel calls > super.start() (Line 768 at SpillableMemoryChannel.java), then FileChannel > calls channelCounter.start() to register channelCounter. But this > channelCounter is the one of FileChannle, not the one of > SpillableMemoryChannel! > This bug results that when using SpillableMemoryChannel, the values of all > metrics of channel component is zero. > Maybe the qualifier of FileChannel's channelCounter filed should be changed > to prtected, and remove the channelCounter filed of SpillableMemoryChannel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389876#comment-15389876 ] Denes Arvay commented on FLUME-2797: Ok, thanks for the reply. > SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source > --- > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay reassigned FLUME-2797: -- Assignee: Denes Arvay > SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source > --- > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391704#comment-15391704 ] Denes Arvay commented on FLUME-2960: Hi [~wenqiao], thanks for the patch. I haven't checked the reviewboard in details yet, but one question popped into my mind: is there any specific reason for choosing glob pattern instead of regex? You get the expected result with the current implementation but for the first sight I found it a bit confusing having regexes and glob patterns together. cc [~sati], [~bessbd], [~mpercy] > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.7.0 > > Attachments: FLUME-2960_1.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391895#comment-15391895 ] Denes Arvay commented on FLUME-2960: hi, I didn't really think it through but my first idea was to use regex in both the directory path and in the filename to avoid the confusion because of the mixed usage. Although I don't know yet whether it has any drawbacks, that's why I'm interested in your and others' opinion. > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.7.0 > > Attachments: FLUME-2960_1.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2797: --- Summary: SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource (was: SyslogTcpSource uses Deprecated Class + Deprecate Syslog Source) > SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource > - > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2797: --- Attachment: FLUME-2797-2.patch > SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource > - > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch, FLUME-2797-2.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2797: --- Attachment: FLUME-2797-3.patch > SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource > - > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch, FLUME-2797-2.patch, > FLUME-2797-3.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2844) ChannelCounter of SpillableMemoryChannel doesn't register actually.
[
https://issues.apache.org/jira/browse/FLUME-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2844:
---
Attachment: FLUME-2844-2.patch
I've attached a new patch. I decided not to touch the superclass'
{{channelCounter}}
> ChannelCounter of SpillableMemoryChannel doesn't register actually.
> ---
>
> Key: FLUME-2844
> URL: https://issues.apache.org/jira/browse/FLUME-2844
> Project: Flume
> Issue Type: Bug
> Components: Channel
>Affects Versions: v1.6.0
>Reporter: Daniel Wen
> Attachments: FLUME-2844-2.patch, FLUME-2844..patch
>
>
> When using SpillableMemoryChannel, the values of all metrics of channel
> component int monitoring system is zero.It's caused by a bug belowed.
> SpillableMemoryChannel extends FileChannel,and each of them have a
> ChannelCounter-Type field named of 'channelCounter'. FileChannel's
> channelCounter filed is private, so it can not be override by
> SpillableMemoryChannel. SpillableMemoryChannel's channelCounter is
> initialized in configure method(Line 600 at SpillableMemoryChannel.java),
> then super.configure() is called(Line 727 at SpillableMemoryChannel.java). So
> FileChannel's channelCounter is also initialized(Line 267 at
> SpillableMemoryChannel.java).
> When SpillableMemoryChannel start(call start() method), it should and want to
> register ChannelCounter. In start() method, SpillableMemoryChannel calls
> super.start() (Line 768 at SpillableMemoryChannel.java), then FileChannel
> calls channelCounter.start() to register channelCounter. But this
> channelCounter is the one of FileChannle, not the one of
> SpillableMemoryChannel!
> This bug results that when using SpillableMemoryChannel, the values of all
> metrics of channel component is zero.
> Maybe the qualifier of FileChannel's channelCounter filed should be changed
> to prtected, and remove the channelCounter filed of SpillableMemoryChannel.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2844) ChannelCounter of SpillableMemoryChannel doesn't register actually.
[ https://issues.apache.org/jira/browse/FLUME-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay reassigned FLUME-2844: -- Assignee: Denes Arvay > ChannelCounter of SpillableMemoryChannel doesn't register actually. > --- > > Key: FLUME-2844 > URL: https://issues.apache.org/jira/browse/FLUME-2844 > Project: Flume > Issue Type: Bug > Components: Channel >Affects Versions: v1.6.0 >Reporter: Daniel Wen >Assignee: Denes Arvay > Attachments: FLUME-2844-2.patch, FLUME-2844..patch > > > When using SpillableMemoryChannel, the values of all metrics of channel > component int monitoring system is zero.It's caused by a bug belowed. > SpillableMemoryChannel extends FileChannel,and each of them have a > ChannelCounter-Type field named of 'channelCounter'. FileChannel's > channelCounter filed is private, so it can not be override by > SpillableMemoryChannel. SpillableMemoryChannel's channelCounter is > initialized in configure method(Line 600 at SpillableMemoryChannel.java), > then super.configure() is called(Line 727 at SpillableMemoryChannel.java). So > FileChannel's channelCounter is also initialized(Line 267 at > SpillableMemoryChannel.java). > When SpillableMemoryChannel start(call start() method), it should and want to > register ChannelCounter. In start() method, SpillableMemoryChannel calls > super.start() (Line 768 at SpillableMemoryChannel.java), then FileChannel > calls channelCounter.start() to register channelCounter. But this > channelCounter is the one of FileChannle, not the one of > SpillableMemoryChannel! > This bug results that when using SpillableMemoryChannel, the values of all > metrics of channel component is zero. > Maybe the qualifier of FileChannel's channelCounter filed should be changed > to prtected, and remove the channelCounter filed of SpillableMemoryChannel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2963) FlumeUserGuide - error in Kafka Source properties table
Denes Arvay created FLUME-2963:
--
Summary: FlumeUserGuide - error in Kafka Source properties table
Key: FLUME-2963
URL: https://issues.apache.org/jira/browse/FLUME-2963
Project: Flume
Issue Type: Documentation
Components: Docs
Reporter: Denes Arvay
Assignee: Denes Arvay
Priority: Trivial
{noformat}
System Message: ERROR/3 (FlumeUserGuide.rst, line 1208)
Malformed table. Column span alignment problem at line offset 27.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2963) FlumeUserGuide - error in Kafka Source properties table
[
https://issues.apache.org/jira/browse/FLUME-2963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2963:
---
Attachment: FLUME-2963.patch
attached patch fixes the table parsing issue + some minor formatting glitches
in that row
> FlumeUserGuide - error in Kafka Source properties table
> ---
>
> Key: FLUME-2963
> URL: https://issues.apache.org/jira/browse/FLUME-2963
> Project: Flume
> Issue Type: Documentation
> Components: Docs
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Trivial
> Attachments: FLUME-2963.patch
>
>
> {noformat}
> System Message: ERROR/3 (FlumeUserGuide.rst, line 1208)
> Malformed table. Column span alignment problem at line offset 27.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2844) ChannelCounter of SpillableMemoryChannel doesn't register actually.
[ https://issues.apache.org/jira/browse/FLUME-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399149#comment-15399149 ] Denes Arvay commented on FLUME-2844: Thanks [~mpercy] for your comment and sorry for the confusion. [~sandywei], of course I'm totally fine with dropping my patch and moving forward with yours if you are working on it. Please let me know your thoughts on this. > ChannelCounter of SpillableMemoryChannel doesn't register actually. > --- > > Key: FLUME-2844 > URL: https://issues.apache.org/jira/browse/FLUME-2844 > Project: Flume > Issue Type: Bug > Components: Channel >Affects Versions: v1.6.0 >Reporter: Daniel Wen >Assignee: Denes Arvay > Attachments: FLUME-2844-2.patch, FLUME-2844..patch > > > When using SpillableMemoryChannel, the values of all metrics of channel > component int monitoring system is zero.It's caused by a bug belowed. > SpillableMemoryChannel extends FileChannel,and each of them have a > ChannelCounter-Type field named of 'channelCounter'. FileChannel's > channelCounter filed is private, so it can not be override by > SpillableMemoryChannel. SpillableMemoryChannel's channelCounter is > initialized in configure method(Line 600 at SpillableMemoryChannel.java), > then super.configure() is called(Line 727 at SpillableMemoryChannel.java). So > FileChannel's channelCounter is also initialized(Line 267 at > SpillableMemoryChannel.java). > When SpillableMemoryChannel start(call start() method), it should and want to > register ChannelCounter. In start() method, SpillableMemoryChannel calls > super.start() (Line 768 at SpillableMemoryChannel.java), then FileChannel > calls channelCounter.start() to register channelCounter. But this > channelCounter is the one of FileChannle, not the one of > SpillableMemoryChannel! > This bug results that when using SpillableMemoryChannel, the values of all > metrics of channel component is zero. > Maybe the qualifier of FileChannel's channelCounter filed should be changed > to prtected, and remove the channelCounter filed of SpillableMemoryChannel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2797: --- Attachment: FLUME-2797-4.patch new patch: test logging statement removed, thanks [~sati] for the review > SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource > - > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch, FLUME-2797-2.patch, > FLUME-2797-3.patch, FLUME-2797-4.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2965) race condition in SpillableMemoryChannel log print
[
https://issues.apache.org/jira/browse/FLUME-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15406313#comment-15406313
]
Denes Arvay commented on FLUME-2965:
Hi [~lqp276], thank you for your bug report and fix. Could you please submit it
to the Review board at https://reviews.apache.org? If you need any assistance
on this, let me know.
Please also add some unit tests for your fix, if it's possible. Thanks.
> race condition in SpillableMemoryChannel log print
> --
>
> Key: FLUME-2965
> URL: https://issues.apache.org/jira/browse/FLUME-2965
> Project: Flume
> Issue Type: Bug
> Components: Channel
>Affects Versions: v1.7.0
>Reporter: liqiaoping
>Priority: Minor
> Attachments: SpillableMemoryChannel.java
>
>
> use SpillableMemoryChannel with http blob handler, and send many request
> concurrently, As the jetty has a threadpool to handle incoming request, the
> commit to SpillableMemoryChannel will be concurrent.
> the Following code :
> @Override
> protected void doCommit() throws InterruptedException {
> if (putCalled) {
> putCommit();
> if (LOGGER.isDebugEnabled()) {
> LOGGER.debug("Put Committed. Drain Order Queue state : "
> + drainOrder.dump());
> }
> in method - >drainOrder.dump() will iterate its internal queue, in the
> meantime, has changed by other thread, thus throw a concurrent modification
> exception. thus will result the channel processor try to rollback, but
> actually the transaction has commit succefully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2844) ChannelCounter of SpillableMemoryChannel doesn't register actually.
[ https://issues.apache.org/jira/browse/FLUME-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15420954#comment-15420954 ] Denes Arvay commented on FLUME-2844: Hi [~sandywei], I hope you don't mind if I'm taking over on this patch. Please let me know if you have any concerns regarding this. > ChannelCounter of SpillableMemoryChannel doesn't register actually. > --- > > Key: FLUME-2844 > URL: https://issues.apache.org/jira/browse/FLUME-2844 > Project: Flume > Issue Type: Bug > Components: Channel >Affects Versions: v1.6.0 >Reporter: Daniel Wen >Assignee: Denes Arvay > Attachments: FLUME-2844-2.patch, FLUME-2844..patch > > > When using SpillableMemoryChannel, the values of all metrics of channel > component int monitoring system is zero.It's caused by a bug belowed. > SpillableMemoryChannel extends FileChannel,and each of them have a > ChannelCounter-Type field named of 'channelCounter'. FileChannel's > channelCounter filed is private, so it can not be override by > SpillableMemoryChannel. SpillableMemoryChannel's channelCounter is > initialized in configure method(Line 600 at SpillableMemoryChannel.java), > then super.configure() is called(Line 727 at SpillableMemoryChannel.java). So > FileChannel's channelCounter is also initialized(Line 267 at > SpillableMemoryChannel.java). > When SpillableMemoryChannel start(call start() method), it should and want to > register ChannelCounter. In start() method, SpillableMemoryChannel calls > super.start() (Line 768 at SpillableMemoryChannel.java), then FileChannel > calls channelCounter.start() to register channelCounter. But this > channelCounter is the one of FileChannle, not the one of > SpillableMemoryChannel! > This bug results that when using SpillableMemoryChannel, the values of all > metrics of channel component is zero. > Maybe the qualifier of FileChannel's channelCounter filed should be changed > to prtected, and remove the channelCounter filed of SpillableMemoryChannel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2973) Deadlock in hdfs sink
Denes Arvay created FLUME-2973:
--
Summary: Deadlock in hdfs sink
Key: FLUME-2973
URL: https://issues.apache.org/jira/browse/FLUME-2973
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.7.0
Reporter: Denes Arvay
Assignee: Denes Arvay
Priority: Critical
Automatic close of BucketWriters (when open file count reached
{{hdfs.maxOpenFiles}}) and the file rolling thread can end up in deadlock.
When creating a new {{BucketWriter}} in {{HDFSEventSink}} it locks
{{HDFSEventSink.sfWritersLock}} and the {{close()}} called in
{{HDFSEventSink.sfWritersLock.removeEldestEntry}} tries to lock the
{{BucketWriter}} instance.
On the other hand if the file is being rolled in
{{BucketWriter.close(boolean)}} it locks the {{BucketWriter}} instance first
and in the close callback it tries to lock the {{sfWritersLock}}.
The chances for this deadlock is higher when the {{hdfs.maxOpenFiles}}'s value
is low (1).
Script to reproduce:
https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
(put to
{{flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs}})
Deadlock usually occurs before ~30 iterations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2973) Deadlock in hdfs sink
[
https://issues.apache.org/jira/browse/FLUME-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2973:
---
Attachment: FLUME-2973.patch
> Deadlock in hdfs sink
> -
>
> Key: FLUME-2973
> URL: https://issues.apache.org/jira/browse/FLUME-2973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Critical
> Labels: hdfssink
> Attachments: FLUME-2973.patch
>
>
> Automatic close of BucketWriters (when open file count reached
> {{hdfs.maxOpenFiles}}) and the file rolling thread can end up in deadlock.
> When creating a new {{BucketWriter}} in {{HDFSEventSink}} it locks
> {{HDFSEventSink.sfWritersLock}} and the {{close()}} called in
> {{HDFSEventSink.sfWritersLock.removeEldestEntry}} tries to lock the
> {{BucketWriter}} instance.
> On the other hand if the file is being rolled in
> {{BucketWriter.close(boolean)}} it locks the {{BucketWriter}} instance first
> and in the close callback it tries to lock the {{sfWritersLock}}.
> The chances for this deadlock is higher when the {{hdfs.maxOpenFiles}}'s
> value is low (1).
> Script to reproduce:
> https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
> (put to
> {{flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs}})
> Deadlock usually occurs before ~30 iterations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2797: --- Attachment: FLUME-2797-5.patch Patch updated, thanks [~bessbd] for the review. > SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource > - > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch, FLUME-2797-2.patch, > FLUME-2797-3.patch, FLUME-2797-4.patch, FLUME-2797-5.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2976) Exception when JMS source tries to connect to a Weblogic server without authentication
Denes Arvay created FLUME-2976:
--
Summary: Exception when JMS source tries to connect to a Weblogic
server without authentication
Key: FLUME-2976
URL: https://issues.apache.org/jira/browse/FLUME-2976
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.6.0
Reporter: Denes Arvay
If no {{userName}} and {{passwordFile}} is set for the JMS source it sets the
password to {{Optional("")}} (see:
https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java#L127)
This leads to an exception in the weblogic jndi context implementation when
trying to connect to a weblogic jms server.
{noformat}
java.lang.IllegalArgumentException: The 'java.naming.security.principal'
property has not been specified
at weblogic.jndi.Environment.getSecurityUser(Environment.java:562)
at
weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:665)
at
weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:485)
at
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:373)
at weblogic.jndi.Environment.getContext(Environment.java:319)
at weblogic.jndi.Environment.getContext(Environment.java:288)
at
weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
at
javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at
javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
at javax.naming.InitialContext.init(InitialContext.java:242)
at javax.naming.InitialContext.(InitialContext.java:216)
at
org.apache.flume.source.jms.InitialContextFactory.create(InitialContextFactory.java:28)
at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:223)
{noformat}
Changing the above mentioned line to {{Optional.absent()}} fixes the issue.
[~brocknoland]: Is there any specific reason for setting the password to
{{Optional("")}} when there is no {{passwordFile}} set?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2797) SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
[ https://issues.apache.org/jira/browse/FLUME-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay updated FLUME-2797: --- Attachment: FLUME-2797-6.patch Thanks [~mpercy] for the review, I've uploaded a new patch. > SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource > - > > Key: FLUME-2797 > URL: https://issues.apache.org/jira/browse/FLUME-2797 > Project: Flume > Issue Type: Bug >Reporter: Otis Gospodnetic >Assignee: Denes Arvay >Priority: Minor > Fix For: v1.7.0 > > Attachments: FLUME-2797-0.patch, FLUME-2797-2.patch, > FLUME-2797-3.patch, FLUME-2797-4.patch, FLUME-2797-5.patch, FLUME-2797-6.patch > > > From the mailing list: > From Ashish: > Source uses an deprecated class. Can you please file a JIRA for this? > The fix is simple. In SyslogTcpSource Line#61, replace the > CounterGroup usage with SourceCounter and make related changes in > code. You can refer other Sources for details. Same is applicable for > SyslogUDPSource. > From Hari: > I think the Syslog TCP source should be deprecated in favor of the Multiport > Syslog Source - that is more stable and gives better performance too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-2988) Kafka Sink metrics missing eventDrainAttemptCount
Denes Arvay created FLUME-2988:
--
Summary: Kafka Sink metrics missing eventDrainAttemptCount
Key: FLUME-2988
URL: https://issues.apache.org/jira/browse/FLUME-2988
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Reporter: Denes Arvay
Priority: Minor
{{eventDrainAttemptCount}} doesn't get incremented in Kafka Sink, only the
{{eventDrainSuccessCount}}
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L210)
resulting in misleading statistics (e.g. when calculating {{attemptCount -
successCount}})
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-2989) Kafka Channel metrics missing eventTakeAttemptCount and eventPutAttemptCount
Denes Arvay created FLUME-2989:
--
Summary: Kafka Channel metrics missing eventTakeAttemptCount and
eventPutAttemptCount
Key: FLUME-2989
URL: https://issues.apache.org/jira/browse/FLUME-2989
Project: Flume
Issue Type: Bug
Components: Kafka Channel
Reporter: Denes Arvay
Priority: Minor
{{eventTakeAttemptCount}} and {{eventPutAttemptCount}} counters don't get
incremented in Kafka Channel
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2911) Add includePattern option in SpoolDirectorySource configuration
[
https://issues.apache.org/jira/browse/FLUME-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470576#comment-15470576
]
Denes Arvay commented on FLUME-2911:
Hi [~arota], I've also commented on your pull request. Please correct those
issues (nothing serious) and I can +1 it too, then it'll be committed by a
Flume committer. Thanks.
> Add includePattern option in SpoolDirectorySource configuration
> ---
>
> Key: FLUME-2911
> URL: https://issues.apache.org/jira/browse/FLUME-2911
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Affects Versions: notrack, v1.6.0, v1.7.0
>Reporter: Andrea Rota
> Labels: features
> Attachments: FLUME-2911.patch
>
>
> Current implementation of SpoolDirectorySource does not allow users to
> specify a regex pattern to select which files should be monitored. Instead,
> the current implementation allows users to specify which should *not*
> monitored, via the ignorePattern parameter.
> I implemented the feature, allowing users to specify the include pattern as
> {{a1.sources.src-1.includePattern=^foo.*$}} (includes all the files that
> starts in "foo").
> By default, the includePattern regex is set to {{^.*$}} (all files). Include
> and exclude patterns can be used at same time and the results are combined.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-2991) ExecSource command execution starts before starting the sourceCounter
Denes Arvay created FLUME-2991: -- Summary: ExecSource command execution starts before starting the sourceCounter Key: FLUME-2991 URL: https://issues.apache.org/jira/browse/FLUME-2991 Project: Flume Issue Type: Bug Components: Sinks+Sources Reporter: Denes Arvay Assignee: Denes Arvay It causes invalid data to appear in JMX monitoring and also makes `TestExecSource.testMonitoredCounterGroup` failing/flaky. starting the `sourceCounter` before submitting the `ExecRunnable` to the `executor` solves the problem. https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2991) ExecSource command execution starts before starting the sourceCounter
[
https://issues.apache.org/jira/browse/FLUME-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2991:
---
Description:
It causes invalid data to appear in JMX monitoring and also makes
{{TestExecSource.testMonitoredCounterGroup}} failing/flaky.
starting the `sourceCounter` before submitting the `ExecRunnable` to the
`executor` solves the problem.
https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183
was:
It causes invalid data to appear in JMX monitoring and also makes
`TestExecSource.testMonitoredCounterGroup` failing/flaky.
starting the `sourceCounter` before submitting the `ExecRunnable` to the
`executor` solves the problem.
https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183
> ExecSource command execution starts before starting the sourceCounter
> -
>
> Key: FLUME-2991
> URL: https://issues.apache.org/jira/browse/FLUME-2991
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> It causes invalid data to appear in JMX monitoring and also makes
> {{TestExecSource.testMonitoredCounterGroup}} failing/flaky.
> starting the `sourceCounter` before submitting the `ExecRunnable` to the
> `executor` solves the problem.
> https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2991) ExecSource command execution starts before starting the sourceCounter
[
https://issues.apache.org/jira/browse/FLUME-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2991:
---
Description:
It causes invalid data to appear in JMX monitoring and also makes
{{TestExecSource.testMonitoredCounterGroup}} failing/flaky.
starting the {{sourceCounter}} before submitting the {{ExecRunnable}} to the
{{executor}} solves the problem.
https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183
was:
It causes invalid data to appear in JMX monitoring and also makes
{{TestExecSource.testMonitoredCounterGroup}} failing/flaky.
starting the `sourceCounter` before submitting the `ExecRunnable` to the
`executor` solves the problem.
https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183
> ExecSource command execution starts before starting the sourceCounter
> -
>
> Key: FLUME-2991
> URL: https://issues.apache.org/jira/browse/FLUME-2991
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> It causes invalid data to appear in JMX monitoring and also makes
> {{TestExecSource.testMonitoredCounterGroup}} failing/flaky.
> starting the {{sourceCounter}} before submitting the {{ExecRunnable}} to the
> {{executor}} solves the problem.
> https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2991) ExecSource command execution starts before starting the sourceCounter
[
https://issues.apache.org/jira/browse/FLUME-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2991:
---
Attachment: FLUME-2991.patch
> ExecSource command execution starts before starting the sourceCounter
> -
>
> Key: FLUME-2991
> URL: https://issues.apache.org/jira/browse/FLUME-2991
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Attachments: FLUME-2991.patch
>
>
> It causes invalid data to appear in JMX monitoring and also makes
> {{TestExecSource.testMonitoredCounterGroup}} failing/flaky.
> starting the {{sourceCounter}} before submitting the {{ExecRunnable}} to the
> {{executor}} solves the problem.
> https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java#L176-L183
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2976) Exception when JMS source tries to connect to a Weblogic server without authentication
[
https://issues.apache.org/jira/browse/FLUME-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay reassigned FLUME-2976:
--
Assignee: Denes Arvay
> Exception when JMS source tries to connect to a Weblogic server without
> authentication
> --
>
> Key: FLUME-2976
> URL: https://issues.apache.org/jira/browse/FLUME-2976
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Labels: jms
>
> If no {{userName}} and {{passwordFile}} is set for the JMS source it sets the
> password to {{Optional("")}} (see:
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java#L127)
> This leads to an exception in the weblogic jndi context implementation when
> trying to connect to a weblogic jms server.
> {noformat}
> java.lang.IllegalArgumentException: The 'java.naming.security.principal'
> property has not been specified
> at weblogic.jndi.Environment.getSecurityUser(Environment.java:562)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:665)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:485)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:373)
> at weblogic.jndi.Environment.getContext(Environment.java:319)
> at weblogic.jndi.Environment.getContext(Environment.java:288)
> at
> weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
> at
> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
> at
> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
> at javax.naming.InitialContext.init(InitialContext.java:242)
> at javax.naming.InitialContext.(InitialContext.java:216)
> at
> org.apache.flume.source.jms.InitialContextFactory.create(InitialContextFactory.java:28)
> at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:223)
> {noformat}
> Changing the above mentioned line to {{Optional.absent()}} fixes the issue.
> [~brocknoland]: Is there any specific reason for setting the password to
> {{Optional("")}} when there is no {{passwordFile}} set?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLUME-2976) Exception when JMS source tries to connect to a Weblogic server without authentication
[
https://issues.apache.org/jira/browse/FLUME-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2976:
---
Attachment: FLUME-2976.patch
> Exception when JMS source tries to connect to a Weblogic server without
> authentication
> --
>
> Key: FLUME-2976
> URL: https://issues.apache.org/jira/browse/FLUME-2976
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Labels: jms
> Attachments: FLUME-2976.patch
>
>
> If no {{userName}} and {{passwordFile}} is set for the JMS source it sets the
> password to {{Optional("")}} (see:
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java#L127)
> This leads to an exception in the weblogic jndi context implementation when
> trying to connect to a weblogic jms server.
> {noformat}
> java.lang.IllegalArgumentException: The 'java.naming.security.principal'
> property has not been specified
> at weblogic.jndi.Environment.getSecurityUser(Environment.java:562)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:665)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:485)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:373)
> at weblogic.jndi.Environment.getContext(Environment.java:319)
> at weblogic.jndi.Environment.getContext(Environment.java:288)
> at
> weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
> at
> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
> at
> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
> at javax.naming.InitialContext.init(InitialContext.java:242)
> at javax.naming.InitialContext.(InitialContext.java:216)
> at
> org.apache.flume.source.jms.InitialContextFactory.create(InitialContextFactory.java:28)
> at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:223)
> {noformat}
> Changing the above mentioned line to {{Optional.absent()}} fixes the issue.
> [~brocknoland]: Is there any specific reason for setting the password to
> {{Optional("")}} when there is no {{passwordFile}} set?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2994) flume-taildir-source: support for windows
[ https://issues.apache.org/jira/browse/FLUME-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509797#comment-15509797 ] Denes Arvay commented on FLUME-2994: Hi [~jkushmaul], I'm assigning this to you. Thanks for reporting the issue and submitting the patch, I'll review and validate your solution on a windows box soon. > flume-taildir-source: support for windows > - > > Key: FLUME-2994 > URL: https://issues.apache.org/jira/browse/FLUME-2994 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources, Windows >Affects Versions: v1.7.0 >Reporter: Jason Kushmaul >Priority: Trivial > Fix For: v1.7.0 > > Attachments: flume-2994-1.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > The current implementation of flume-taildir-source does not support windows. > The only reason for this from what I can see is a simple call to > Files.getAttribute(file.toPath(), "unix:ino"); > I've tested an equivalent for windows (which of course does not work on > non-windows). With an OS switch we should be able to identify a file > independent of file name on either system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2994) flume-taildir-source: support for windows
[ https://issues.apache.org/jira/browse/FLUME-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509844#comment-15509844 ] Denes Arvay commented on FLUME-2994: Hm, apparently you have to have contributor privileges to be an assignee of a task. [~mpercy], could you please help us? Thanks. > flume-taildir-source: support for windows > - > > Key: FLUME-2994 > URL: https://issues.apache.org/jira/browse/FLUME-2994 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources, Windows >Affects Versions: v1.7.0 >Reporter: Jason Kushmaul >Priority: Trivial > Fix For: v1.7.0 > > Attachments: FLUME-2994-2.patch, flume-2994-1.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > The current implementation of flume-taildir-source does not support windows. > The only reason for this from what I can see is a simple call to > Files.getAttribute(file.toPath(), "unix:ino"); > I've tested an equivalent for windows (which of course does not work on > non-windows). With an OS switch we should be able to identify a file > independent of file name on either system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2994) flume-taildir-source: support for windows
[ https://issues.apache.org/jira/browse/FLUME-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509846#comment-15509846 ] Denes Arvay commented on FLUME-2994: Yes, please attach the conf file too for easier validation. Thanks. > flume-taildir-source: support for windows > - > > Key: FLUME-2994 > URL: https://issues.apache.org/jira/browse/FLUME-2994 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources, Windows >Affects Versions: v1.7.0 >Reporter: Jason Kushmaul >Priority: Trivial > Fix For: v1.7.0 > > Attachments: FLUME-2994-2.patch, flume-2994-1.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > The current implementation of flume-taildir-source does not support windows. > The only reason for this from what I can see is a simple call to > Files.getAttribute(file.toPath(), "unix:ino"); > I've tested an equivalent for windows (which of course does not work on > non-windows). With an OS switch we should be able to identify a file > independent of file name on either system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2966) NULL text in a TextMessage from a JMS source in Flume can lead to NPE
[
https://issues.apache.org/jira/browse/FLUME-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522304#comment-15522304
]
Denes Arvay commented on FLUME-2966:
[~sahuja], thanks for the patch, LGTM, +1
> NULL text in a TextMessage from a JMS source in Flume can lead to NPE
> -
>
> Key: FLUME-2966
> URL: https://issues.apache.org/jira/browse/FLUME-2966
> Project: Flume
> Issue Type: Bug
>Affects Versions: v1.5.0
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
> Attachments: App.java, FLUME-2966-0.patch, FLUME-2966-1.patch
>
>
> Code at
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java#L103
> does not check for a NULL text in a TextMessage from a Flume JMS source.
> This can lead to a NullPointerException here:
> {code}textMessage.getText().getBytes(charset){code} while trying to
> de-reference a null text from the textmessage.
> We should probably skip these like the NULL Objects in the ObjectMessage just
> below at:
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/DefaultJMSMessageConverter.java#L107.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2794) Flume 1.6 HBase 1.12 java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
[ https://issues.apache.org/jira/browse/FLUME-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay reassigned FLUME-2794: -- Assignee: Denes Arvay > Flume 1.6 HBase 1.12 java.lang.NoSuchMethodError: > org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V > -- > > Key: FLUME-2794 > URL: https://issues.apache.org/jira/browse/FLUME-2794 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: v1.6.0 >Reporter: Emmanuel Leroy >Assignee: Denes Arvay > > with Hbase 1.1.2 and Flume 1.6 > getting this error: > ase_flume-9k2a4-1441998442358-ec273f9a-0-10152166], Starting > 2015-09-11 19:07:28,037 (SinkRunner-PollingRunner-DefaultSinkProcessor) > [ERROR - org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:351)] > Failed to commit transaction.Transaction rolled back. > java.lang.NoSuchMethodError: > org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V > at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:377) > at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:372) > at > org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) > at > org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:372) > at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:342) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535465#comment-15535465 ] Denes Arvay commented on FLUME-2998: I agree with [~bessbd]'s comments. Regarding to the default value: I'd vote for using the Long.MAX_VALUE. Yes, it can't be set as value in this form but it's a lot cleaner than the exact number. > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2998) Add missing configuration parameter to SequenceSource docs
[ https://issues.apache.org/jira/browse/FLUME-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535534#comment-15535534 ] Denes Arvay commented on FLUME-2998: +1, thanks [~sati] for the patch > Add missing configuration parameter to SequenceSource docs > -- > > Key: FLUME-2998 > URL: https://issues.apache.org/jira/browse/FLUME-2998 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: v1.7.0 >Reporter: Attila Simon >Assignee: Attila Simon > Fix For: v1.7.0 > > Attachments: FLUME-2998.patch > > > totalEvents can be specified -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLUME-3003) testSourceCounter in TestSyslogUdpSource is flaky
Denes Arvay created FLUME-3003: -- Summary: testSourceCounter in TestSyslogUdpSource is flaky Key: FLUME-3003 URL: https://issues.apache.org/jira/browse/FLUME-3003 Project: Flume Issue Type: Test Components: Sinks+Sources Affects Versions: v1.6.0 Reporter: Denes Arvay Assignee: Denes Arvay Fix For: v1.7.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2994) flume-taildir-source: support for windows
[
https://issues.apache.org/jira/browse/FLUME-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548703#comment-15548703
]
Denes Arvay commented on FLUME-2994:
We only use the inode long value in the {{tailFiles}} map. If we changed the
{{tailFiles}} to {{Map}} then we could use it regardless of
the environment. So there would be no need for platform check and - as it's not
windows-only - I wouldn't consider that change as proliferation of a windows
workaround. As per my understanding {{FileKey}} is a unique identifier for a
file, so in our usecase it can replace the current inode value.
> flume-taildir-source: support for windows
> -
>
> Key: FLUME-2994
> URL: https://issues.apache.org/jira/browse/FLUME-2994
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources, Windows
>Affects Versions: v1.7.0
>Reporter: Jason Kushmaul
>Assignee: Jason Kushmaul
>Priority: Trivial
> Fix For: v1.7.0
>
> Attachments: FLUME-2994-2.patch, taildir-mac.conf, taildir-win8.1.conf
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The current implementation of flume-taildir-source does not support windows.
> The only reason for this from what I can see is a simple call to
> Files.getAttribute(file.toPath(), "unix:ino");
> I've tested an equivalent for windows (which of course does not work on
> non-windows). With an OS switch we should be able to identify a file
> independent of file name on either system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2994) flume-taildir-source: support for windows
[
https://issues.apache.org/jira/browse/FLUME-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15548781#comment-15548781
]
Denes Arvay commented on FLUME-2994:
If we choose to use the {{FileKey}} as key then it's not needed to check the
uniqueness of its {{hashCode}}.
> flume-taildir-source: support for windows
> -
>
> Key: FLUME-2994
> URL: https://issues.apache.org/jira/browse/FLUME-2994
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources, Windows
>Affects Versions: v1.7.0
>Reporter: Jason Kushmaul
>Assignee: Jason Kushmaul
>Priority: Trivial
> Fix For: v1.7.0
>
> Attachments: FLUME-2994-2.patch, taildir-mac.conf, taildir-win8.1.conf
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> The current implementation of flume-taildir-source does not support windows.
> The only reason for this from what I can see is a simple call to
> Files.getAttribute(file.toPath(), "unix:ino");
> I've tested an equivalent for windows (which of course does not work on
> non-windows). With an OS switch we should be able to identify a file
> independent of file name on either system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2857) Kafka Source/Channel/Sink does not restore default values when live update config
[
https://issues.apache.org/jira/browse/FLUME-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612227#comment-15612227
]
Denes Arvay commented on FLUME-2857:
Actually it's not silent, the test fails due to the exception.
{noformat}
kafka.admin.AdminOperationException: replication factor: 1 larger than
available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:77)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:236)
at kafka.admin.AdminUtils.createTopic(AdminUtils.scala)
at
org.apache.flume.source.kafka.KafkaSourceEmbeddedKafka.createTopic(KafkaSourceEmbeddedKafka.java:134)
at
org.apache.flume.source.kafka.TestKafkaSource.checkDefaultSettingsOnRestart(TestKafkaSource.java:645)
at
org.apache.flume.source.kafka.TestKafkaSource.testDefaultSettingsOnReConfigure(TestKafkaSource.java:632)
...
{noformat}
I agree with Donat, I also think that there is no need for starting the whole
kafka source, calling two {{configure}}s with different context should be
enough, as the context's properties are copied to the {{kafkaProps}} in
{{KafkaSource}}, which is checked by the test. We'd hit two birds with one
stone by removing the unneeded lines: the test would be simpler and the above
mentioned issue wouldn't come up.
> Kafka Source/Channel/Sink does not restore default values when live update
> config
> -
>
> Key: FLUME-2857
> URL: https://issues.apache.org/jira/browse/FLUME-2857
> Project: Flume
> Issue Type: Bug
> Components: Channel
>Reporter: Tristan Stevens
>Assignee: Tristan Stevens
> Attachments: FLUME-2857.patch
>
>
> Been using the following config:
> {noformat}
> tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel
> tier1.channels.channel1.capacity = 1
> tier1.channels.channel1.transactionCapacity = 1
> tier1.channels.channel1.brokerList =
> 10.0.0.64:9092,10.0.0.65:9092,10.0.0.66:9092
> tier1.channels.channel1.topic = flume.aggregator.channel
> tier1.channels.channel1.zookeeperConnect = 10.0.0.64:2181
> tier1.channels.channel1.kafka.producer.type=async
> tier1.channels.channel1.kafka.batch.num.messages=200
> {noformat}
> If I remove the producer.type and batch.num.messages (i.e. restore them to
> default values) when doing a live update config the new values do not take
> effect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-2812) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded
[ https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay reassigned FLUME-2812: -- Assignee: Denes Arvay > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" > java.lang.Error: Maximum permit count exceeded > -- > > Key: FLUME-2812 > URL: https://issues.apache.org/jira/browse/FLUME-2812 > Project: Flume > Issue Type: Bug > Components: Channel, Sinks+Sources >Affects Versions: v1.6.0 > Environment: **OS INFO** > CentOS release 6.6 (Final) > Kernel \r on an \m > **JAVA INFO** > java version "1.8.0_40" > Java(TM) SE Runtime Environment (build 1.8.0_40-b26) > Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode) >Reporter: Rollin Crittendon >Assignee: Denes Arvay >Priority: Critical > > We are finding that around after an hour or so of heavy processing of Flume > data in an agent we are getting the following exception. This is after > processing about 5-7 k lines/second during that time. > The configuration of this agent is using a Kafka source, the one that comes > with 1.6.0. > It is also using a Memory channel, and a Thrift sink. > === > Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" > java.lang.Error: Maximum permit count exceeded > at > java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341) > at java.util.concurrent.Semaphore.release(Semaphore.java:609) > at > org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147) > at > org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151) > at > org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379) > at > org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > === > The above error is from standard error when running the Flume agent. The > effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread > disappears from the agent, this can be seen on a JMX console. > For us, this means that the Flume agent needs to get restarted. It is an > error that is terminal in that instance of the Java process due to the thread > disappearing as a result. > It sounds like something in JDK 7+ got stricter?! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2812) Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded
[
https://issues.apache.org/jira/browse/FLUME-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15615773#comment-15615773
]
Denes Arvay commented on FLUME-2812:
I have investigated this issue and the cause is that
{{bytesRemaining.release(putByteCounter)}} gets called in
{{MemoryTransaction.doRollback}}
(https://github.com/apache/flume/blob/trunk/flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java#L174)
while acquire is called only in {{doCommit}}. This results in semaphore leak
and the number of permits in the semaphore eventually exceeds
{{Integer.MAX_VALUE}} and {{Semaphore.release()}} throws this error.
I think this bug was introduced unintentionally when the {{bytesRemaining}}
semaphore handling was refactored and moved to {{doCommit}} from {{doPut}} in
FLUME-2233.
If acquire is called in the {{doPut}} then release should happen in
{{doRollback}} but since it has been moved to {{doCommit}} the release should
have been deleted.
GitHub Pull Request for this ticket: https://github.com/apache/flume/pull/83
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.Error: Maximum permit count exceeded
> --
>
> Key: FLUME-2812
> URL: https://issues.apache.org/jira/browse/FLUME-2812
> Project: Flume
> Issue Type: Bug
> Components: Channel, Sinks+Sources
>Affects Versions: v1.6.0
> Environment: **OS INFO**
> CentOS release 6.6 (Final)
> Kernel \r on an \m
> **JAVA INFO**
> java version "1.8.0_40"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
>Reporter: Rollin Crittendon
>Assignee: Denes Arvay
>Priority: Critical
>
> We are finding that around after an hour or so of heavy processing of Flume
> data in an agent we are getting the following exception. This is after
> processing about 5-7 k lines/second during that time.
> The configuration of this agent is using a Kafka source, the one that comes
> with 1.6.0.
> It is also using a Memory channel, and a Thrift sink.
> ===
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.Error: Maximum permit count exceeded
> at
> java.util.concurrent.Semaphore$Sync.tryReleaseShared(Semaphore.java:192)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at java.util.concurrent.Semaphore.release(Semaphore.java:609)
> at
> org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:147)
> at
> org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
> at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:379)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> ===
> The above error is from standard error when running the Flume agent. The
> effect is that the "SinkRunner-PollingRunner-DefaultSinkProcessor" thread
> disappears from the agent, this can be seen on a JMX console.
> For us, this means that the Flume agent needs to get restarted. It is an
> error that is terminal in that instance of the Java process due to the thread
> disappearing as a result.
> It sounds like something in JDK 7+ got stricter?!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2786) It will enter a deadlock state when modify the conf file before I stop flume-ng
[ https://issues.apache.org/jira/browse/FLUME-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15628572#comment-15628572 ] Denes Arvay commented on FLUME-2786: Hi [~adrianjian], thanks for the patch. What do you think, is it possible to create a unit test for this? > It will enter a deadlock state when modify the conf file before I stop > flume-ng > > > Key: FLUME-2786 > URL: https://issues.apache.org/jira/browse/FLUME-2786 > Project: Flume > Issue Type: Bug > Components: Master >Affects Versions: v1.6.0 >Reporter: godfrey he >Priority: Blocker > Attachments: flume-2786-v1.6.0.patch > > > When modify the conf file，and then I stop the flume-ng, It will enter a > deadlock state. > jstack result: > "agent-shutdown-hook" prio=10 tid=0x7f2e26419800 nid=0x333ae waiting on > condition [0x42c16000] >java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0xeaff3df8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) > at > java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468) > at > java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:635) > at > org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop(PollingPropertiesFileConfigurationProvider.java:87) > at > org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:106) > - locked <0xeaf2daa0> (a > org.apache.flume.lifecycle.LifecycleSupervisor) > at org.apache.flume.node.Application.stop(Application.java:93) > - locked <0xeaf3c580> (a org.apache.flume.node.Application) > at org.apache.flume.node.Application$1.run(Application.java:348) > "conf-file-poller-0" prio=10 tid=0x7f2e2e8cd000 nid=0x21819 waiting for > monitor entry [0x41e3f000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.flume.node.Application.handleConfigurationEvent(Application.java:88) > - waiting to lock <0xeaf3c580> (a > org.apache.flume.node.Application) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2782) Extension-friendly HttpSource
[
https://issues.apache.org/jira/browse/FLUME-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629355#comment-15629355
]
Denes Arvay commented on FLUME-2782:
I'm a bit concerned about exposing the whole {{FlumeHTTPServlet}}, so what
about moving the {{customizeServletResponse()}} to the {{HTTPSource}} class
itself? The {{FlumeHTTPServlet}} is able to access it as it's a non-static
inner class. We would hit two birds with one stone: the {{FlumeHTTPServlet}}
wouldn't be exposed and the custom {{HTTPSource}} implementation would be
simpler: there wouldn't be need for creating a subclass for the
{{FlumeHTTPServlet}} as both of the customization methods could be overwritten
in the {{HTTPSource}} subclass.
Btw, what are the potential use-cases for this? I'm just wondering whether the
same result can be achieved by http servlet filters. Not sure though.
Regarding to the PR/Reviewboard: I don't know the historical reasons of
preferring Reviewboard over GitHub PRs, but now we are moving towards GitHub as
it seems to be more convenient to use.
> Extension-friendly HttpSource
> -
>
> Key: FLUME-2782
> URL: https://issues.apache.org/jira/browse/FLUME-2782
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Affects Versions: v1.6.0, v1.5.2
>Reporter: Luca Rosellini
> Labels: HTTP, Source
> Attachments: extensible-flume-HttpSource.patch
>
>
> This patch allows third-party to developers extend HttpSource behavior
> easily, without having to reimplement the whole thing.
> More specifically:
> * Adds an hook to customize {{org.mortbay.jetty.servlet.Context}}.
> * Delegates the HttpServlet instance creation to a protected factory method.
> * Changes FlumeHttpServlet visibility to {{protected}}, to let third party
> developers extend from it.
> * Adds an hook in {{FlumeHTTPServlet}} to customize {{HttpServletResponse}}
> before it is flushed.
> We developed this patch because we had to add a custom servlet filter to the
> Jetty context.
> Having this patch applied, a third-party developer could easily extend
> {{HttpSource}} like this:
> {code:title=CustomHTTPSource.java|borderStyle=solid}
> public class CustomHTTPSource extends HTTPSource {
> @Override
> protected void customizeServletContext(Context context) {
> super.customizeServletContext(context);
> context.addFilter(MyCustomHttpServletFilter.class,"/*",0);
> }
> @Override
> protected HttpServlet getServlet() {
> return new KeedioFlumeHTTPServlet();
> }
> protected class KeedioFlumeHTTPServlet extends FlumeHTTPServlet{
> @Override
> protected void customizeServletResponse(HttpServletRequest request,
> HttpServletResponse response) {
> super.customizeServletResponse(request, response);
> response.addHeader("Accept-Encoding","...");
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2782) Extension-friendly HttpSource
[
https://issues.apache.org/jira/browse/FLUME-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629395#comment-15629395
]
Denes Arvay commented on FLUME-2782:
I see, [~jmb], that's a valid example for subsclassing FlumeHTTPServlet, thanks.
> Extension-friendly HttpSource
> -
>
> Key: FLUME-2782
> URL: https://issues.apache.org/jira/browse/FLUME-2782
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
>Affects Versions: v1.6.0, v1.5.2
>Reporter: Luca Rosellini
> Labels: HTTP, Source
> Attachments: extensible-flume-HttpSource.patch
>
>
> This patch allows third-party to developers extend HttpSource behavior
> easily, without having to reimplement the whole thing.
> More specifically:
> * Adds an hook to customize {{org.mortbay.jetty.servlet.Context}}.
> * Delegates the HttpServlet instance creation to a protected factory method.
> * Changes FlumeHttpServlet visibility to {{protected}}, to let third party
> developers extend from it.
> * Adds an hook in {{FlumeHTTPServlet}} to customize {{HttpServletResponse}}
> before it is flushed.
> We developed this patch because we had to add a custom servlet filter to the
> Jetty context.
> Having this patch applied, a third-party developer could easily extend
> {{HttpSource}} like this:
> {code:title=CustomHTTPSource.java|borderStyle=solid}
> public class CustomHTTPSource extends HTTPSource {
> @Override
> protected void customizeServletContext(Context context) {
> super.customizeServletContext(context);
> context.addFilter(MyCustomHttpServletFilter.class,"/*",0);
> }
> @Override
> protected HttpServlet getServlet() {
> return new KeedioFlumeHTTPServlet();
> }
> protected class KeedioFlumeHTTPServlet extends FlumeHTTPServlet{
> @Override
> protected void customizeServletResponse(HttpServletRequest request,
> HttpServletResponse response) {
> super.customizeServletResponse(request, response);
> response.addHeader("Accept-Encoding","...");
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-2171) Add Interceptor to remove headers from event
[ https://issues.apache.org/jira/browse/FLUME-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653726#comment-15653726 ] Denes Arvay commented on FLUME-2171: [~bessbd], could you please have a look on this ticket? There's only one minor comment on the reviewboard, could you please fix and commit it? Thanks in advance. > Add Interceptor to remove headers from event > > > Key: FLUME-2171 > URL: https://issues.apache.org/jira/browse/FLUME-2171 > Project: Flume > Issue Type: New Feature > Components: Easy >Affects Versions: v1.4.0 >Reporter: Gabriel Commeau >Assignee: Bessenyei Balázs Donát > Labels: reviewboard-missing > Attachments: FLUME-2171-0.patch > > > I found Flume OG's decorators to handle event headers useful and some to be > missing from Flume NG. More specifically, we could have an interceptor to > remove headers from an event. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLUME-2988) Kafka Sink metrics missing eventDrainAttemptCount
[
https://issues.apache.org/jira/browse/FLUME-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15663241#comment-15663241
]
Denes Arvay commented on FLUME-2988:
Hi [~udai], thanks for the patch. The fix looks good for me, one comment: the
{{Long.valueOf}} is not needed here, simply passing the {{kafkaFutures.size()}}
to the method is enough.
> Kafka Sink metrics missing eventDrainAttemptCount
> -
>
> Key: FLUME-2988
> URL: https://issues.apache.org/jira/browse/FLUME-2988
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Reporter: Denes Arvay
>Assignee: Udai Kiran Potluri
>Priority: Minor
> Attachments: FLUME-2988.1.patch
>
>
> {{eventDrainAttemptCount}} doesn't get incremented in Kafka Sink, only the
> {{eventDrainSuccessCount}}
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java#L210)
> resulting in misleading statistics (e.g. when calculating {{attemptCount -
> successCount}})
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-3025) Expose FileChannel.open on JMX
Denes Arvay created FLUME-3025:
--
Summary: Expose FileChannel.open on JMX
Key: FLUME-3025
URL: https://issues.apache.org/jira/browse/FLUME-3025
Project: Flume
Issue Type: Improvement
Reporter: Denes Arvay
We've encountered an issue several times when the {{FileChannel}} wasn't able
to start because of some startup-time exception.
In this case {{FileChannel.open}} stays false {{false}}, exposing this field on
JMX would make it easier to detect when the file channel hasn't been started /
or has been stopped because of some reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-3025) Expose FileChannel.open on JMX
[
https://issues.apache.org/jira/browse/FLUME-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay reassigned FLUME-3025:
--
Assignee: Denes Arvay
> Expose FileChannel.open on JMX
> --
>
> Key: FLUME-3025
> URL: https://issues.apache.org/jira/browse/FLUME-3025
> Project: Flume
> Issue Type: Improvement
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> We've encountered an issue several times when the {{FileChannel}} wasn't able
> to start because of some startup-time exception.
> In this case {{FileChannel.open}} stays false {{false}}, exposing this field
> on JMX would make it easier to detect when the file channel hasn't been
> started / or has been stopped because of some reason.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLUME-2053) TestExecSource.testMonitoredCounterGroup fails
[ https://issues.apache.org/jira/browse/FLUME-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay resolved FLUME-2053. Resolution: Duplicate > TestExecSource.testMonitoredCounterGroup fails > -- > > Key: FLUME-2053 > URL: https://issues.apache.org/jira/browse/FLUME-2053 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: v1.4.0, v1.5.1 > Environment: Ubuntu 13.04 > Java(TM) SE Runtime Environment (build 1.7.0_13-b20) > Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode) > Maven 3.0.5 >Reporter: Hans Uhlig >Assignee: Ashish Paliwal > > Failed tests: > testMonitoredCounterGroup(org.apache.flume.source.TestExecSource): Expected > Value: 5 expected:<[5]> but was:<[0]> -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLUME-2976) Exception when JMS source tries to connect to a Weblogic server without authentication
[
https://issues.apache.org/jira/browse/FLUME-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-2976:
---
Attachment: FLUME-2976-2.patch
> Exception when JMS source tries to connect to a Weblogic server without
> authentication
> --
>
> Key: FLUME-2976
> URL: https://issues.apache.org/jira/browse/FLUME-2976
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: v1.6.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Labels: jms
> Attachments: FLUME-2976-2.patch, FLUME-2976.patch
>
>
> If no {{userName}} and {{passwordFile}} is set for the JMS source it sets the
> password to {{Optional("")}} (see:
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java#L127)
> This leads to an exception in the weblogic jndi context implementation when
> trying to connect to a weblogic jms server.
> {noformat}
> java.lang.IllegalArgumentException: The 'java.naming.security.principal'
> property has not been specified
> at weblogic.jndi.Environment.getSecurityUser(Environment.java:562)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:665)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:485)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:373)
> at weblogic.jndi.Environment.getContext(Environment.java:319)
> at weblogic.jndi.Environment.getContext(Environment.java:288)
> at
> weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
> at
> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
> at
> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
> at javax.naming.InitialContext.init(InitialContext.java:242)
> at javax.naming.InitialContext.(InitialContext.java:216)
> at
> org.apache.flume.source.jms.InitialContextFactory.create(InitialContextFactory.java:28)
> at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:223)
> {noformat}
> Changing the above mentioned line to {{Optional.absent()}} fixes the issue.
> [~brocknoland]: Is there any specific reason for setting the password to
> {{Optional("")}} when there is no {{passwordFile}} set?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLUME-3049) Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
Denes Arvay created FLUME-3049:
--
Summary: Wrapping the exception into SecurityException in
UGIExecutor.execute hides the original one
Key: FLUME-3049
URL: https://issues.apache.org/jira/browse/FLUME-3049
Project: Flume
Issue Type: Bug
Reporter: Denes Arvay
see:
https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
This has unexpected side effects as the callers try to catch the wrapped
exception, for example in {{BucketWriter.append()}}:
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
I don't know the original intend behind this wrapping, [~jrufus] or
[~hshreedharan], do you happen to remember? (You were involved in the original
implementation in FLUME-2631)
Right now I don't see any problem in removing this and letting the original
exception to propagate as the {{org.apache.flume.auth.SecurityException}}
doesn't appear anywhere in the public interface.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLUME-3049) Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
[
https://issues.apache.org/jira/browse/FLUME-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay reassigned FLUME-3049:
--
Assignee: Denes Arvay
> Wrapping the exception into SecurityException in UGIExecutor.execute hides
> the original one
> ---
>
> Key: FLUME-3049
> URL: https://issues.apache.org/jira/browse/FLUME-3049
> Project: Flume
> Issue Type: Bug
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> see:
> https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
> This has unexpected side effects as the callers try to catch the wrapped
> exception, for example in {{BucketWriter.append()}}:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
> I don't know the original intend behind this wrapping, [~jrufus] or
> [~hshreedharan], do you happen to remember? (You were involved in the
> original implementation in FLUME-2631)
> Right now I don't see any problem in removing this and letting the original
> exception to propagate as the {{org.apache.flume.auth.SecurityException}}
> doesn't appear anywhere in the public interface.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLUME-3049) Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
[
https://issues.apache.org/jira/browse/FLUME-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15842561#comment-15842561
]
Denes Arvay commented on FLUME-3049:
Thanks [~hshreedharan], I have opened a pull request for this change.
> Wrapping the exception into SecurityException in UGIExecutor.execute hides
> the original one
> ---
>
> Key: FLUME-3049
> URL: https://issues.apache.org/jira/browse/FLUME-3049
> Project: Flume
> Issue Type: Bug
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> see:
> https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
> This has unexpected side effects as the callers try to catch the wrapped
> exception, for example in {{BucketWriter.append()}}:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
> I don't know the original intend behind this wrapping, [~jrufus] or
> [~hshreedharan], do you happen to remember? (You were involved in the
> original implementation in FLUME-2631)
> Right now I don't see any problem in removing this and letting the original
> exception to propagate as the {{org.apache.flume.auth.SecurityException}}
> doesn't appear anywhere in the public interface.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] (FLUME-3049) Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
[
https://issues.apache.org/jira/browse/FLUME-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-3049:
---
Description:
see:
https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
This has unexpected side effects as the callers try to catch the wrapped
exception, for example in {{BucketWriter.append()}}:
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
Here IOException is considered as non-transient failure thus the {{close()}} is
called, but when the original exception is wrapped into {{SecurityException}}
it doesn't trigger the close of the file.
Similarly in {{HDFSEventSink.process()}} method the `IOException` is handled in
a different way than any other exception. It might come from
{{BucketWriter.append()}} or {{BucketWriter.flush()}} for example, and both of
them invoke the hdfs call via a {{PrivilegedExecutor}} instance which might be
the problematic {{UGIExecutor}}.
The bottom line is that the contract in {{PrivilegedExecutor.execute()}} is
that they shouldn't change the exception thrown in the business logic - at
least it's not indicated in its signature in any way. The default
implementation ({{SimpleAuthenticator}}) behaves according to this.
I don't know the original intend behind this wrapping, [~jrufus] or
[~hshreedharan], do you happen to remember? (You were involved in the original
implementation in FLUME-2631)
Right now I don't see any problem in removing this and letting the original
exception to propagate as the {{org.apache.flume.auth.SecurityException}}
doesn't appear anywhere in the public interface.
was:
see:
https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
This has unexpected side effects as the callers try to catch the wrapped
exception, for example in {{BucketWriter.append()}}:
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
I don't know the original intend behind this wrapping, [~jrufus] or
[~hshreedharan], do you happen to remember? (You were involved in the original
implementation in FLUME-2631)
Right now I don't see any problem in removing this and letting the original
exception to propagate as the {{org.apache.flume.auth.SecurityException}}
doesn't appear anywhere in the public interface.
> Wrapping the exception into SecurityException in UGIExecutor.execute hides
> the original one
> ---
>
> Key: FLUME-3049
> URL: https://issues.apache.org/jira/browse/FLUME-3049
> Project: Flume
> Issue Type: Bug
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> see:
> https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
> This has unexpected side effects as the callers try to catch the wrapped
> exception, for example in {{BucketWriter.append()}}:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
> Here IOException is considered as non-transient failure thus the {{close()}}
> is called, but when the original exception is wrapped into
> {{SecurityException}} it doesn't trigger the close of the file.
> Similarly in {{HDFSEventSink.process()}} method the `IOException` is handled
> in a different way than any other exception. It might come from
> {{BucketWriter.append()}} or {{BucketWriter.flush()}} for example, and both
> of them invoke the hdfs call via a {{PrivilegedExecutor}} instance which
> might be the problematic {{UGIExecutor}}.
> The bottom line is that the contract in {{PrivilegedExecutor.execute()}} is
> that they shouldn't change the exception thrown in the business logic - at
> least it's not indicated in its signature in any way. The default
> implementation ({{SimpleAuthenticator}}) behaves according to this.
> I don't know the original intend behind this wrapping, [~jrufus] or
> [~hshreedharan], do you happen to remember? (You were involved in the
> original implementation in FLUME-2631)
> Right now I don't see any problem in removing this and letting the original
> exception to propagate as the {{org.apache.flume.auth.SecurityException}}
> doesn't appear anywhere in the public interface.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] (FLUME-3049) Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
[
https://issues.apache.org/jira/browse/FLUME-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847116#comment-15847116
]
Denes Arvay commented on FLUME-3049:
Note: I updated the description to make the intent more clear.
> Wrapping the exception into SecurityException in UGIExecutor.execute hides
> the original one
> ---
>
> Key: FLUME-3049
> URL: https://issues.apache.org/jira/browse/FLUME-3049
> Project: Flume
> Issue Type: Bug
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> see:
> https://github.com/apache/flume/blob/trunk/flume-ng-auth/src/main/java/org/apache/flume/auth/UGIExecutor.java#L49
> This has unexpected side effects as the callers try to catch the wrapped
> exception, for example in {{BucketWriter.append()}}:
> https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L563
> Here IOException is considered as non-transient failure thus the {{close()}}
> is called, but when the original exception is wrapped into
> {{SecurityException}} it doesn't trigger the close of the file.
> Similarly in {{HDFSEventSink.process()}} method the `IOException` is handled
> in a different way than any other exception. It might come from
> {{BucketWriter.append()}} or {{BucketWriter.flush()}} for example, and both
> of them invoke the hdfs call via a {{PrivilegedExecutor}} instance which
> might be the problematic {{UGIExecutor}}.
> The bottom line is that the contract in {{PrivilegedExecutor.execute()}} is
> that they shouldn't change the exception thrown in the business logic - at
> least it's not indicated in its signature in any way. The default
> implementation ({{SimpleAuthenticator}}) behaves according to this.
> I don't know the original intend behind this wrapping, [~jrufus] or
> [~hshreedharan], do you happen to remember? (You were involved in the
> original implementation in FLUME-2631)
> Right now I don't see any problem in removing this and letting the original
> exception to propagate as the {{org.apache.flume.auth.SecurityException}}
> doesn't appear anywhere in the public interface.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-2960) Support Wildcards in directory name in TaildirSource
[ https://issues.apache.org/jira/browse/FLUME-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15894488#comment-15894488 ] Denes Arvay commented on FLUME-2960: Hi [~wenqiao], I think it's a good idea to unify the patterns, and because the current implementation supports regexes in the filename I'd go with it to stay backward compatible. > Support Wildcards in directory name in TaildirSource > > > Key: FLUME-2960 > URL: https://issues.apache.org/jira/browse/FLUME-2960 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: v1.7.0 >Reporter: tinawenqiao >Assignee: tinawenqiao > Labels: wildcards > Fix For: v1.8.0 > > Attachments: FLUME-2960_1.patch, FLUME-2960_2.patch, > FLUME-2960_3.patch, FLUME-2960_4.patch, FLUME-2960_5.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > In our log management project, we wan't to track many log files like this: > /app/dir1/log.* > /app/dir2/log.* > ... > /app/dirn/log.* > But TaildirSource can't support wildcards in filegroup directory name. The > following config is expected: > a1.sources.r1.filegroups.fg = /app/\*/log.* -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLUME-3080) Close failure in HDFS Sink might cause data loss
Denes Arvay created FLUME-3080:
--
Summary: Close failure in HDFS Sink might cause data loss
Key: FLUME-3080
URL: https://issues.apache.org/jira/browse/FLUME-3080
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: 1.7.0
Reporter: Denes Arvay
Assignee: Denes Arvay
Priority: Blocker
If the HDFS Sink tries to close a file but it fails (e.g. due to timeout) the
last block might not end up in COMPLETE state. In this case block recovery
should happen but as the lease is still held by Flume the NameNode will start
the recovery process only after the hard limit of 1 hour expires.
The lease recovery can be started manually by the {{hdfs debug recoverLease}}
command.
For reproduction I removed the close call from the {{BucketWriter}}
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L380)
to simulate the failure and used the following config:
{noformat}
agent.sources = testsrc
agent.sinks = testsink
agent.channels = testch
agent.sources.testsrc.type = netcat
agent.sources.testsrc.bind = localhost
agent.sources.testsrc.port = 9494
agent.sources.testsrc.channels = testch
agent.sinks.testsink.type = hdfs
agent.sinks.testsink.hdfs.path = /user/flume/test
agent.sinks.testsink.hdfs.rollInterval = 0
agent.sinks.testsink.hdfs.rollCount = 3
agent.sinks.testsink.serializer = avro_event
agent.sinks.testsink.serializer.compressionCodec = snappy
agent.sinks.testsink.hdfs.fileSuffix = .avro
agent.sinks.testsink.hdfs.fileType = DataStream
agent.sinks.testsink.hdfs.batchSize = 2
agent.sinks.testsink.hdfs.writeFormat = Text
agent.sinks.testsink.hdfs.idleTimeout=20
agent.sinks.testsink.channel = testch
agent.channels.testch.type = memory
{noformat}
After ingesting 6 events ("a" - "f") 2 files were created on HDFS, as expected.
But there are missing events when listing the contents in Spark shell:
{noformat}
scala>
sqlContext.read.avro("/user/flume/test/FlumeData.14908867127*.avro").collect().map(a
=> new String(a(1).asInstanceOf[Array[Byte]])).foreach(println)
a
b
d
{noformat}
{{hdfs fsck}} also confirms that the blocks are still in {{UNDER_CONSTRUCTION}}
state:
{noformat}
$ hdfs fsck /user/flume/test/ -openforwrite -files -blocks
FSCK started by root (auth:SIMPLE) from /172.31.114.3 for path
/user/flume/test/ at Thu Mar 30 08:23:56 PDT 2017
/user/flume/test/
/user/flume/test/FlumeData.1490887185312.avro 310 bytes, 1 block(s),
OPENFORWRITE: MISSING 1 blocks of total size 310 B
0.
BP-1285398861-172.31.114.3-1489845696835:blk_1073761923_21128{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-e0d04bef-a861-40b0-99dd-27bfb2871ecd:NORMAL:172.31.114.27:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-d1979e0c-db81-4790-b225-ae8a4cf42dd8:NORMAL:172.31.114.32:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-ca00550d-702e-4892-a54a-7105af0c19ee:NORMAL:172.31.114.24:20002|RBW]]}
len=310 MISSING!
/user/flume/test/FlumeData.1490887185313.avro 292 bytes, 1 block(s),
OPENFORWRITE: MISSING 1 blocks of total size 292 B
0.
BP-1285398861-172.31.114.3-1489845696835:blk_1073761924_21129{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-ca00550d-702e-4892-a54a-7105af0c19ee:NORMAL:172.31.114.24:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-e0d04bef-a861-40b0-99dd-27bfb2871ecd:NORMAL:172.31.114.27:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-d1979e0c-db81-4790-b225-ae8a4cf42dd8:NORMAL:172.31.114.32:20002|RBW]]}
len=292 MISSING!
{noformat}
These blocks need to be recovered by starting a lease recovery process on the
NameNode (which will then run the block recovery as well). This can be
triggered programmatically via the DFSClient.
Adding this call if the close fails solves the issue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (FLUME-3080) Close failure in HDFS Sink might cause data loss
[
https://issues.apache.org/jira/browse/FLUME-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-3080:
---
Description:
If the HDFS Sink tries to close a file but it fails (e.g. due to timeout) the
last block might not end up in COMPLETE state. In this case block recovery
should happen but as the lease is still held by Flume the NameNode will start
the recovery process only after the hard limit of 1 hour expires.
The lease recovery can be started manually by the {{hdfs debug recoverLease}}
command.
For reproduction I removed the close call from the {{BucketWriter}}
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L380)
to simulate the failure and used the following config:
{noformat}
agent.sources = testsrc
agent.sinks = testsink
agent.channels = testch
agent.sources.testsrc.type = netcat
agent.sources.testsrc.bind = localhost
agent.sources.testsrc.port = 9494
agent.sources.testsrc.channels = testch
agent.sinks.testsink.type = hdfs
agent.sinks.testsink.hdfs.path = /user/flume/test
agent.sinks.testsink.hdfs.rollInterval = 0
agent.sinks.testsink.hdfs.rollCount = 3
agent.sinks.testsink.serializer = avro_event
agent.sinks.testsink.serializer.compressionCodec = snappy
agent.sinks.testsink.hdfs.fileSuffix = .avro
agent.sinks.testsink.hdfs.fileType = DataStream
agent.sinks.testsink.hdfs.batchSize = 2
agent.sinks.testsink.hdfs.writeFormat = Text
agent.sinks.testsink.hdfs.idleTimeout=20
agent.sinks.testsink.channel = testch
agent.channels.testch.type = memory
{noformat}
After ingesting 6 events ("a" - "f") 2 files were created on HDFS, as expected.
But there are missing events when listing the contents in Spark shell:
{noformat}
scala>
sqlContext.read.avro("/user/flume/test/FlumeData.14908867127*.avro").collect().map(a
=> new String(a(1).asInstanceOf[Array[Byte]])).foreach(println)
a
b
d
{noformat}
{{hdfs fsck}} also confirms that the blocks are still in {{UNDER_CONSTRUCTION}}
state:
{noformat}
$ hdfs fsck /user/flume/test/ -openforwrite -files -blocks
FSCK started by root (auth:SIMPLE) from /172.31.114.3 for path
/user/flume/test/ at Thu Mar 30 08:23:56 PDT 2017
/user/flume/test/
/user/flume/test/FlumeData.1490887185312.avro 310 bytes, 1 block(s),
OPENFORWRITE: MISSING 1 blocks of total size 310 B
0.
BP-1285398861-172.31.114.3-1489845696835:blk_1073761923_21128{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-e0d04bef-a861-40b0-99dd-27bfb2871ecd:NORMAL:172.31.114.27:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-d1979e0c-db81-4790-b225-ae8a4cf42dd8:NORMAL:172.31.114.32:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-ca00550d-702e-4892-a54a-7105af0c19ee:NORMAL:172.31.114.24:20002|RBW]]}
len=310 MISSING!
/user/flume/test/FlumeData.1490887185313.avro 292 bytes, 1 block(s),
OPENFORWRITE: MISSING 1 blocks of total size 292 B
0.
BP-1285398861-172.31.114.3-1489845696835:blk_1073761924_21129{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-ca00550d-702e-4892-a54a-7105af0c19ee:NORMAL:172.31.114.24:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-e0d04bef-a861-40b0-99dd-27bfb2871ecd:NORMAL:172.31.114.27:20002|RBW],
ReplicaUnderConstruction[[DISK]DS-d1979e0c-db81-4790-b225-ae8a4cf42dd8:NORMAL:172.31.114.32:20002|RBW]]}
len=292 MISSING!
{noformat}
These blocks need to be recovered by starting a lease recovery process on the
NameNode (which will then run the block recovery as well). This can be
triggered programmatically via the DFSClient.
Adding this call if the close fails solves the issue.
cc [~jojochuang]
was:
If the HDFS Sink tries to close a file but it fails (e.g. due to timeout) the
last block might not end up in COMPLETE state. In this case block recovery
should happen but as the lease is still held by Flume the NameNode will start
the recovery process only after the hard limit of 1 hour expires.
The lease recovery can be started manually by the {{hdfs debug recoverLease}}
command.
For reproduction I removed the close call from the {{BucketWriter}}
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L380)
to simulate the failure and used the following config:
{noformat}
agent.sources = testsrc
agent.sinks = testsink
agent.channels = testch
agent.sources.testsrc.type = netcat
agent.sources.testsrc.bind = localhost
agent.sources.testsrc.port = 9494
agent.sources.testsrc.channels = testch
agent.sinks.testsink.type = hdfs
agent.sinks.testsink.hdfs.path = /user/flume/test
agent.sinks.testsink.hdfs.rollInterval = 0
agent.sinks.testsink.hdfs.rollCount = 3
agent.sinks.testsink.serializer = avro_event
agent.sinks.testsink.serializer.compressionCodec = snappy
agent.sinks.testsink.hdfs.fileSuffix = .avro
agent.sinks.testsink.hdfs.f
[jira] [Created] (FLUME-3085) HDFS Sink can skip flushing some BucketWriters, might lead to data loss
Denes Arvay created FLUME-3085:
--
Summary: HDFS Sink can skip flushing some BucketWriters, might
lead to data loss
Key: FLUME-3085
URL: https://issues.apache.org/jira/browse/FLUME-3085
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: 1.7.0
Reporter: Denes Arvay
Assignee: Denes Arvay
Priority: Critical
The {{HDFSEventSink.process()}} is already prepared for a rare race condition,
namely when the BucketWriter acquired in [line
389|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L389]
gets closed by an other thread (e.g. because the {{idleTimeout}} or the
{{rollInterval}}) before the {{append()}} is called in [line
406|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L406].
If this is the case the {{BucketWriter.append()}} call throws a
{{BucketClosedException}} and the sink creates a new {{BucketWriter}} instance
and appends to it.
But this newly created instance won't be added to the {{writers}} list, which
means that it won't be flushed after the processing loop finished:
https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429
This has multiple consequences:
- unflushed data might get lost
- the {{BucketWriter}}'s {{idleAction}} won't be scheduled
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450),
which means that it won't be closed nor renamed if the idle timeout is the
only trigger for closing the file.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-3087) Change log level from WARN to INFO when using default "maxIOWorkers" value.
[
https://issues.apache.org/jira/browse/FLUME-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978346#comment-15978346
]
Denes Arvay commented on FLUME-3087:
I agree with the change, thank you. +1 from me.
> Change log level from WARN to INFO when using default "maxIOWorkers" value.
> ---
>
> Key: FLUME-3087
> URL: https://issues.apache.org/jira/browse/FLUME-3087
> Project: Flume
> Issue Type: Wish
> Components: Client SDK
>Affects Versions: 1.7.0
>Reporter: Takafumi Saito
>Priority: Minor
> Attachments: FLUME-3087.patch
>
>
> This message confuse us when looking warn log.
> {noformat}
> WARN 2017-04-21 00:23:38,155 [SinkRunner-PollingRunner-DefaultSinkProcessor]
> org.apache.flume.api.NettyAvroRpcClient:634 - Using default maxIOWorkers
> {noformat}
> When using default value, I think the log level should be set INFO rather
> than WARN.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (FLUME-3092) Extend the FileChannel's monitoring metrics
Denes Arvay created FLUME-3092:
--
Summary: Extend the FileChannel's monitoring metrics
Key: FLUME-3092
URL: https://issues.apache.org/jira/browse/FLUME-3092
Project: Flume
Issue Type: Improvement
Components: File Channel
Affects Versions: 1.7.0
Reporter: Denes Arvay
Assignee: Denes Arvay
There are already several generic metrics (e.g. {{eventPutAttemptCount}} and
{{eventPutSuccessCount}}) which can be used to create compound metrics for
monitoring the FileChannel's health.
Some monitoring system's aren't capable to calculate such derived metrics,
though, so I recommend to add the following extra counters to represent if a
channel operation failed or the channel is in an unhealthy state.
- {{eventPutErrorCount}}: incremented if an {{IOException}} occurs during
{{put}} operation.
- {{eventTakeErrorCount}}: incremented if an {{IOException}} or
{{CorruptEventException}} occurs during {{take}} operation.
- {{checkpointWriteErrorCount}}: incremented if an exception occurs during
checkpoint write.
- {{unhealthy}}: this flag represents whether the channel has started
successfully (i.e. the replay ran without any problem). This is similar to the
already existing {{open}} flag except that the latter is initially false and is
set to {{true}} if the initialization (including the log replay) is
successfully done. The {{unhealthy}}, in contrary, is {{false}} by default and
is set to {{true}} if there is an error during startup.
Beside these flags I'd also introduce a {{closed}} flag which is the numeric
representation (1: closed, 0: open) of the negated (already existing) {{open}}
flag.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLUME-2945) Bump java target version to 1.8
[ https://issues.apache.org/jira/browse/FLUME-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042662#comment-16042662 ] Denes Arvay commented on FLUME-2945: Hi [~liorze], I think this patch is ready to go. I'm going to do a final review and if nobody has any concerns I'll commit it. > Bump java target version to 1.8 > --- > > Key: FLUME-2945 > URL: https://issues.apache.org/jira/browse/FLUME-2945 > Project: Flume > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: 1.8.0 > > Attachments: FLUME-2945-0.patch > > > We should move to Java 8 as a minimum requirement. > # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. > # Many projects are adopting java 8 as a minimum requirement, for instance: > * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 > * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 > * elasticsearch 5: > https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html > The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91". > All unit tests, except for the known flaky ones, successfully passed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (FLUME-3109) Make HDFS Sink's idleTimeout more robust
Denes Arvay created FLUME-3109:
--
Summary: Make HDFS Sink's idleTimeout more robust
Key: FLUME-3109
URL: https://issues.apache.org/jira/browse/FLUME-3109
Project: Flume
Issue Type: Improvement
Affects Versions: 1.7.0
Reporter: Denes Arvay
Assignee: Denes Arvay
Currently the {{idleFuture}} which is responsible of closing idle files is
scheduled in {{BucketWriter.flush()}} while it is cancelled in
{{BucketWriter.append()}}.
If events were appended to a BucketWriter but the next flush failed then the
{{idleFuture}} won't be scheduled, which lead to unclosed/unrenamed files in a
setup where there is no other logic to close/rename the files (i.e.
hdfs.rollInterval, hdfs.rollSize and hdfs.rollCount are all 0).
Moreover, if any of the {{flush()}} close calls fail in
{{HDFSEventSink.process()}}
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429)
then all the subsequent flushes will be skipped thus no idleFutures will be
scheduled.
I'd recommend to move the scheduling of the idleFuture
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450-L467)
to {{BucketWriter.append()}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-3109) Make HDFS Sink's idleTimeout more robust
[
https://issues.apache.org/jira/browse/FLUME-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay updated FLUME-3109:
---
Description:
Currently the {{idleFuture}} which is responsible of closing idle files is
scheduled in {{BucketWriter.flush()}} while it is cancelled in
{{BucketWriter.append()}}.
If events were appended to a BucketWriter but the next flush failed then the
{{idleFuture}} won't be scheduled, which lead to unclosed/unrenamed files in a
setup where there is no other logic to close/rename the files (i.e.
hdfs.rollInterval, hdfs.rollSize and hdfs.rollCount are all 0).
Moreover, if any of the {{flush()}} calls fail in {{HDFSEventSink.process()}}
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429)
then all the subsequent flushes will be skipped thus no idleFutures will be
scheduled.
I'd recommend to move the scheduling of the idleFuture
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450-L467)
to {{BucketWriter.append()}}.
was:
Currently the {{idleFuture}} which is responsible of closing idle files is
scheduled in {{BucketWriter.flush()}} while it is cancelled in
{{BucketWriter.append()}}.
If events were appended to a BucketWriter but the next flush failed then the
{{idleFuture}} won't be scheduled, which lead to unclosed/unrenamed files in a
setup where there is no other logic to close/rename the files (i.e.
hdfs.rollInterval, hdfs.rollSize and hdfs.rollCount are all 0).
Moreover, if any of the {{flush()}} close calls fail in
{{HDFSEventSink.process()}}
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429)
then all the subsequent flushes will be skipped thus no idleFutures will be
scheduled.
I'd recommend to move the scheduling of the idleFuture
(https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450-L467)
to {{BucketWriter.append()}}.
> Make HDFS Sink's idleTimeout more robust
>
>
> Key: FLUME-3109
> URL: https://issues.apache.org/jira/browse/FLUME-3109
> Project: Flume
> Issue Type: Improvement
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> Currently the {{idleFuture}} which is responsible of closing idle files is
> scheduled in {{BucketWriter.flush()}} while it is cancelled in
> {{BucketWriter.append()}}.
> If events were appended to a BucketWriter but the next flush failed then the
> {{idleFuture}} won't be scheduled, which lead to unclosed/unrenamed files in
> a setup where there is no other logic to close/rename the files (i.e.
> hdfs.rollInterval, hdfs.rollSize and hdfs.rollCount are all 0).
> Moreover, if any of the {{flush()}} calls fail in {{HDFSEventSink.process()}}
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429)
> then all the subsequent flushes will be skipped thus no idleFutures will be
> scheduled.
> I'd recommend to move the scheduling of the idleFuture
> (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450-L467)
> to {{BucketWriter.append()}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (FLUME-3072) Add IP address to headers in flume log4j appender
[ https://issues.apache.org/jira/browse/FLUME-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay resolved FLUME-3072. Resolution: Fixed Fix Version/s: 1.8.0 > Add IP address to headers in flume log4j appender > - > > Key: FLUME-3072 > URL: https://issues.apache.org/jira/browse/FLUME-3072 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 >Reporter: Andras Beni >Assignee: Andras Beni > Fix For: 1.8.0 > > > Add client's host address to log event headers in Flume NG Log4j Appender. > By receiving such events through AvroSource, one can route events based on > origin host in the receiving Flume agent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (FLUME-2579) JMS source support durable subscriptions and message listening
[ https://issues.apache.org/jira/browse/FLUME-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay resolved FLUME-2579. Resolution: Fixed > JMS source support durable subscriptions and message listening > -- > > Key: FLUME-2579 > URL: https://issues.apache.org/jira/browse/FLUME-2579 > Project: Flume > Issue Type: Wish > Components: Sinks+Sources >Affects Versions: 1.5.2 >Reporter: zhangshengbing >Assignee: Andras Beni > Fix For: 1.8.0 > > > best wishes -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3072) Add IP address to headers in flume log4j appender
[ https://issues.apache.org/jira/browse/FLUME-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16057671#comment-16057671 ] Denes Arvay commented on FLUME-3072: Thanks for the patch [~andrasbeni], thanks for the review [~sati] > Add IP address to headers in flume log4j appender > - > > Key: FLUME-3072 > URL: https://issues.apache.org/jira/browse/FLUME-3072 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.8.0 >Reporter: Andras Beni >Assignee: Andras Beni > Fix For: 1.8.0 > > > Add client's host address to log event headers in Flume NG Log4j Appender. > By receiving such events through AvroSource, one can route events based on > origin host in the receiving Flume agent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2945) Bump java target version to 1.8
[ https://issues.apache.org/jira/browse/FLUME-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16057675#comment-16057675 ] Denes Arvay commented on FLUME-2945: Thanks for the patch [~liorze], thank you for the review [~sati] > Bump java target version to 1.8 > --- > > Key: FLUME-2945 > URL: https://issues.apache.org/jira/browse/FLUME-2945 > Project: Flume > Issue Type: Improvement >Affects Versions: 1.8.0 >Reporter: Lior Zeno >Assignee: Lior Zeno > Fix For: 1.8.0 > > Attachments: FLUME-2945-0.patch > > > We should move to Java 8 as a minimum requirement. > # Java 7 is EOL'ed http://www.oracle.com/technetwork/java/eol-135779.html. > # Many projects are adopting java 8 as a minimum requirement, for instance: > * Solr 6: https://issues.apache.org/jira/browse/LUCENE-6722 > * Hbase 2: https://issues.apache.org/jira/browse/HBASE-15624 > * elasticsearch 5: > https://www.elastic.co/guide/en/elasticsearch/reference/master/setup.html > The attached patch was tested on Ubuntu 16.04 with openjdk version "1.8.0_91". > All unit tests, except for the known flaky ones, successfully passed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2579) JMS source support durable subscriptions and message listening
[ https://issues.apache.org/jira/browse/FLUME-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16057669#comment-16057669 ] Denes Arvay commented on FLUME-2579: Thanks for the patch [~andrasbeni], thanks for the review [~sati] > JMS source support durable subscriptions and message listening > -- > > Key: FLUME-2579 > URL: https://issues.apache.org/jira/browse/FLUME-2579 > Project: Flume > Issue Type: Wish > Components: Sinks+Sources >Affects Versions: 1.5.2 >Reporter: zhangshengbing >Assignee: Andras Beni > Fix For: 1.8.0 > > > best wishes -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (FLUME-2752) Flume AvroSource will leak the memory and the OOM will be happened.
[
https://issues.apache.org/jira/browse/FLUME-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay resolved FLUME-2752.
Resolution: Fixed
Thank you [~sati] for the patch, I have committed it.
> Flume AvroSource will leak the memory and the OOM will be happened.
> ---
>
> Key: FLUME-2752
> URL: https://issues.apache.org/jira/browse/FLUME-2752
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: yinghua_zh
>Assignee: Attila Simon
>
> If the flume agent config the nonexist IP for the avro source,the exception
> will be happened as follow:
> 2015-07-21 19:57:47,054 | ERROR | [lifecycleSupervisor-1-2] | Unable to
> start EventDrivenSourceRunner: { source:Avro source avro_source_21155: {
> bindAddress: 51.196.27.32, port: 21155 } } - Exception follows. |
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)
> org.jboss.netty.channel.ChannelException: Failed to bind to:
> /51.196.27.32:21155
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:297)
> at org.apache.avro.ipc.NettyServer.(NettyServer.java:106)
> at org.apache.flume.source.AvroSource.start(AvroSource.java:294)
> at
> org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.BindException: Cannot assign requested address
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:437)
> at sun.nio.ch.Net.bind(Net.java:429)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.bind(NioServerSocketPipelineSink.java:140)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleServerSocket(NioServerSocketPipelineSink.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:64)
> at org.jboss.netty.channel.Channels.bind(Channels.java:569)
> at
> org.jboss.netty.channel.AbstractChannel.bind(AbstractChannel.java:189)
> at
> org.jboss.netty.bootstrap.ServerBootstrap$Binder.channelOpen(ServerBootstrap.java:342)
> at org.jboss.netty.channel.Channels.fireChannelOpen(Channels.java:170)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:80)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:158)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.newChannel(NioServerSocketChannelFactory.java:86)
> at
> org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:276)
> if the above exception happened for 2 hours,and the agent JVM -Xxx is 4G,the
> OutOfMemory will be happened.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2175) Update Developer Guide with notes on how to upgrade Protocol Buffer version
[ https://issues.apache.org/jira/browse/FLUME-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073401#comment-16073401 ] Denes Arvay commented on FLUME-2175: Thanks [~roshan_naik] for the patch, [~ashishpaliwal] and [~sati] for the review, I'm going to commit this. > Update Developer Guide with notes on how to upgrade Protocol Buffer version > --- > > Key: FLUME-2175 > URL: https://issues.apache.org/jira/browse/FLUME-2175 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: 1.4.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > Labels: build, doc > Attachments: FLUME-2175.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-1520) Timestamp interceptor should support custom headers
[ https://issues.apache.org/jira/browse/FLUME-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16096017#comment-16096017 ] Denes Arvay commented on FLUME-1520: Thank you all for the contribution, I'll commit the latest patch if there are no concerns. > Timestamp interceptor should support custom headers > --- > > Key: FLUME-1520 > URL: https://issues.apache.org/jira/browse/FLUME-1520 > Project: Flume > Issue Type: Improvement >Reporter: Hari Shreedharan >Assignee: Hari Shreedharan > Fix For: 1.8.0 > > Attachments: FLUME-1520-2.patch, FLUME-1520-3.patch, FLUME-1520.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-2620) File channel throws NullPointerException if a header value is null
[
https://issues.apache.org/jira/browse/FLUME-2620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099934#comment-16099934
]
Denes Arvay commented on FLUME-2620:
Thank you all for the contribution, I have committed and pushed this change.
Marking the ticket as resolved.
> File channel throws NullPointerException if a header value is null
> --
>
> Key: FLUME-2620
> URL: https://issues.apache.org/jira/browse/FLUME-2620
> Project: Flume
> Issue Type: Bug
> Components: File Channel
>Reporter: Santiago M. Mola
>Assignee: Marcell Hegedus
> Attachments: FLUME-2620-0.patch, FLUME-2620-1.patch,
> FLUME-2620-2.patch, FLUME-2620-3.patch, FLUME-2620-4.patch,
> FLUME-2620-5.patch, FLUME-2620.patch, FLUME-2620.patch
>
>
> File channel throws NullPointerException if a header value is null.
> If this is intended, it should be reported correctly in the logs.
> Sample trace:
> org.apache.flume.ChannelException: Unable to put batch on required channel:
> FileChannel chan { dataDirs: [/var/lib/ingestion-csv/chan/data] }
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
> at
> org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:236)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.flume.channel.file.proto.ProtosFactory$FlumeEventHeader$Builder.setValue(ProtosFactory.java:7415)
> at org.apache.flume.channel.file.Put.writeProtos(Put.java:85)
> at
> org.apache.flume.channel.file.TransactionEventRecord.toByteBuffer(TransactionEventRecord.java:174)
> at org.apache.flume.channel.file.Log.put(Log.java:622)
> at
> org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
> at
> org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
> at
> org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
> at
> org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:189)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3135) property logger in org.apache.flume.interceptor.RegexFilteringInterceptor confused
[
https://issues.apache.org/jira/browse/FLUME-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16101294#comment-16101294
]
Denes Arvay commented on FLUME-3135:
Thank you [~peterchenhdu] for the patch, [~marcellhegedus] and [~sati] for the
review.
> property logger in org.apache.flume.interceptor.RegexFilteringInterceptor
> confused
> --
>
> Key: FLUME-3135
> URL: https://issues.apache.org/jira/browse/FLUME-3135
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Peter Chen
>Priority: Minor
> Labels: easyfix
>
> *+org.apache.flume.interceptor.RegexFilteringInterceptor.java+*
> * *line 72-75:*
> the parameter to the getLogger method should be
> RegexFilteringInterceptor.class
> {code:java}
> public class RegexFilteringInterceptor implements Interceptor {
> private static final Logger logger = LoggerFactory
> .getLogger(StaticInterceptor.class);
> {code}
> * *line 141-143:*
> Javadoc
> {code:java}
> /**
>* Builder which builds new instance of the StaticInterceptor.
>*/
> {code}
> :D
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (FLUME-3135) property logger in org.apache.flume.interceptor.RegexFilteringInterceptor confused
[
https://issues.apache.org/jira/browse/FLUME-3135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay resolved FLUME-3135.
Resolution: Fixed
> property logger in org.apache.flume.interceptor.RegexFilteringInterceptor
> confused
> --
>
> Key: FLUME-3135
> URL: https://issues.apache.org/jira/browse/FLUME-3135
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Peter Chen
>Priority: Minor
> Labels: easyfix
>
> *+org.apache.flume.interceptor.RegexFilteringInterceptor.java+*
> * *line 72-75:*
> the parameter to the getLogger method should be
> RegexFilteringInterceptor.class
> {code:java}
> public class RegexFilteringInterceptor implements Interceptor {
> private static final Logger logger = LoggerFactory
> .getLogger(StaticInterceptor.class);
> {code}
> * *line 141-143:*
> Javadoc
> {code:java}
> /**
>* Builder which builds new instance of the StaticInterceptor.
>*/
> {code}
> :D
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (FLUME-3093) Groundwork for version changes in root pom
[ https://issues.apache.org/jira/browse/FLUME-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay resolved FLUME-3093. Resolution: Fixed > Groundwork for version changes in root pom > -- > > Key: FLUME-3093 > URL: https://issues.apache.org/jira/browse/FLUME-3093 > Project: Flume > Issue Type: Task >Affects Versions: 1.7.0 >Reporter: Miklos Csanady >Assignee: Miklos Csanady > Labels: dependency > > Flume's root pom should have a parameter block where all the dependency > versions are listed. > This is fundamental to later version changes required to time effectively > overcame 3rd party security vulnerabilities. > This should be done in two steps: first just refactoring with no change, the > second is getting rid of unnecessary different versions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (FLUME-3141) Small typo found in RegexHbaseEventSerializer.java
[ https://issues.apache.org/jira/browse/FLUME-3141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay resolved FLUME-3141. Resolution: Fixed Thanks [~mcsanady] for the fix, I've committed it. > Small typo found in RegexHbaseEventSerializer.java > -- > > Key: FLUME-3141 > URL: https://issues.apache.org/jira/browse/FLUME-3141 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Miklos Csanady >Assignee: Miklos Csanady >Priority: Trivial > Labels: trivial > > Trivial typo found in RegexHbaseEventSerializer.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3144) Improve Log4jAppender's performance by allowing logging collection of messages
Denes Arvay created FLUME-3144:
--
Summary: Improve Log4jAppender's performance by allowing logging
collection of messages
Key: FLUME-3144
URL: https://issues.apache.org/jira/browse/FLUME-3144
Project: Flume
Issue Type: Improvement
Affects Versions: 1.7.0
Reporter: Denes Arvay
Assignee: Denes Arvay
Currently it's only possible to send events one by one with the Log4j appender.
(Except if the events are wrapped in an avro record but it's quite cumbersome
and might need special handling on the receiving side)
As the Log4j methods can handle any {{Object}} I'd suggest to improve the Log4j
appender to treat {{Collection}} event as a special case and send its content
to Flume with one {{rpcClient.appendBatch()}} call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (FLUME-3144) Improve Log4jAppender's performance by allowing logging collection of messages
[
https://issues.apache.org/jira/browse/FLUME-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denes Arvay resolved FLUME-3144.
Resolution: Fixed
> Improve Log4jAppender's performance by allowing logging collection of messages
> --
>
> Key: FLUME-3144
> URL: https://issues.apache.org/jira/browse/FLUME-3144
> Project: Flume
> Issue Type: Improvement
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>
> Currently it's only possible to send events one by one with the Log4j
> appender. (Except if the events are wrapped in an avro record but it's quite
> cumbersome and might need special handling on the receiving side)
> As the Log4j methods can handle any {{Object}} I'd suggest to improve the
> Log4j appender to treat {{Collection}} event as a special case and send its
> content to Flume with one {{rpcClient.appendBatch()}} call.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (FLUME-3152) Add Flume Metric for Backup Checkpoint Errors
[ https://issues.apache.org/jira/browse/FLUME-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Arvay resolved FLUME-3152. Resolution: Fixed > Add Flume Metric for Backup Checkpoint Errors > - > > Key: FLUME-3152 > URL: https://issues.apache.org/jira/browse/FLUME-3152 > Project: Flume > Issue Type: Bug > Components: File Channel >Affects Versions: 1.7.0 >Reporter: Ferenc Szabo >Assignee: Ferenc Szabo > Fix For: 1.8.0 > > > File channel metrics should contain a counter for checkpoint backup errors -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3033) Add SSL support to JMS source
[ https://issues.apache.org/jira/browse/FLUME-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139997#comment-16139997 ] Denes Arvay commented on FLUME-3033: [~aleck7], thank you for the patch, it seems to be a good addition to Flume. Could you please either open a pull request or submit the patch to the reviewboard? (I agree with [~bessbd] and I also prefer the pull requests). This would help the reviewers to properly review your change. Thanks in advance. > Add SSL support to JMS source > - > > Key: FLUME-3033 > URL: https://issues.apache.org/jira/browse/FLUME-3033 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: 1.7.0 >Reporter: Aleck Kulabukhov > Labels: features > Fix For: notrack > > Attachments: FLUME_3033_Add_SSL_support_to_JMS_source.patch.zip > > Original Estimate: 48h > Remaining Estimate: 48h > > As a user I want to connect to JMS sources via SSL. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3112) Upgrade jackson-core library dependency
[ https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16140461#comment-16140461 ] Denes Arvay commented on FLUME-3112: Thank you [~fszabo] for the patch, [~sati] and [~mcsanady] for the review. > Upgrade jackson-core library dependency > --- > > Key: FLUME-3112 > URL: https://issues.apache.org/jira/browse/FLUME-3112 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > Attachments: FLUME-3112.patch > > > ||Group||Artifact||Version used||Upgrade target|| > |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9| > Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
