[jira] [Commented] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le
[ https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050534#comment-16050534 ] Ferenc Szabo commented on FLUME-3057: - I had the same issue. Updating the version solved it. > Build fails due to unsupported snappy-java version on ppc64le > - > > Key: FLUME-3057 > URL: https://issues.apache.org/jira/browse/FLUME-3057 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.8.0 > Environment: $ uname -a > Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC > 2016 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Pravin Dsilva > Labels: powerpc, ppc64le > Attachments: FLUME-3057.patch > > > Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on > ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=ppc64le" are seen > Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence > Flume needs to have this higher version of snappy-java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2973) Deadlock in hdfs sink
[
https://issues.apache.org/jira/browse/FLUME-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2973:
Fix Version/s: 1.8.0
> Deadlock in hdfs sink
> -
>
> Key: FLUME-2973
> URL: https://issues.apache.org/jira/browse/FLUME-2973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Critical
> Labels: hdfssink
> Fix For: 1.8.0
>
> Attachments: FLUME-2973-1.patch, FLUME-2973.patch
>
>
> Automatic close of BucketWriters (when open file count reached
> {{hdfs.maxOpenFiles}}) and the file rolling thread can end up in deadlock.
> When creating a new {{BucketWriter}} in {{HDFSEventSink}} it locks
> {{HDFSEventSink.sfWritersLock}} and the {{close()}} called in
> {{HDFSEventSink.sfWritersLock.removeEldestEntry}} tries to lock the
> {{BucketWriter}} instance.
> On the other hand if the file is being rolled in
> {{BucketWriter.close(boolean)}} it locks the {{BucketWriter}} instance first
> and in the close callback it tries to lock the {{sfWritersLock}}.
> The chances for this deadlock is higher when the {{hdfs.maxOpenFiles}}'s
> value is low (1).
> Script to reproduce:
> https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
> (put to
> {{flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs}})
> Deadlock usually occurs before ~30 iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-3101) taildir source may endless loop when tail a file
[
https://issues.apache.org/jira/browse/FLUME-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-3101:
Fix Version/s: (was: 1.7.0)
1.8.0
> taildir source may endless loop when tail a file
>
>
> Key: FLUME-3101
> URL: https://issues.apache.org/jira/browse/FLUME-3101
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: hunshenshi
> Labels: patch, taildirsource
> Fix For: 1.8.0
>
> Attachments: FLUME-3101-0.patch, FLUME-3101-1.patch,
> FLUME-3101-2.patch, FLUME-3101-3.patch
>
>
> If there are many files in the path that need to be tail, and there is a file
> written by *high frequency* (for example, there are file a, file b and file c
> in the path, file a is written at high frequency), *taildir can read the
> batchSize size event from file a everytime*, then taildir will only read data
> from file a, other files will not to be read, because in
> TaildirSource.tailFileProcess will into an endless loop.
> code:
> {code:title=TaildirSource.java|borderStyle=solid}
> private void tailFileProcess(TailFile tf, boolean backoffWithoutNL)
> throws IOException, InterruptedException {
> while (true) {
> // if events.size >= batchSize will not break while,
> // then into endless loop to only read tf
> if (events.size() < batchSize) {
> break;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-3068) The taildir source can tail hidden files.
[ https://issues.apache.org/jira/browse/FLUME-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3068: Fix Version/s: 1.8.0 > The taildir source can tail hidden files. > - > > Key: FLUME-3068 > URL: https://issues.apache.org/jira/browse/FLUME-3068 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: 1.7.0 > Environment: Linux >Reporter: Jinjiang Ling > Fix For: 1.8.0 > > Attachments: FLUME-3068-0.patch, FLUME-3068-1.patch, > FLUME-3068-2.patch > > > When using the taildir to monitor a directory, hidden files in the directory > are tailed by the source. > As the hidden files are always temp files of vim, etc. So it's necessary to > ignore the hidden file when using the taildir source. > Now Add a new configuration option named "ignoreHiddenFile", and the default > value is "false". When set it to "true", the hidden file will be ingored. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3057) Build fails due to unsupported snappy-java version on ppc64le
[ https://issues.apache.org/jira/browse/FLUME-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3057: Fix Version/s: 1.8.0 > Build fails due to unsupported snappy-java version on ppc64le > - > > Key: FLUME-3057 > URL: https://issues.apache.org/jira/browse/FLUME-3057 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.8.0 > Environment: $ uname -a > Linux 2f63413ff231 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:05:18 UTC > 2016 ppc64le ppc64le ppc64le GNU/Linux >Reporter: Pravin Dsilva > Labels: powerpc, ppc64le > Fix For: 1.8.0 > > Attachments: FLUME-3057.patch > > > Flume has a snappy-java dependency with version 1.1.0. Upon building Flume on > ppc64le architecture, errors such as "[FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=ppc64le" are seen > Native libraries for ppc64le were added in snappy-java version 1.1.1. Hence > Flume needs to have this higher version of snappy-java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3046) Kafka Sink and Source Configuration Improvements
[ https://issues.apache.org/jira/browse/FLUME-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3046: Fix Version/s: 1.8.0 > Kafka Sink and Source Configuration Improvements > > > Key: FLUME-3046 > URL: https://issues.apache.org/jira/browse/FLUME-3046 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Jeff Holoman >Assignee: Tristan Stevens > Fix For: 1.8.0 > > > Currently the Kafka Source sets the header for the topic. The sink reads this > value, rather than the statically defined topic value. We should fix this so > that you can either change the topic header that is used, or just choose to > prefer the statically defined topic in the sink. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2984) Default maxReadBufferBytes might cause OOM and cause thrift source exit
[
https://issues.apache.org/jira/browse/FLUME-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2984:
Fix Version/s: 1.8.0
> Default maxReadBufferBytes might cause OOM and cause thrift source exit
> ---
>
> Key: FLUME-2984
> URL: https://issues.apache.org/jira/browse/FLUME-2984
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Laxman
>Assignee: Laxman
> Fix For: 1.8.0
>
> Attachments: FLUME-2984.patch
>
>
> Our usecase has multi agent flow and we are achieving this with thrift source
> and thrift sink combination. When there is a surge in incoming traffic
> downstream agent having thrift source is running out of memory. On further
> investigation of thrift transport/server code and scribe source found that we
> may need to fix the thrift transport's maxReadBufferBytes. Similar issue has
> been fixed in ScribeSource as part of FLUME-2404 by making this buffer
> configurable and capping it to a 16MB if not configured.
> *thrift code (org.apache.thrift.server.AbstractNonblockingServer)*
> {code}
> public abstract static class AbstractNonblockingServerArgs AbstractNonblockingServer.AbstractNonblockingServerArgs> extends
> AbstractServerArgs {
> public long maxReadBufferBytes = 9223372036854775807L;
> public AbstractNonblockingServerArgs(TNonblockingServerTransport
> transport) {
> super(transport);
> this.transportFactory(new Factory());
> }
> }
> {code}
> *stacktrace*
> {noformat}
> 31 Aug 2016 12:52:24 ERROR [Thread-12] - run() exiting due to uncaught error
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
> at
> org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(TThreadedSelectorServer.java:310)
> at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:209)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:576)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:536)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2980) Automated concurrent Kafka offset migration test
[ https://issues.apache.org/jira/browse/FLUME-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-2980: Fix Version/s: 1.8.0 > Automated concurrent Kafka offset migration test > > > Key: FLUME-2980 > URL: https://issues.apache.org/jira/browse/FLUME-2980 > Project: Flume > Issue Type: Bug > Components: Kafka Channel >Affects Versions: 1.6.0 >Reporter: Mike Percy >Assignee: Umesh Chaudhary > Fix For: 1.8.0 > > Attachments: FLUME-2980-0.patch > > > The Kafka Channel needs an automated offset migration test. See also > FLUME-2972 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2976) Exception when JMS source tries to connect to a Weblogic server without authentication
[
https://issues.apache.org/jira/browse/FLUME-2976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2976:
Fix Version/s: 1.8.0
> Exception when JMS source tries to connect to a Weblogic server without
> authentication
> --
>
> Key: FLUME-2976
> URL: https://issues.apache.org/jira/browse/FLUME-2976
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
> Labels: jms
> Fix For: 1.8.0
>
> Attachments: FLUME-2976-2.patch, FLUME-2976.patch
>
>
> If no {{userName}} and {{passwordFile}} is set for the JMS source it sets the
> password to {{Optional("")}} (see:
> https://github.com/apache/flume/blob/trunk/flume-ng-sources/flume-jms-source/src/main/java/org/apache/flume/source/jms/JMSSource.java#L127)
> This leads to an exception in the weblogic jndi context implementation when
> trying to connect to a weblogic jms server.
> {noformat}
> java.lang.IllegalArgumentException: The 'java.naming.security.principal'
> property has not been specified
> at weblogic.jndi.Environment.getSecurityUser(Environment.java:562)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.pushSubject(WLInitialContextFactoryDelegate.java:665)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.newContext(WLInitialContextFactoryDelegate.java:485)
> at
> weblogic.jndi.WLInitialContextFactoryDelegate.getInitialContext(WLInitialContextFactoryDelegate.java:373)
> at weblogic.jndi.Environment.getContext(Environment.java:319)
> at weblogic.jndi.Environment.getContext(Environment.java:288)
> at
> weblogic.jndi.WLInitialContextFactory.getInitialContext(WLInitialContextFactory.java:117)
> at
> javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
> at
> javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:307)
> at javax.naming.InitialContext.init(InitialContext.java:242)
> at javax.naming.InitialContext.(InitialContext.java:216)
> at
> org.apache.flume.source.jms.InitialContextFactory.create(InitialContextFactory.java:28)
> at org.apache.flume.source.jms.JMSSource.doConfigure(JMSSource.java:223)
> {noformat}
> Changing the above mentioned line to {{Optional.absent()}} fixes the issue.
> [~brocknoland]: Is there any specific reason for setting the password to
> {{Optional("")}} when there is no {{passwordFile}} set?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2984) Default maxReadBufferBytes might cause OOM and cause thrift source exit
[
https://issues.apache.org/jira/browse/FLUME-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2984:
Fix Version/s: (was: 1.8.0)
> Default maxReadBufferBytes might cause OOM and cause thrift source exit
> ---
>
> Key: FLUME-2984
> URL: https://issues.apache.org/jira/browse/FLUME-2984
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Laxman
>Assignee: Laxman
> Attachments: FLUME-2984.patch
>
>
> Our usecase has multi agent flow and we are achieving this with thrift source
> and thrift sink combination. When there is a surge in incoming traffic
> downstream agent having thrift source is running out of memory. On further
> investigation of thrift transport/server code and scribe source found that we
> may need to fix the thrift transport's maxReadBufferBytes. Similar issue has
> been fixed in ScribeSource as part of FLUME-2404 by making this buffer
> configurable and capping it to a 16MB if not configured.
> *thrift code (org.apache.thrift.server.AbstractNonblockingServer)*
> {code}
> public abstract static class AbstractNonblockingServerArgs AbstractNonblockingServer.AbstractNonblockingServerArgs> extends
> AbstractServerArgs {
> public long maxReadBufferBytes = 9223372036854775807L;
> public AbstractNonblockingServerArgs(TNonblockingServerTransport
> transport) {
> super(transport);
> this.transportFactory(new Factory());
> }
> }
> {code}
> *stacktrace*
> {noformat}
> 31 Aug 2016 12:52:24 ERROR [Thread-12] - run() exiting due to uncaught error
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
> at
> org.apache.thrift.server.TThreadedSelectorServer.requestInvoke(TThreadedSelectorServer.java:310)
> at
> org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:209)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.select(TThreadedSelectorServer.java:576)
> at
> org.apache.thrift.server.TThreadedSelectorServer$SelectorThread.run(TThreadedSelectorServer.java:536)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2942) AvroEventDeserializer ignores header from spool source
[ https://issues.apache.org/jira/browse/FLUME-2942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-2942: Fix Version/s: 1.8.0 > AvroEventDeserializer ignores header from spool source > -- > > Key: FLUME-2942 > URL: https://issues.apache.org/jira/browse/FLUME-2942 > Project: Flume > Issue Type: Bug >Affects Versions: 1.6.0 >Reporter: Sebastian Alfers > Fix For: 1.8.0 > > Attachments: FLUME-2942-0.patch > > > I have a spool file source and use avro for de-/serialization > In detail, serialized events store the topic of the kafka sink in the header. > When I load the events from the spool directory, the header are ignored. > Please see: > https://github.com/apache/flume/blob/caa64a1a6d4bc97be5993cb468516e9ffe862794/flume-ng-core/src/main/java/org/apache/flume/serialization/AvroEventDeserializer.java#L122 > You can see, it uses the whole event as body but does not distinguish between > the header and body encoded by avro. > Please verify that this is a bug. > I fixed this but by using the record that stores header and body separately. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2894) Flume components should stop in the correct order (graceful shutdown)
[
https://issues.apache.org/jira/browse/FLUME-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2894:
Fix Version/s: 1.8.0
> Flume components should stop in the correct order (graceful shutdown)
> -
>
> Key: FLUME-2894
> URL: https://issues.apache.org/jira/browse/FLUME-2894
> Project: Flume
> Issue Type: Bug
> Components: Channel, Node, Sinks+Sources
>Affects Versions: 1.6.0, 1.7.0
>Reporter: Piotr Wiecek
>Assignee: Laxman
> Fix For: 1.8.0
>
> Attachments: FLUME-2894.patch
>
>
> Flume components should be stopped in the right way:
> * stop the sources (in order to not receiving further notifications),
> * wait until all events within the channels are consumed by the sinks,
> * stop the channels and the sinks.
> Currently, the shutdown hook stops the components in a random manner.
> E.g.: SINK, CHANNEL, SOURCE.
> Components are stored in the HashMap:
> {code:borderStyle=solid}
> Map supervisedProcesses;
> ...
> supervisedProcesses = new HashMap();
> ...
> @Override
> public synchronized void stop() {
> ...
> for (final Entry entry : supervisedProcesses
> .entrySet()) {
> if (entry.getKey().getLifecycleState().equals(LifecycleState.START)) {
> entry.getValue().status.desiredState = LifecycleState.STOP;
> entry.getKey().stop();
> }
> }
>
> {code}
> The problems which we can have:
> * not all Events will be consumed (Sink will be stopped first)
> * Source will continue to accept messages even though other components are
> stopped
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2989) Kafka Channel metrics missing eventTakeAttemptCount and eventPutAttemptCount
[
https://issues.apache.org/jira/browse/FLUME-2989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2989:
Fix Version/s: 1.8.0
> Kafka Channel metrics missing eventTakeAttemptCount and eventPutAttemptCount
>
>
> Key: FLUME-2989
> URL: https://issues.apache.org/jira/browse/FLUME-2989
> Project: Flume
> Issue Type: Bug
> Components: Kafka Channel
>Affects Versions: 1.5.0, 1.6.0, 1.5.1
>Reporter: Denes Arvay
>Assignee: Umesh Chaudhary
>Priority: Minor
> Fix For: 1.8.0
>
> Attachments: FLUME-2989-0.patch, FLUME-2989-1.patch
>
>
> {{eventTakeAttemptCount}} and {{eventPutAttemptCount}} counters don't get
> incremented in Kafka Channel
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2867) Spelling mistake in SinkCounter "sucess"
[ https://issues.apache.org/jira/browse/FLUME-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-2867: Labels: breaking_change easyfix patch (was: easyfix patch) > Spelling mistake in SinkCounter "sucess" > > > Key: FLUME-2867 > URL: https://issues.apache.org/jira/browse/FLUME-2867 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources >Affects Versions: 1.6.0 >Reporter: Jonathan Smith >Priority: Trivial > Labels: breaking_change, easyfix, patch > Attachments: sink_counter_typo.patch > > > org.apache.flume.instrumentation.SinkCounter.java "sink.event.drain.sucess" > -> success -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-542) S3 sink cannot be used if your AWS keys needs escaping
[
https://issues.apache.org/jira/browse/FLUME-542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-542:
---
Labels: possibly_obsolete (was: )
> S3 sink cannot be used if your AWS keys needs escaping
> --
>
> Key: FLUME-542
> URL: https://issues.apache.org/jira/browse/FLUME-542
> Project: Flume
> Issue Type: Bug
> Components: Node
>Affects Versions: 0.9.3
> Environment: Amazon EC2 Ubuntu 10.04 LTS
>Reporter: Disabled imported user
>Priority: Blocker
> Labels: possibly_obsolete
> Attachments: ASF.LICENSE.NOT.GRANTED--FLUME-542.log
>
>
> If you key or secret contains "/" characters, you are instructed to escape
> them (see http://wiki.apache.org/hadoop/AmazonS3).
> For example:
> {noformat}
> 19823791823/1232323/nm3nm
> {noformat}
> ...should be changed to:
> {noformat}
> 19823791823%2F1232323%2Fnm3nm
> {noformat}
> When using that in your sink configuration:
> {noformat}
> collectorSink(
> "s3n://key:19823791823%2F1232323%2Fnm3nm@str-flume-ec2production/%Y-%m-%d/%H00/",
> "nginx-" )
> {noformat}
> ...the Flume collector outputs this:
> {noformat}
> 2011-02-28 23:06:20,009 WARN com.cloudera.flume.core.Event: Unrecognized
> escape in event format string: %2
> {noformat}
> So then we tried double-escaping it:
> {noformat}
> 19823791823%%2F1232323%%2Fnm3nm
> {noformat}
> ...which passed the Flume escape check but now fails S3 auth.
> Rob
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-744) Update docs explaining difference between %t and %{timestamp} escape codes
[
https://issues.apache.org/jira/browse/FLUME-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-744:
---
Labels: possibly_obsolete (was: )
> Update docs explaining difference between %t and %{timestamp} escape codes
> --
>
> Key: FLUME-744
> URL: https://issues.apache.org/jira/browse/FLUME-744
> Project: Flume
> Issue Type: Documentation
> Components: Docs
>Affects Versions: 0.9.5
>Reporter: Jonathan Hsieh
>Priority: Minor
> Labels: possibly_obsolete
>
> FLUME-688 added a new escape code but did not update documentation describing
> the difference between two similar escapecodes -- %t and %{timestamp}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-750) Refactor NaiveFileWALManager
[ https://issues.apache.org/jira/browse/FLUME-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-750: --- Labels: possibly_obsolete (was: ) > Refactor NaiveFileWALManager > > > Key: FLUME-750 > URL: https://issues.apache.org/jira/browse/FLUME-750 > Project: Flume > Issue Type: Improvement >Affects Versions: 0.9.5 >Reporter: Jonathan Hsieh > Labels: possibly_obsolete > > As noted in the reviews for FLUME-745, the internal structure of > NaiveFileWalManager is very confusing. Initially, the WALManager assumed > that it would be a singleton but over time evolved so that there was on per > agent. The locking in this is critical but difficult to follow. It should be > refactored for clarity. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-813) Build Flume site from Maven
[ https://issues.apache.org/jira/browse/FLUME-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-813: --- Labels: possibly_complete (was: ) > Build Flume site from Maven > --- > > Key: FLUME-813 > URL: https://issues.apache.org/jira/browse/FLUME-813 > Project: Flume > Issue Type: Task > Components: Build >Affects Versions: NG alpha 1 >Reporter: E. Sammer > Labels: possibly_complete > > Enable and configure Flume site build from maven. Generate javadoc, > dependency info, authors / contributor data, etc. from pom. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-800) suffix of PathManager temp file should be changable
[ https://issues.apache.org/jira/browse/FLUME-800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-800: --- Labels: possibly_obsolete (was: ) > suffix of PathManager temp file should be changable > --- > > Key: FLUME-800 > URL: https://issues.apache.org/jira/browse/FLUME-800 > Project: Flume > Issue Type: Improvement >Affects Versions: 0.9.5 >Reporter: Sho Shimauchi > Labels: possibly_obsolete > > A temporary file is created when PathManager.open() is called but the suffix > of the file is hard-coded to ".tmp" > This hard-coding may cause problems with data received from different sources. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3116) Refactor ng-core module, move sources, channels and sinks out of it
Ferenc Szabo created FLUME-3116: --- Summary: Refactor ng-core module, move sources, channels and sinks out of it Key: FLUME-3116 URL: https://issues.apache.org/jira/browse/FLUME-3116 Project: Flume Issue Type: Improvement Components: Channel, Sinks+Sources Reporter: Ferenc Szabo Priority: Minor It would be more consistent to have the concrete source, channel and sink implementations in their own module under the flume-ng-sources, flume-ng-channels and flume-ng-sinks module. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3116) Refactor ng-core module, move sources, channels and sinks out of it
[ https://issues.apache.org/jira/browse/FLUME-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3116: Description: It would be more consistent to have the concrete source, channel and sink implementations in their own module under the flume-ng-sources, flume-ng-channels and flume-ng-sinks module. After moving the implementations out, the FLUME-829 package prefix refactor could also be done was:It would be more consistent to have the concrete source, channel and sink implementations in their own module under the flume-ng-sources, flume-ng-channels and flume-ng-sinks module. > Refactor ng-core module, move sources, channels and sinks out of it > --- > > Key: FLUME-3116 > URL: https://issues.apache.org/jira/browse/FLUME-3116 > Project: Flume > Issue Type: Improvement > Components: Channel, Sinks+Sources >Reporter: Ferenc Szabo >Priority: Minor > > It would be more consistent to have the concrete source, channel and sink > implementations in their own module under the flume-ng-sources, > flume-ng-channels and flume-ng-sinks module. > After moving the implementations out, the FLUME-829 package prefix refactor > could also be done -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-890) Add unit tests for flume ng channels
[ https://issues.apache.org/jira/browse/FLUME-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-890: --- Labels: possibly_complete (was: ) > Add unit tests for flume ng channels > > > Key: FLUME-890 > URL: https://issues.apache.org/jira/browse/FLUME-890 > Project: Flume > Issue Type: Improvement > Components: Channel >Affects Versions: NG alpha 2 >Reporter: Shu Zhang >Assignee: Shu Zhang > Labels: possibly_complete > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-893) Fix bin/flume-ng executable
[ https://issues.apache.org/jira/browse/FLUME-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-893: --- Labels: possibly_obsolete (was: ) > Fix bin/flume-ng executable > --- > > Key: FLUME-893 > URL: https://issues.apache.org/jira/browse/FLUME-893 > Project: Flume > Issue Type: Bug > Components: Test >Affects Versions: NG alpha 2 > Environment: ubuntu 10.10 VM >Reporter: Sid Askary >Assignee: E. Sammer >Priority: Minor > Labels: possibly_obsolete > > bin/flume-ng script needs to be updated to run a smoke test. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-918) customizable avro source/sink events
[ https://issues.apache.org/jira/browse/FLUME-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-918: --- Labels: possibly_obsolete (was: ) > customizable avro source/sink events > - > > Key: FLUME-918 > URL: https://issues.apache.org/jira/browse/FLUME-918 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Reporter: Karthik K >Assignee: Karthik K > Labels: possibly_obsolete > Attachments: FLUME-918.patch > > > Currently, flume ships with a flume.avdl that has a reference implementation > of a AvroSource / AvroSink , to deal with the default flume transports ( > Event / Channel - s) . > We need to customize the avro schema that goes in, for our purposes - but > would still like to reuse a majority of code of AvroSource / AvroSink that > does the marshalling/ unmarshalling. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-943) ThriftEventSource should set TCP keepalive on socket to avoid leaking connections and ultimately hitting ulimit (java.net.SocketException: Too many open files)
[ https://issues.apache.org/jira/browse/FLUME-943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-943: --- Labels: possibly_obsolete (was: ) > ThriftEventSource should set TCP keepalive on socket to avoid leaking > connections and ultimately hitting ulimit (java.net.SocketException: Too many > open files) > --- > > Key: FLUME-943 > URL: https://issues.apache.org/jira/browse/FLUME-943 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 0.9.5 > Environment: Linux >Reporter: Frank Grimes >Assignee: Frank Grimes > Labels: possibly_obsolete > Attachments: FLUME-943.patch > > > See > http://mail-archives.apache.org/mod_mbox/incubator-flume-user/201201.mbox/%3c716cd793-6cbc-4d08-b984-97daee71d...@yahoo.com%3e > Not sure if AvroEventSource is affected by this... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-961) Create a source decorator which allows one to specify the checkperiod (sleep_ms) when tailing a directory.
[ https://issues.apache.org/jira/browse/FLUME-961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-961: --- Labels: flume possibly_obsolete source tailDir (was: flume source tailDir) > Create a source decorator which allows one to specify the checkperiod > (sleep_ms) when tailing a directory. > -- > > Key: FLUME-961 > URL: https://issues.apache.org/jira/browse/FLUME-961 > Project: Flume > Issue Type: New Feature > Components: Sinks+Sources >Affects Versions: 0.9.5 > Environment: All environments which run the flume node would be > affected by this feature. >Reporter: Thomas Rega >Priority: Minor > Labels: flume, possibly_obsolete, source, tailDir > Original Estimate: 48h > Remaining Estimate: 48h > > Currently, the sleep_ms (or checkPeriod) in TailDirSource is set to a fixed > value of 250ms. Can we have a decorator which allows us to modify the amount > of time between checks? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-976) Git mirror of apache flume has a wrong HEAD ref
[ https://issues.apache.org/jira/browse/FLUME-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077255#comment-16077255 ] Ferenc Szabo commented on FLUME-976: [~qwertymaniac] Is this issue still present? > Git mirror of apache flume has a wrong HEAD ref > --- > > Key: FLUME-976 > URL: https://issues.apache.org/jira/browse/FLUME-976 > Project: Flume > Issue Type: Bug > Components: Build >Reporter: Harsh J >Priority: Minor > Labels: git, infrastructure, mirror > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-994) Support connection initiation in reverse direction
[ https://issues.apache.org/jira/browse/FLUME-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-994: --- Labels: possibl (was: ) > Support connection initiation in reverse direction > -- > > Key: FLUME-994 > URL: https://issues.apache.org/jira/browse/FLUME-994 > Project: Flume > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: Mike Percy > Labels: possibl > > Support initiating a network connection in the "reverse" direction: from Sink > to Source. > Use case: In an environment where data is being collected into the corp > network from the production network, often there is a firewall specifying > one-way connection initiatiion from corp to prod. This feature is to allow > connection initiation from corp to prod, and then allow data to flow through > that connection from a prod Source back to a corp Sink. > I know of one situation where Flume was considered for adoption, then > abandoned because this feature was not available. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1008) Implement Hive sink
[ https://issues.apache.org/jira/browse/FLUME-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-1008: Labels: possibly_complete (was: ) > Implement Hive sink > --- > > Key: FLUME-1008 > URL: https://issues.apache.org/jira/browse/FLUME-1008 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.1.0 >Reporter: Prasad Mujumdar >Assignee: Prasad Mujumdar > Labels: possibly_complete > > The HDFS sink's directory structure and bucketing is mostly sufficient for > Hive tables. The bucketing could result into new directories getting created. > This requrires altering Hive table to add a new partition for the directory. > This hive sink is basically to handle new directories getting added to a > target table. The functionality discussed in FLUME-990 will be tracked as > part of that ticket. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (FLUME-1044) Remove the configuration constants from the system into their own separate class
[ https://issues.apache.org/jira/browse/FLUME-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo reassigned FLUME-1044: --- Assignee: Ferenc Szabo > Remove the configuration constants from the system into their own separate > class > > > Key: FLUME-1044 > URL: https://issues.apache.org/jira/browse/FLUME-1044 > Project: Flume > Issue Type: Improvement >Reporter: Mike Percy >Assignee: Ferenc Szabo > > We want to do less of putting context constants into configurable classes as > strings and instead use constants/enums for these things. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (FLUME-3116) Refactor ng-core module, move sources, channels and sinks out of it
[ https://issues.apache.org/jira/browse/FLUME-3116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo reassigned FLUME-3116: --- Assignee: Ferenc Szabo > Refactor ng-core module, move sources, channels and sinks out of it > --- > > Key: FLUME-3116 > URL: https://issues.apache.org/jira/browse/FLUME-3116 > Project: Flume > Issue Type: Improvement > Components: Channel, Sinks+Sources >Reporter: Ferenc Szabo >Assignee: Ferenc Szabo >Priority: Minor > > It would be more consistent to have the concrete source, channel and sink > implementations in their own module under the flume-ng-sources, > flume-ng-channels and flume-ng-sinks module. > After moving the implementations out, the FLUME-829 package prefix refactor > could also be done -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1111) Expose Flume metrics for monitoring flume in production
[ https://issues.apache.org/jira/browse/FLUME-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-: Labels: possibly_complete (was: ) > Expose Flume metrics for monitoring flume in production > --- > > Key: FLUME- > URL: https://issues.apache.org/jira/browse/FLUME- > Project: Flume > Issue Type: New Feature >Reporter: Inder SIngh > Labels: possibly_complete > > Folks, > Opening this one to start discussion on exposing different stats across FLUME > i.e. sources, sinks, channels for monitoring. > Is there a plan in pipeline? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1144) Update Flume User Guide with a section on Flume properties such as flume.called.from.service
[ https://issues.apache.org/jira/browse/FLUME-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-1144: Labels: possibly_obsolete (was: ) > Update Flume User Guide with a section on Flume properties such as > flume.called.from.service > > > Key: FLUME-1144 > URL: https://issues.apache.org/jira/browse/FLUME-1144 > Project: Flume > Issue Type: Documentation > Components: Docs >Affects Versions: 1.2.0 >Reporter: Will McQueen > Labels: possibly_obsolete > > Flume User Guide is at flume-ng-doc/xhtml/FlumeUserGuide.xhtml -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1168) Support retries in log4jappender
[ https://issues.apache.org/jira/browse/FLUME-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-1168: Labels: possibly_complete (was: ) > Support retries in log4jappender > > > Key: FLUME-1168 > URL: https://issues.apache.org/jira/browse/FLUME-1168 > Project: Flume > Issue Type: Improvement > Components: Sinks+Sources >Affects Versions: 1.2.0 >Reporter: Hari Shreedharan >Assignee: Ashish Paliwal >Priority: Minor > Labels: possibly_complete > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1293) NIO channel selector open throws "Too many open files" exception
[
https://issues.apache.org/jira/browse/FLUME-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-1293:
Labels: newbie possibly_obsolete (was: newbie)
> NIO channel selector open throws "Too many open files" exception
>
>
> Key: FLUME-1293
> URL: https://issues.apache.org/jira/browse/FLUME-1293
> Project: Flume
> Issue Type: Wish
> Components: Docs
>Affects Versions: 1.1.0
> Environment: {code}
> -bash-3.2$ java -version
> java version "1.6.0_24"
> Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
> Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
> -bash-3.2$ uname -a
> Linux test.test.com 2.6.18-194.26.1.0.1.el5 #1 SMP Tue Nov 9 18:48:15 EST
> 2010 x86_64 x86_64 x86_64 GNU/Linux
> {code}
>Reporter: Mubarak Seyed
>Priority: Minor
> Labels: newbie, possibly_obsolete
> Attachments: flume-1293-stacktrace.txt
>
>
> {code}
> 2012-06-19 19:01:21,998 ERROR api.NettyAvroRpcClient: RPC connection error :
> java.io.IOException: Error connecting to /x.x.x.x:9090
> at
> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249)
> at
> org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:198)
> at
> org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:147)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:115)
> at
> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:104)
> at
> org.apache.flume.api.NettyAvroRpcClient.(NettyAvroRpcClient.java:89)
> at
> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:131)
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:148)
> at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:176)
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:231)
> at
> org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:162)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.jboss.netty.channel.ChannelException: Failed to create a
> selector.
> at
> org.jboss.netty.channel.socket.nio.NioWorker.register(NioWorker.java:105)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:403)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370)
> at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
> Caused by: java.io.IOException: Too many open files
> at sun.nio.ch.IOUtil.initPipe(Native Method)
> at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
> at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> at java.nio.channels.Selector.open(Selector.java:209)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.register(NioWorker.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLUME-1375) Add unit tests for ganglia and custom monitoring.
[ https://issues.apache.org/jira/browse/FLUME-1375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo reassigned FLUME-1375: --- Assignee: Ferenc Szabo > Add unit tests for ganglia and custom monitoring. > - > > Key: FLUME-1375 > URL: https://issues.apache.org/jira/browse/FLUME-1375 > Project: Flume > Issue Type: Test >Reporter: Hari Shreedharan >Assignee: Ferenc Szabo > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1404) Cannot build Apache Flume 1.2.0 with profile hadoop23
[
https://issues.apache.org/jira/browse/FLUME-1404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-1404:
Labels: bigtop possibly_obsolete (was: bigtop)
> Cannot build Apache Flume 1.2.0 with profile hadoop23
> -
>
> Key: FLUME-1404
> URL: https://issues.apache.org/jira/browse/FLUME-1404
> Project: Flume
> Issue Type: Bug
> Components: Build
>Affects Versions: 1.2.0
>Reporter: Bruno Mahé
> Labels: bigtop, possibly_obsolete
>
> Here is the output:
> {noformat}
> [bruno@p8700 apache-flume-1.2.0]$ mvn -DskipTests -Dhadoop.profile=23 clean
> install
> /usr/lib/jvm/java
> [INFO] Scanning for projects...
> [ERROR] The build could not read 1 project -> [Help 1]
> [ERROR]
> [ERROR] The project
> org.apache.flume.flume-ng-sinks:flume-ng-hbase-sink:1.2.0
> (/home/bruno/projects/bigtop-git-svn/temp/apache-flume-1.2.0/flume-ng-sinks/flume-ng-hbase-sink/pom.xml)
> has 1 error
> [ERROR] 'dependencies.dependency.version' for
> org.apache.hadoop:hadoop-test:jar is missing. @ line 88, column 17
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
> {noformat}
> From http://mvnrepository.com/artifact/org.apache.hadoop/hadoop-test only
> Apache Hadoop 1.X has a hadoop-test artefact.
> This is also why hadoop-test version is only defined for the default Apache
> Hadoop 1.X profile and not the 0.23 one
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-1551) Create a home for custom Flume components
[ https://issues.apache.org/jira/browse/FLUME-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077328#comment-16077328 ] Ferenc Szabo commented on FLUME-1551: - Looks like flume has this feature already: https://flume.apache.org/FlumeUserGuide.html#installing-third-party-plugins [~w...@cloudera.com] does this resolve this ticket? > Create a home for custom Flume components > - > > Key: FLUME-1551 > URL: https://issues.apache.org/jira/browse/FLUME-1551 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.3.0 >Reporter: Will McQueen > Labels: possibly_complete > > Create an 'ext' dir where all Flume NG custom components can live (eg, custom > channels, sources, sinks, interceptors, serializers). Modify the flume-ng > script to include this 'ext' dir in the classpath (probably include it right > after the lib dir). Doing this will standardize the location of custom > components. > An manual alternative is for a user to create the 'ext' dir and include that > dir in the flume-env.sh file's FLUME_CLASSPATH value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1551) Create a home for custom Flume components
[ https://issues.apache.org/jira/browse/FLUME-1551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-1551: Labels: possibly_complete (was: ) > Create a home for custom Flume components > - > > Key: FLUME-1551 > URL: https://issues.apache.org/jira/browse/FLUME-1551 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.3.0 >Reporter: Will McQueen > Labels: possibly_complete > > Create an 'ext' dir where all Flume NG custom components can live (eg, custom > channels, sources, sinks, interceptors, serializers). Modify the flume-ng > script to include this 'ext' dir in the classpath (probably include it right > after the lib dir). Doing this will standardize the location of custom > components. > An manual alternative is for a user to create the 'ext' dir and include that > dir in the flume-env.sh file's FLUME_CLASSPATH value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-1584) Document load balancing RPC client
[ https://issues.apache.org/jira/browse/FLUME-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077335#comment-16077335 ] Ferenc Szabo commented on FLUME-1584: - It looks documented for me: https://flume.apache.org/FlumeDeveloperGuide.html#loadbalancing-rpc-client [~mpercy] does this resolve this ticket? > Document load balancing RPC client > -- > > Key: FLUME-1584 > URL: https://issues.apache.org/jira/browse/FLUME-1584 > Project: Flume > Issue Type: Documentation >Reporter: Mike Percy > > We need to document the load balancing RPC client -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1584) Document load balancing RPC client
[ https://issues.apache.org/jira/browse/FLUME-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-1584: Labels: possibly_complete (was: ) > Document load balancing RPC client > -- > > Key: FLUME-1584 > URL: https://issues.apache.org/jira/browse/FLUME-1584 > Project: Flume > Issue Type: Documentation >Reporter: Mike Percy > Labels: possibly_complete > > We need to document the load balancing RPC client -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-1944) parameterising the dependency variables in pom file.
[ https://issues.apache.org/jira/browse/FLUME-1944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-1944: Labels: possibly_complete (was: ) > parameterising the dependency variables in pom file. > > > Key: FLUME-1944 > URL: https://issues.apache.org/jira/browse/FLUME-1944 > Project: Flume > Issue Type: Bug > Components: Build >Affects Versions: 1.3.1 > Environment: Build overriding versions. >Reporter: Ashish Singh > Labels: possibly_complete > Attachments: FLUME-1944.patch > > > Making the dependency version driven from properties. This will require > change of version only in property section. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093618#comment-16093618
]
Ferenc Szabo commented on FLUME-3131:
-
[~sati]
In this case I would recommend to change the activemq dependency to have a test
scope because it is only used in one test, than the vulnerability is not going
to be present in production.
For the `javax.jms.*` packages use the following dependency:
{code}
org.apache.geronimo.specs
geronimo-jms_1.1_spec
1.1.1
{code}
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo reassigned FLUME-3131:
---
Assignee: Ferenc Szabo
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093618#comment-16093618
]
Ferenc Szabo edited comment on FLUME-3131 at 7/20/17 10:56 PM:
---
[~sati]
In this case I would recommend to change the activemq dependency to have a test
scope because it is only used in one test, than the vulnerability is not going
to be present in production.
For the `javax.jms.*` packages use the following dependency:
{code}
javax.jms
jms
{code}
was (Author: fszabo):
[~sati]
In this case I would recommend to change the activemq dependency to have a test
scope because it is only used in one test, than the vulnerability is not going
to be present in production.
For the `javax.jms.*` packages use the following dependency:
{code}
org.apache.geronimo.specs
geronimo-jms_1.1_spec
1.1.1
{code}
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093618#comment-16093618
]
Ferenc Szabo edited comment on FLUME-3131 at 7/20/17 11:23 PM:
---
[~sati]
In this case I would recommend to change the activemq dependency to have a test
scope because it is only used in one test, than the vulnerability is not going
to be present in production.
For the `javax.jms.*` packages use the following dependency:
{code}
org.apache.geronimo.specs
geronimo-jms_1.1_spec
1.1.1
{code}
was (Author: fszabo):
[~sati]
In this case I would recommend to change the activemq dependency to have a test
scope because it is only used in one test, than the vulnerability is not going
to be present in production.
For the `javax.jms.*` packages use the following dependency:
{code}
javax.jms
jms
{code}
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-3131:
Attachment: FLUME-3131.patch
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-3131:
Attachment: FLUME-3131-1.patch
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131-1.patch, FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3131) Upgrade spring framework library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16097596#comment-16097596
]
Ferenc Szabo commented on FLUME-3131:
-
[~sati] thanks for your comment, I have uploaded an updated patch.
Review board: https://reviews.apache.org/r/61014/
> Upgrade spring framework library dependencies
> -
>
> Key: FLUME-3131
> URL: https://issues.apache.org/jira/browse/FLUME-3131
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
> Attachments: FLUME-3131-1.patch, FLUME-3131.patch
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.springframework|spring-aop|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-context|3.0.7.RELEASE|4.3.9.RELEASE,|
> |org.springframework|spring-core|3.0.7.RELEASE|4.3.9.RELEASE,|
> Security vulnerability:
> https://www.cvedetails.com/vulnerability-list/vendor_id-9664/product_id-17274/Springsource-Spring-Framework.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.springframework/spring-aop
> - https://mvnrepository.com/artifact/org.springframework/spring-context
> - https://mvnrepository.com/artifact/org.springframework/spring-core
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume.flume-ng-sources:flume-jms-source:jar:1.8.0-SNAPSHOT
> \- org.apache.activemq:activemq-core:jar:5.7.0:provided
>+- org.springframework:spring-context:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-aop:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-beans:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-core:jar:3.0.7.RELEASE:provided
>| +- org.springframework:spring-expression:jar:3.0.7.RELEASE:provided
>| \- org.springframework:spring-asm:jar:3.0.7.RELEASE:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLUME-3132) Upgrade tomcat jasper library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo reassigned FLUME-3132:
---
Assignee: Ferenc Szabo
> Upgrade tomcat jasper library dependencies
> --
>
> Key: FLUME-3132
> URL: https://issues.apache.org/jira/browse/FLUME-3132
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |tomcat|jasper-compiler|5.5.23|8.5.x|
> |tomcat|jasper-runtime|5.5.23|8.5.x|
> Security vulnerability:
> - https://www.cvedetails.com/cve/CVE-2011-1318/
> -
> http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
> Note: These artifacts were moved to:
> * New Group org.apache.tomcat
> * New Artifact
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
> +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> | +- tomcat:jasper-compiler:jar:5.5.23:runtime
> | +- tomcat:jasper-runtime:jar:5.5.23:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3132) Upgrade tomcat jasper library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103698#comment-16103698
]
Ferenc Szabo commented on FLUME-3132:
-
I am afraid that this version upgrade is not possible, because the dependency
comes with {{hadoop-common}} and the jasper artifacts moved to a different
group with a different artifactId.
https://mvnrepository.com/artifact/tomcat/jasper-compiler
->
https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
We have the latest version possible.
We can increase the hadoop-common version to get rid of jasper from the 2.7.0
there is no jasper lib.
However with hadoop-common version upgrade we would have a lot of other
dependency change.
Should we go with the new hadoop-common or with excluding the jasper from the
dependencies, because it looks like we do not need them.
> Upgrade tomcat jasper library dependencies
> --
>
> Key: FLUME-3132
> URL: https://issues.apache.org/jira/browse/FLUME-3132
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |tomcat|jasper-compiler|5.5.23|8.5.x|
> |tomcat|jasper-runtime|5.5.23|8.5.x|
> Security vulnerability:
> - https://www.cvedetails.com/cve/CVE-2011-1318/
> -
> http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
> Note: These artifacts were moved to:
> * New Group org.apache.tomcat
> * New Artifact
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
> +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> | +- tomcat:jasper-compiler:jar:5.5.23:runtime
> | +- tomcat:jasper-runtime:jar:5.5.23:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3139) Resolve tilde (~) character in file channel config as the users home directory
Ferenc Szabo created FLUME-3139: --- Summary: Resolve tilde (~) character in file channel config as the users home directory Key: FLUME-3139 URL: https://issues.apache.org/jira/browse/FLUME-3139 Project: Flume Issue Type: Bug Components: File Channel Affects Versions: 1.7.0 Reporter: Ferenc Szabo Fix For: 1.8.0 Paths in the following config properties are not resolved properly checkpointDir backupCheckpointDir dataDir expected behaviour: ~/myDataDir should be resolved as /home//myDataDir actual behaviour: ~/myDataDir is resolved as /myDataDir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (FLUME-3127) Upgrade libfb303 library dependency
[
https://issues.apache.org/jira/browse/FLUME-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo reassigned FLUME-3127:
---
Assignee: Ferenc Szabo
> Upgrade libfb303 library dependency
> ---
>
> Key: FLUME-3127
> URL: https://issues.apache.org/jira/browse/FLUME-3127
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |org.apache.thrift|libthrift|0.9.0|0.9.3,0.10.0|
> |org.apache.thrift|libfb303|0.9.0|0.9.3|
> Security vulnerability: http://www.cvedetails.com/cve/CVE-2015-3254/
> Maven repository:
> - https://mvnrepository.com/artifact/org.apache.thrift/libthrift
> - https://mvnrepository.com/artifact/org.apache.thrift/libfb303
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-sdk:jar:1.8.0-SNAPSHOT
> \- org.apache.thrift:libthrift:jar:0.9.0:compile
> org.apache.flume.flume-ng-sinks:flume-hive-sink:jar:1.8.0-SNAPSHOT
> +- org.apache.hive.hcatalog:hive-hcatalog-streaming:jar:1.0.0:provided
> | +- org.apache.hive:hive-metastore:jar:1.0.0:provided
> | | \- org.apache.thrift:libfb303:jar:0.9.0:provided
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (FLUME-3112) Upgrade jackson-core library dependency
[ https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo reassigned FLUME-3112: --- Assignee: Ferenc Szabo > Upgrade jackson-core library dependency > --- > > Key: FLUME-3112 > URL: https://issues.apache.org/jira/browse/FLUME-3112 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > > ||Group||Artifact||Version used||Upgrade target|| > |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9| > Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3112) Upgrade jackson-core library dependency
[ https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3112: Attachment: FLUME-3112.patch > Upgrade jackson-core library dependency > --- > > Key: FLUME-3112 > URL: https://issues.apache.org/jira/browse/FLUME-3112 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > Attachments: FLUME-3112.patch > > > ||Group||Artifact||Version used||Upgrade target|| > |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9| > Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3143) Fix undeclared and unused dependencies
Ferenc Szabo created FLUME-3143:
---
Summary: Fix undeclared and unused dependencies
Key: FLUME-3143
URL: https://issues.apache.org/jira/browse/FLUME-3143
Project: Flume
Issue Type: Bug
Affects Versions: 1.7.0
Reporter: Ferenc Szabo
Assignee: Ferenc Szabo
Fix For: 1.8.0
Dependency analysis shows some warnings in the project that should be fixed
{noformat}
[INFO]
[INFO] Building Flume NG SDK 1.8.0-SNAPSHOT
[INFO]
[WARNING] Used undeclared dependencies found:
[WARNING]commons-lang:commons-lang:jar:2.5:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
[INFO]
[INFO]
[INFO] Building Flume NG Configuration 1.8.0-SNAPSHOT
[INFO]
[WARNING] Unused declared dependencies found:
[WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
[INFO]
[INFO]
[INFO] Building Flume Auth 1.8.0-SNAPSHOT
[INFO]
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.thrift:libthrift:jar:0.9.0:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
[INFO]
[INFO]
[INFO] Building Flume NG Core 1.8.0-SNAPSHOT
[INFO]
[WARNING] Used undeclared dependencies found:
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]org.apache.httpcomponents:httpcore:jar:4.2.1:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[WARNING]log4j:log4j:jar:1.2.17:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
[INFO]
[INFO]
[INFO] Building Flume NG HDFS Sink 1.8.0-SNAPSHOT
[INFO]
[WARNING] Used undeclared dependencies found:
[WARNING]org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.4.0:test
[WARNING]org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT:compile
[WARNING]org.apache.avro:avro:jar:1.7.4:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.apache.hadoop:hadoop-minicluster:jar:2.4.0:test
[WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
[WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
[INFO]
[INFO]
[INFO] Building Flume NG IRC Sink 1.8.0-SNAPSHOT
[INFO]
[WARNING] Used undeclared dependencies found:
[WARNING]com.google.guava:guava:jar:11.0.2:compile
[WARNING]commons-io:commons-io:jar:2.1:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
[INFO]
[INFO]
[INFO] Building Flume NG JDBC channel 1.8.0-SNAPSHOT
[INFO]
[WARNING] Used undeclared dependencies found:
[WARNING]org.slf4j:slf4j-api:jar:1.6.1:compile
[WARNING]commons-pool:commons-pool:jar:1.5.4:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
[WARNING]org.apache.derby:derby:jar:10.11.1.1:compile
[INFO]
[INFO]
[INFO] Building Flume NG file-based channel 1.8.0-SNAPSHOT
[INFO]
[WARNING] Used undeclared dependencies found:
[WARNING]com.google.guava:guava:jar:11.0.2:compile
[WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
[WARNING]commons-io:commons-io:jar:2.1:compile
[WARNING]commons-cli:commons-cli:jar:1.2:compile
[WARNING] Unused declared dependencies found:
[WARNING]org.slf4j:slf4j-log
[jira] [Created] (FLUME-3148) Make tests run in parallel
Ferenc Szabo created FLUME-3148: --- Summary: Make tests run in parallel Key: FLUME-3148 URL: https://issues.apache.org/jira/browse/FLUME-3148 Project: Flume Issue Type: Improvement Reporter: Ferenc Szabo Assignee: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3112) Upgrade jackson-core library dependency
[ https://issues.apache.org/jira/browse/FLUME-3112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128753#comment-16128753 ] Ferenc Szabo commented on FLUME-3112: - Done. Thanks! > Upgrade jackson-core library dependency > --- > > Key: FLUME-3112 > URL: https://issues.apache.org/jira/browse/FLUME-3112 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > Attachments: FLUME-3112.patch > > > ||Group||Artifact||Version used||Upgrade target|| > |com.fasterxml.jackson.core|jackson-core|2.3.1|2.8.9| > Security vulnerability: http://www.cvedetails.com/cve/CVE-2016-7051/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2973) Deadlock in hdfs sink
[
https://issues.apache.org/jira/browse/FLUME-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2973:
Labels: hdfssink on-kanban-board (was: hdfssink)
> Deadlock in hdfs sink
> -
>
> Key: FLUME-2973
> URL: https://issues.apache.org/jira/browse/FLUME-2973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Critical
> Labels: hdfssink, on-kanban-board
> Fix For: 1.8.0
>
> Attachments: FLUME-2973-1.patch, FLUME-2973.patch
>
>
> Automatic close of BucketWriters (when open file count reached
> {{hdfs.maxOpenFiles}}) and the file rolling thread can end up in deadlock.
> When creating a new {{BucketWriter}} in {{HDFSEventSink}} it locks
> {{HDFSEventSink.sfWritersLock}} and the {{close()}} called in
> {{HDFSEventSink.sfWritersLock.removeEldestEntry}} tries to lock the
> {{BucketWriter}} instance.
> On the other hand if the file is being rolled in
> {{BucketWriter.close(boolean)}} it locks the {{BucketWriter}} instance first
> and in the close callback it tries to lock the {{sfWritersLock}}.
> The chances for this deadlock is higher when the {{hdfs.maxOpenFiles}}'s
> value is low (1).
> Script to reproduce:
> https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
> (put to
> {{flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs}})
> Deadlock usually occurs before ~30 iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2973) Deadlock in hdfs sink
[
https://issues.apache.org/jira/browse/FLUME-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2973:
Labels: hdfssink (was: hdfssink on-kanban-board)
> Deadlock in hdfs sink
> -
>
> Key: FLUME-2973
> URL: https://issues.apache.org/jira/browse/FLUME-2973
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
>Affects Versions: 1.7.0
>Reporter: Denes Arvay
>Assignee: Denes Arvay
>Priority: Critical
> Labels: hdfssink
> Fix For: 1.8.0
>
> Attachments: FLUME-2973-1.patch, FLUME-2973.patch
>
>
> Automatic close of BucketWriters (when open file count reached
> {{hdfs.maxOpenFiles}}) and the file rolling thread can end up in deadlock.
> When creating a new {{BucketWriter}} in {{HDFSEventSink}} it locks
> {{HDFSEventSink.sfWritersLock}} and the {{close()}} called in
> {{HDFSEventSink.sfWritersLock.removeEldestEntry}} tries to lock the
> {{BucketWriter}} instance.
> On the other hand if the file is being rolled in
> {{BucketWriter.close(boolean)}} it locks the {{BucketWriter}} instance first
> and in the close callback it tries to lock the {{sfWritersLock}}.
> The chances for this deadlock is higher when the {{hdfs.maxOpenFiles}}'s
> value is low (1).
> Script to reproduce:
> https://gist.github.com/adenes/96503a6e737f9604ab3ee9397a5809ff
> (put to
> {{flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs}})
> Deadlock usually occurs before ~30 iterations.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3150) Upgrade Kite version
Ferenc Szabo created FLUME-3150: --- Summary: Upgrade Kite version Key: FLUME-3150 URL: https://issues.apache.org/jira/browse/FLUME-3150 Project: Flume Issue Type: Bug Reporter: Ferenc Szabo Assignee: Ferenc Szabo Multiple security vulnerabilities comes from Kite update them with a Kite upgrade -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (FLUME-3115) Upgrade netty library dependency
[ https://issues.apache.org/jira/browse/FLUME-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo reassigned FLUME-3115: --- Assignee: Ferenc Szabo > Upgrade netty library dependency > > > Key: FLUME-3115 > URL: https://issues.apache.org/jira/browse/FLUME-3115 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > > ||Group||Artifact||Version used||Upgrade target|| > |io.netty|netty|3.2.2.Final, 3.9.4.Final|4.1.12.Final| > Note: This artifact was moved to: > - New Group io.netty > - New Artifactnetty-all > Security vulnerability: http://www.cvedetails.com/cve/CVE-2014-3488/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3132) Upgrade tomcat jasper library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-3132:
Labels: dependency hadoop-common_upgrade_needed (was: dependency)
> Upgrade tomcat jasper library dependencies
> --
>
> Key: FLUME-3132
> URL: https://issues.apache.org/jira/browse/FLUME-3132
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency, hadoop-common_upgrade_needed
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |tomcat|jasper-compiler|5.5.23|8.5.x|
> |tomcat|jasper-runtime|5.5.23|8.5.x|
> Security vulnerability:
> - https://www.cvedetails.com/cve/CVE-2011-1318/
> -
> http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
> Note: These artifacts were moved to:
> * New Group org.apache.tomcat
> * New Artifact
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
> +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> | +- tomcat:jasper-compiler:jar:5.5.23:runtime
> | +- tomcat:jasper-runtime:jar:5.5.23:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLUME-3151) Upgrade Hadoop common version
Ferenc Szabo created FLUME-3151: --- Summary: Upgrade Hadoop common version Key: FLUME-3151 URL: https://issues.apache.org/jira/browse/FLUME-3151 Project: Flume Issue Type: Bug Reporter: Ferenc Szabo Some security vulnerability comes from the hadoop-common library, so it should be updated -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3132) Upgrade tomcat jasper library dependencies
[
https://issues.apache.org/jira/browse/FLUME-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-3132:
Labels: dependency (was: dependency hadoop-common_upgrade_needed)
> Upgrade tomcat jasper library dependencies
> --
>
> Key: FLUME-3132
> URL: https://issues.apache.org/jira/browse/FLUME-3132
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Attila Simon
>Assignee: Ferenc Szabo
>Priority: Critical
> Labels: dependency
> Fix For: 1.8.0
>
>
> ||Group||Artifact||Version used||Upgrade target||
> |tomcat|jasper-compiler|5.5.23|8.5.x|
> |tomcat|jasper-runtime|5.5.23|8.5.x|
> Security vulnerability:
> - https://www.cvedetails.com/cve/CVE-2011-1318/
> -
> http://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-887/Apache-Tomcat.html
> Maven repositories:
> - https://mvnrepository.com/artifact/org.apache.tomcat/tomcat-jasper
> Note: These artifacts were moved to:
> * New Group org.apache.tomcat
> * New Artifact
> Please do:
> - CVE might be a false alarm or mistake. Please double check.
> - double check the newest version.
> - consider to remove a dependency if better alternative is available.
> - check whether the lib change would introduce a backward incompatibility (in
> which case please add this label `breaking_change` and fix version should be
> the next major)
> Excerpt from mvn dependency:tree
> {noformat}
> org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT
> +- org.apache.hadoop:hadoop-common:jar:2.4.0:compile
> | +- tomcat:jasper-compiler:jar:5.5.23:runtime
> | +- tomcat:jasper-runtime:jar:5.5.23:runtime
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3115) Upgrade netty library dependency
[ https://issues.apache.org/jira/browse/FLUME-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16133418#comment-16133418 ] Ferenc Szabo commented on FLUME-3115: - netty 3.x.x needed for avro-ipc so we need to have that version untill it is upgraded. Unfortunately it is part of the public api, so the next opportunity to upgrade that is the next mayor release. However the latest 3.x.x version does not have the security vulnerability so the upgrade there is an option. netty-all 4.x.x is in the hbase-client the latest version of that is also looks ok in the aspect of security vulnerabilities. There are some components that use netty 3.x.x and could be updated to netty-all 4.x.x. I will create JIRA issues to extract them to a single submodule and refactor them to use netty 4. Until that they can work with the latest 3.x.x > Upgrade netty library dependency > > > Key: FLUME-3115 > URL: https://issues.apache.org/jira/browse/FLUME-3115 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.0 > > > ||Group||Artifact||Version used||Upgrade target|| > |io.netty|netty|3.2.2.Final, 3.9.4.Final|4.1.12.Final| > Note: This artifact was moved to: > - New Group io.netty > - New Artifactnetty-all > Security vulnerability: http://www.cvedetails.com/cve/CVE-2014-3488/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3143) Fix undeclared and unused dependencies
[
https://issues.apache.org/jira/browse/FLUME-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16134245#comment-16134245
]
Ferenc Szabo commented on FLUME-3143:
-
Please show a specific example where the report is wrong. If the slf4j
artifact is used by the tests and listed as compile, then it is unused from the
final jars point of view.
ignoreNonCompile: Ignore Runtime/Provided/Test/System scopes for unused
dependency analysis.
In my understanding it would just not list unused dependencies declared
anything else than compile scope. This might make the analysis faster, however
it would let the non compile scopes to have unused dependencies.
> Fix undeclared and unused dependencies
> --
>
> Key: FLUME-3143
> URL: https://issues.apache.org/jira/browse/FLUME-3143
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Ferenc Szabo
>Assignee: Ferenc Szabo
> Labels: dependency
> Fix For: 1.8.0
>
>
> Dependency analysis shows some warnings in the project that should be fixed
> {noformat}
> [INFO]
>
> [INFO] Building Flume NG SDK 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-lang:commons-lang:jar:2.5:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Configuration 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume Auth 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.thrift:libthrift:jar:0.9.0:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Core 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]org.apache.httpcomponents:httpcore:jar:4.2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [WARNING]log4j:log4j:jar:1.2.17:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG HDFS Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.4.0:test
> [WARNING]org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT:compile
> [WARNING]org.apache.avro:avro:jar:1.7.4:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-minicluster:jar:2.4.0:test
> [WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG IRC Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.guava:guava:jar:11.0.2:compile
> [WARNING]commons-io:commons-io:jar:2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG JDBC channel 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared depe
[jira] [Created] (FLUME-3152) Add Flume Metric for Backup Checkpoint Errors
Ferenc Szabo created FLUME-3152: --- Summary: Add Flume Metric for Backup Checkpoint Errors Key: FLUME-3152 URL: https://issues.apache.org/jira/browse/FLUME-3152 Project: Flume Issue Type: Bug Components: File Channel Affects Versions: 1.7.0 Reporter: Ferenc Szabo Assignee: Ferenc Szabo Fix For: 1.8.0 File channel metrics should contain a counter for checkpoint backup errors -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (FLUME-2854) Parameterize jetty version in pom
[ https://issues.apache.org/jira/browse/FLUME-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo resolved FLUME-2854. - Resolution: Implemented Fix Version/s: 1.8.0 This issue is implemented in FLUME-3093. > Parameterize jetty version in pom > - > > Key: FLUME-2854 > URL: https://issues.apache.org/jira/browse/FLUME-2854 > Project: Flume > Issue Type: Bug >Reporter: Sriharsha Chintalapani >Assignee: Sriharsha Chintalapani > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3149) reduce cpu cost for file source transfer while still maintaining reliability
[ https://issues.apache.org/jira/browse/FLUME-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142772#comment-16142772 ] Ferenc Szabo commented on FLUME-3149: - [~zyfo2] I have looked at your PR and I saw that you changed the memory channel in order to make the taildir source use less CPU. The thing is, that none of the channel or transaction implementations should have any information about any source or sink detail. These are interfaces that are working together while the implementations are independent and must not share anything. If you would like to optimize the tailDirSource, the only implementation that can be changed is the tailDirSource itself and not any of the channels. Have you tried to experiment with for example larger batch size in your config? You wrote, you have a kafka sink. Why don't you just use a kafka channel instead? That way you could skip the whole channel CPU usage issue. > reduce cpu cost for file source transfer while still maintaining reliability > > > Key: FLUME-3149 > URL: https://issues.apache.org/jira/browse/FLUME-3149 > Project: Flume > Issue Type: Improvement > Components: File Channel >Reporter: will zhang > > File channel tracks transferred events and use transnational mechanism to > make transfer recoverable. However, it increases CPU cost due to frequent > system calls like write, read, etc. The Cpu cost could be very high if the > transfer rate is high. In contrast, Memory channel has no such issue which > requires only about 10% of CPU cost in the same environment but it's not > recovered if the system is down accidentally. > For sources like taildir/spooldir, I propose we could track offsets of file > and store them locally to achieve reliability while still using memory > channel to reduce CPU cost. Actually, I have already implemented this feature > by storing the offsets in event headers and passing it to my own > "offsetMemoryChannel" and store theses offsets in local disk in our > production which reduces CPU cost by about 90 percent. > Please let me know if it's worthwhile to have this feature in community > version. Thank you. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2698) Upgrade Jetty Version
[
https://issues.apache.org/jira/browse/FLUME-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2698:
Attachment: HttpSourceLoadTest.jmx
jetty-9-https-test.csv
jetty-9-http-test.csv
jetty-6-http-test.csv
jetty-6-https-test.csv
> Upgrade Jetty Version
> -
>
> Key: FLUME-2698
> URL: https://issues.apache.org/jira/browse/FLUME-2698
> Project: Flume
> Issue Type: Bug
> Components: Web
>Affects Versions: 1.6.0, 1.5.1, 1.7.0
>Reporter: Joakim Erdfelt
>Assignee: Tristan Stevens
> Labels: dependency
> Attachments: HttpSourceLoadTest.jmx, jetty-6-https-test.csv,
> jetty-6-http-test.csv, jetty-9-https-test.csv, jetty-9-http-test.csv
>
>
> Flume depends on Jetty 6
> {code:xml}
>
> org.mortbay.jetty
> jetty-util
> 6.1.26
>
> {code}
> Which was EOL (End of Life) back in 2010 and is no longer fit for production
> use (without heavy customizations and modifications like Google does for GAE,
> just to keep it safe and vulnerability free)
> Jetty was moved to Eclipse.org back during the Jetty 7 days.
> http://eclipse.org/jetty/
> Note that [Jetty 7 and Jetty 8 are now also
> EOL|https://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00069.html] (as
> of 2014)
> Jetty 9 is the only stable and supported version of Jetty now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-2698) Upgrade Jetty Version
[
https://issues.apache.org/jira/browse/FLUME-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144281#comment-16144281
]
Ferenc Szabo commented on FLUME-2698:
-
I made a simple load test with a memory channel and a null sink.
The jmeter config and the results are attached.
with the default 20M heap space, the jetty 9 slowed down earlier.
When I set the heap size to 128m, both version was able to handle 1 million
call on 200 thread (5000 each) without GC problems
Each test has been run on a freshly started flume agent
http average throughput :
- jetty 6: 5795/sec
- jetty 9: 12795/sec
https average throughput :
- jetty 6: 8442/sec
- jetty 9: 8919/sec
> Upgrade Jetty Version
> -
>
> Key: FLUME-2698
> URL: https://issues.apache.org/jira/browse/FLUME-2698
> Project: Flume
> Issue Type: Bug
> Components: Web
>Affects Versions: 1.6.0, 1.5.1, 1.7.0
>Reporter: Joakim Erdfelt
>Assignee: Tristan Stevens
> Labels: dependency
> Attachments: HttpSourceLoadTest.jmx, jetty-6-https-test.csv,
> jetty-6-http-test.csv, jetty-9-https-test.csv, jetty-9-http-test.csv
>
>
> Flume depends on Jetty 6
> {code:xml}
>
> org.mortbay.jetty
> jetty-util
> 6.1.26
>
> {code}
> Which was EOL (End of Life) back in 2010 and is no longer fit for production
> use (without heavy customizations and modifications like Google does for GAE,
> just to keep it safe and vulnerability free)
> Jetty was moved to Eclipse.org back during the Jetty 7 days.
> http://eclipse.org/jetty/
> Note that [Jetty 7 and Jetty 8 are now also
> EOL|https://dev.eclipse.org/mhonarc/lists/jetty-announce/msg00069.html] (as
> of 2014)
> Jetty 9 is the only stable and supported version of Jetty now.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLUME-3149) reduce cpu cost for file source transfer while still maintaining reliability
[ https://issues.apache.org/jira/browse/FLUME-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150803#comment-16150803 ] Ferenc Szabo commented on FLUME-3149: - Hi [~zyfo2], [~bessbd] as a brainstorming starter idea: we could have a PollableSourceChannel implementation where the channel would behave as a wrapper around a PollableSource (TailDirSource is one of them) and while the Pollable source can keep track of where it is in the processing, the channel can work without copying the data into a buffer and keeping track of its buffer. That would definitely use less resources. Let me know what you think about the idea. I'm happy to help with the implementation and the testing if needed. Thank you, Ferenc > reduce cpu cost for file source transfer while still maintaining reliability > > > Key: FLUME-3149 > URL: https://issues.apache.org/jira/browse/FLUME-3149 > Project: Flume > Issue Type: Improvement > Components: File Channel >Reporter: will zhang > > File channel tracks transferred events and use transnational mechanism to > make transfer recoverable. However, it increases CPU cost due to frequent > system calls like write, read, etc. The Cpu cost could be very high if the > transfer rate is high. In contrast, Memory channel has no such issue which > requires only about 10% of CPU cost in the same environment but it's not > recovered if the system is down accidentally. > For sources like taildir/spooldir, I propose we could track offsets of file > and store them locally to achieve reliability while still using memory > channel to reduce CPU cost. Actually, I have already implemented this feature > by storing the offsets in event headers and passing it to my own > "offsetMemoryChannel" and store theses offsets in local disk in our > production which reduces CPU cost by about 90 percent. > Please let me know if it's worthwhile to have this feature in community > version. Thank you. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (FLUME-3157) Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API
[ https://issues.apache.org/jira/browse/FLUME-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo reassigned FLUME-3157: --- Assignee: Ferenc Szabo > Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API > --- > > Key: FLUME-3157 > URL: https://issues.apache.org/jira/browse/FLUME-3157 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Ferenc Szabo >Assignee: Ferenc Szabo > Fix For: 1.8.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3157) Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API
Ferenc Szabo created FLUME-3157: --- Summary: Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API Key: FLUME-3157 URL: https://issues.apache.org/jira/browse/FLUME-3157 Project: Flume Issue Type: Bug Affects Versions: 1.7.0 Reporter: Ferenc Szabo Fix For: 1.8.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3158) Upgrade surefire version and config
Ferenc Szabo created FLUME-3158: --- Summary: Upgrade surefire version and config Key: FLUME-3158 URL: https://issues.apache.org/jira/browse/FLUME-3158 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo Assignee: Ferenc Szabo set surefire version to the latest and configure it to rerun failed tests. This way the flaky tests will not brake the build as often -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3159) make jdbc channel test use in-memory database with random name to make the test faster
Ferenc Szabo created FLUME-3159: --- Summary: make jdbc channel test use in-memory database with random name to make the test faster Key: FLUME-3159 URL: https://issues.apache.org/jira/browse/FLUME-3159 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo Assignee: Ferenc Szabo Priority: Minor -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3162) Flume 1.8.0 release
Ferenc Szabo created FLUME-3162: --- Summary: Flume 1.8.0 release Key: FLUME-3162 URL: https://issues.apache.org/jira/browse/FLUME-3162 Project: Flume Issue Type: Umbrella Affects Versions: 1.8.0 Reporter: Ferenc Szabo Assignee: Denes Arvay Fix For: 1.8.0 Release 1.8.0 umbrella issue -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3163) JIRA cleanup for 1.8.0 release
Ferenc Szabo created FLUME-3163: --- Summary: JIRA cleanup for 1.8.0 release Key: FLUME-3163 URL: https://issues.apache.org/jira/browse/FLUME-3163 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo We should resolve all issues targeted in 1.8.0 version or move them to 1.8.1 or 1.9.0 Filter for these issues: https://issues.apache.org/jira/issues/?filter=12341912 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3163) JIRA cleanup for 1.8.0 release
[ https://issues.apache.org/jira/browse/FLUME-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3163: Description: We should resolve all issues targeted in 1.8.0 version or move them to 1.8.1 or 1.9.0 Filter for these issues: https://issues.apache.org/jira/issues/?jql=project%20%3D%20Flume%20AND%20(Status%20!%3D%20Resolved%20OR%20Resolution%20!%3D%20Fixed)%20AND%20fixVersion%20%3D%201.8.0 was: We should resolve all issues targeted in 1.8.0 version or move them to 1.8.1 or 1.9.0 Filter for these issues: https://issues.apache.org/jira/issues/?filter=12341912 > JIRA cleanup for 1.8.0 release > -- > > Key: FLUME-3163 > URL: https://issues.apache.org/jira/browse/FLUME-3163 > Project: Flume > Issue Type: Sub-task >Reporter: Ferenc Szabo > Fix For: 1.8.0 > > > We should resolve all issues targeted in 1.8.0 version or move them to 1.8.1 > or 1.9.0 > Filter for these issues: > https://issues.apache.org/jira/issues/?jql=project%20%3D%20Flume%20AND%20(Status%20!%3D%20Resolved%20OR%20Resolution%20!%3D%20Fixed)%20AND%20fixVersion%20%3D%201.8.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLUME-3163) JIRA cleanup for 1.8.0 release
[ https://issues.apache.org/jira/browse/FLUME-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158362#comment-16158362 ] Ferenc Szabo commented on FLUME-3163: - [~marcellhegedus] thanks! I changed it to a simple query url, because I have no permission to create a public filter. > JIRA cleanup for 1.8.0 release > -- > > Key: FLUME-3163 > URL: https://issues.apache.org/jira/browse/FLUME-3163 > Project: Flume > Issue Type: Sub-task >Reporter: Ferenc Szabo > Fix For: 1.8.0 > > > We should resolve all issues targeted in 1.8.0 version or move them to 1.8.1 > or 1.9.0 > Filter for these issues: > https://issues.apache.org/jira/issues/?jql=project%20%3D%20Flume%20AND%20(Status%20!%3D%20Resolved%20OR%20Resolution%20!%3D%20Fixed)%20AND%20fixVersion%20%3D%201.8.0 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3143) Fix undeclared and unused dependencies
[
https://issues.apache.org/jira/browse/FLUME-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-3143:
Fix Version/s: (was: 1.8.0)
> Fix undeclared and unused dependencies
> --
>
> Key: FLUME-3143
> URL: https://issues.apache.org/jira/browse/FLUME-3143
> Project: Flume
> Issue Type: Bug
>Affects Versions: 1.7.0
>Reporter: Ferenc Szabo
>Assignee: Ferenc Szabo
> Labels: dependency
>
> Dependency analysis shows some warnings in the project that should be fixed
> {noformat}
> [INFO]
>
> [INFO] Building Flume NG SDK 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]commons-lang:commons-lang:jar:2.5:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Configuration 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume Auth 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.thrift:libthrift:jar:0.9.0:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG Core 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]org.apache.httpcomponents:httpcore:jar:4.2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.mortbay.jetty:jetty-util:jar:6.1.26:compile
> [WARNING]log4j:log4j:jar:1.2.17:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG HDFS Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.4.0:test
> [WARNING]org.apache.flume:flume-ng-auth:jar:1.8.0-SNAPSHOT:compile
> [WARNING]org.apache.avro:avro:jar:1.7.4:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.apache.hadoop:hadoop-minicluster:jar:2.4.0:test
> [WARNING]org.apache.hadoop:hadoop-auth:jar:2.4.0:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG IRC Sink 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.google.guava:guava:jar:11.0.2:compile
> [WARNING]commons-io:commons-io:jar:2.1:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG JDBC channel 1.8.0-SNAPSHOT
> [INFO]
>
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.slf4j:slf4j-api:jar:1.6.1:compile
> [WARNING]commons-pool:commons-pool:jar:1.5.4:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.6.1:test
> [WARNING]org.apache.derby:derby:jar:10.11.1.1:compile
> [INFO]
>
> [INFO]
>
> [INFO] Building Flume NG file-based channel 1.8.0-SNAPSHOT
> [INFO]
>
[jira] [Updated] (FLUME-3139) Resolve tilde (~) character in file channel config as the users home directory
[ https://issues.apache.org/jira/browse/FLUME-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3139: Description: Paths in the following config properties are not resolved properly checkpointDir backupCheckpointDir dataDir expected behaviour: ~/myDataDir should be resolved as /home//myDataDir actual behaviour: ~/myDataDir is resolved as /~/myDataDir was: Paths in the following config properties are not resolved properly checkpointDir backupCheckpointDir dataDir expected behaviour: ~/myDataDir should be resolved as /home//myDataDir actual behaviour: ~/myDataDir is resolved as /myDataDir > Resolve tilde (~) character in file channel config as the users home directory > -- > > Key: FLUME-3139 > URL: https://issues.apache.org/jira/browse/FLUME-3139 > Project: Flume > Issue Type: Bug > Components: File Channel >Affects Versions: 1.7.0 >Reporter: Ferenc Szabo > > Paths in the following config properties are not resolved properly > checkpointDir > backupCheckpointDir > dataDir > expected behaviour: > ~/myDataDir should be resolved as /home//myDataDir > actual behaviour: > ~/myDataDir is resolved as /~/myDataDir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3139) Resolve tilde (~) character in file channel config as the users home directory
[ https://issues.apache.org/jira/browse/FLUME-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3139: Fix Version/s: (was: 1.8.0) > Resolve tilde (~) character in file channel config as the users home directory > -- > > Key: FLUME-3139 > URL: https://issues.apache.org/jira/browse/FLUME-3139 > Project: Flume > Issue Type: Bug > Components: File Channel >Affects Versions: 1.7.0 >Reporter: Ferenc Szabo > > Paths in the following config properties are not resolved properly > checkpointDir > backupCheckpointDir > dataDir > expected behaviour: > ~/myDataDir should be resolved as /home//myDataDir > actual behaviour: > ~/myDataDir is resolved as /~/myDataDir -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3164) Update pom.xml version on trunk
Ferenc Szabo created FLUME-3164: --- Summary: Update pom.xml version on trunk Key: FLUME-3164 URL: https://issues.apache.org/jira/browse/FLUME-3164 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3165) Update versions in 1.8.0 branch
Ferenc Szabo created FLUME-3165: --- Summary: Update versions in 1.8.0 branch Key: FLUME-3165 URL: https://issues.apache.org/jira/browse/FLUME-3165 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3166) Update License file for 1.8 release
Ferenc Szabo created FLUME-3166: --- Summary: Update License file for 1.8 release Key: FLUME-3166 URL: https://issues.apache.org/jira/browse/FLUME-3166 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3167) Update KEYS file for 1.8 release
Ferenc Szabo created FLUME-3167: --- Summary: Update KEYS file for 1.8 release Key: FLUME-3167 URL: https://issues.apache.org/jira/browse/FLUME-3167 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3168) Update ChangeLog for 1.8 Release
Ferenc Szabo created FLUME-3168: --- Summary: Update ChangeLog for 1.8 Release Key: FLUME-3168 URL: https://issues.apache.org/jira/browse/FLUME-3168 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3170) Upload the artifacts for 1.8 release
Ferenc Szabo created FLUME-3170: --- Summary: Upload the artifacts for 1.8 release Key: FLUME-3170 URL: https://issues.apache.org/jira/browse/FLUME-3170 Project: Flume Issue Type: Bug Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3171) Deploy Maven artifacts for 1.8 release
Ferenc Szabo created FLUME-3171: --- Summary: Deploy Maven artifacts for 1.8 release Key: FLUME-3171 URL: https://issues.apache.org/jira/browse/FLUME-3171 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3169) Update the website for 1.8.0 release
Ferenc Szabo created FLUME-3169: --- Summary: Update the website for 1.8.0 release Key: FLUME-3169 URL: https://issues.apache.org/jira/browse/FLUME-3169 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (FLUME-3172) Announce the release for 1.8
Ferenc Szabo created FLUME-3172: --- Summary: Announce the release for 1.8 Key: FLUME-3172 URL: https://issues.apache.org/jira/browse/FLUME-3172 Project: Flume Issue Type: Sub-task Reporter: Ferenc Szabo -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3012) Sending a term signal can not shutdown Flume agent when KafkaChannel starting has exceptions
[ https://issues.apache.org/jira/browse/FLUME-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3012: Labels: 1.8.1 1.9 (was: ) > Sending a term signal can not shutdown Flume agent when KafkaChannel starting > has exceptions > > > Key: FLUME-3012 > URL: https://issues.apache.org/jira/browse/FLUME-3012 > Project: Flume > Issue Type: Bug > Components: Kafka Channel >Affects Versions: 1.6.0 > Environment: Flume 1.6.0+Kafka 0.9 >Reporter: Ping Wang > Labels: 1.8.1, 1.9 > Fix For: 1.8.0 > > Attachments: threaddumps.log > > > Use Kafka Channel in the agent configuration: > tier1.sources = source1 > tier1.channels = channel1 > tier1.sinks = sink1 > tier1.sources.source1.type = exec > tier1.sources.source1.command = /usr/bin/vmstat 1 > tier1.sources.source1.channels = channel1 > tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel > tier1.channels.channel1.kafka.bootstrap.servers = a.b.c.d:6667 > tier1.sinks.sink1.type = hdfs > tier1.sinks.sink1.hdfs.path = /tmp/kafka/channel > tier1.sinks.sink1.hdfs.rollInterval = 5 > tier1.sinks.sink1.hdfs.rollSize = 0 > tier1.sinks.sink1.hdfs.rollCount = 0 > tier1.sinks.sink1.hdfs.fileType = DataStream > tier1.sinks.sink1.channel = channel1 > Accidentally kaka.bootstrap.servers is not correct, errors will be thrown > out: > .. > )] Waiting for channel: channel1 to start. Sleeping for 500 ms > 2016-10-21 01:15:50,739 (conf-file-poller-0) [INFO - > org.apache.flume.node.Application.startAllComponents(Application.java:161)] > Waiting for channel: channel1 to start. Sleeping for 500 ms > 2016-10-21 01:15:51,240 (conf-file-poller-0) [INFO - > org.apache.flume.node.Application.startAllComponents(Application.java:161)] > Waiting for channel: channel1 to start. Sleeping for 500 ms > 2016-10-21 01:15:51,735 (lifecycleSupervisor-1-6) [INFO - > org.apache.flume.channel.kafka.KafkaChannel.start(KafkaChannel.java:115)] > Starting Kafka Channel: channel1 > 2016-10-21 01:15:51,737 (lifecycleSupervisor-1-6) [INFO - > org.apache.kafka.common.config.AbstractConfig.logAll(AbstractConfig.java:165)] > ProducerConfig values: > compression.type = none > metric.reporters = [] > metadata.max.age.ms = 30 > metadata.fetch.timeout.ms = 6 > reconnect.backoff.ms = 50 > sasl.kerberos.ticket.renew.window.factor = 0.8 > bootstrap.servers = [a.b.c.d:6667] > retry.backoff.ms = 100 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > buffer.memory = 33554432 > timeout.ms = 3 > key.serializer = class > org.apache.kafka.common.serialization.StringSerializer > sasl.kerberos.service.name = null > sasl.kerberos.ticket.renew.jitter = 0.05 > ssl.keystore.type = JKS > ssl.trustmanager.algorithm = PKIX > block.on.buffer.full = false > ssl.key.password = null > max.block.ms = 6 > sasl.kerberos.min.time.before.relogin = 6 > connections.max.idle.ms = 54 > ssl.truststore.password = null > max.in.flight.requests.per.connection = 5 > metrics.num.samples = 2 > client.id = > ssl.endpoint.identification.algorithm = null > ssl.protocol = TLS > request.timeout.ms = 3 > ssl.provider = null > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > acks = all > batch.size = 16384 > ssl.keystore.location = null > receive.buffer.bytes = 32768 > ssl.cipher.suites = null > ssl.truststore.type = JKS > security.protocol = PLAINTEXT > retries = 0 > max.request.size = 1048576 > value.serializer = class > org.apache.kafka.common.serialization.ByteArraySerializer > ssl.truststore.location = null > ssl.keystore.password = null > ssl.keymanager.algorithm = SunX509 > metrics.sample.window.ms = 3 > partitioner.class = class > org.apache.kafka.clients.producer.internals.DefaultPartitioner > send.buffer.bytes = 131072 > linger.ms = 0 > 2016-10-21 01:15:51,742 (lifecycleSupervisor-1-6) [DEBUG - > org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added > sensor with name bufferpool-wait-time > 2016-10-21 01:15:51,743 (lifecycleSupervisor-1-6) [DEBUG - > org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added > sensor with name buffer-exhausted-records > 2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [INFO - > org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:613)] > Closing the Kafka producer with timeoutMillis = 0 ms. > 2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [DEBUG - > org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer
[jira] [Updated] (FLUME-3012) Sending a term signal can not shutdown Flume agent when KafkaChannel starting has exceptions
[ https://issues.apache.org/jira/browse/FLUME-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3012: Fix Version/s: (was: 1.8.0) > Sending a term signal can not shutdown Flume agent when KafkaChannel starting > has exceptions > > > Key: FLUME-3012 > URL: https://issues.apache.org/jira/browse/FLUME-3012 > Project: Flume > Issue Type: Bug > Components: Kafka Channel >Affects Versions: 1.6.0 > Environment: Flume 1.6.0+Kafka 0.9 >Reporter: Ping Wang > Labels: 1.8.1, 1.9 > Attachments: threaddumps.log > > > Use Kafka Channel in the agent configuration: > tier1.sources = source1 > tier1.channels = channel1 > tier1.sinks = sink1 > tier1.sources.source1.type = exec > tier1.sources.source1.command = /usr/bin/vmstat 1 > tier1.sources.source1.channels = channel1 > tier1.channels.channel1.type = org.apache.flume.channel.kafka.KafkaChannel > tier1.channels.channel1.kafka.bootstrap.servers = a.b.c.d:6667 > tier1.sinks.sink1.type = hdfs > tier1.sinks.sink1.hdfs.path = /tmp/kafka/channel > tier1.sinks.sink1.hdfs.rollInterval = 5 > tier1.sinks.sink1.hdfs.rollSize = 0 > tier1.sinks.sink1.hdfs.rollCount = 0 > tier1.sinks.sink1.hdfs.fileType = DataStream > tier1.sinks.sink1.channel = channel1 > Accidentally kaka.bootstrap.servers is not correct, errors will be thrown > out: > .. > )] Waiting for channel: channel1 to start. Sleeping for 500 ms > 2016-10-21 01:15:50,739 (conf-file-poller-0) [INFO - > org.apache.flume.node.Application.startAllComponents(Application.java:161)] > Waiting for channel: channel1 to start. Sleeping for 500 ms > 2016-10-21 01:15:51,240 (conf-file-poller-0) [INFO - > org.apache.flume.node.Application.startAllComponents(Application.java:161)] > Waiting for channel: channel1 to start. Sleeping for 500 ms > 2016-10-21 01:15:51,735 (lifecycleSupervisor-1-6) [INFO - > org.apache.flume.channel.kafka.KafkaChannel.start(KafkaChannel.java:115)] > Starting Kafka Channel: channel1 > 2016-10-21 01:15:51,737 (lifecycleSupervisor-1-6) [INFO - > org.apache.kafka.common.config.AbstractConfig.logAll(AbstractConfig.java:165)] > ProducerConfig values: > compression.type = none > metric.reporters = [] > metadata.max.age.ms = 30 > metadata.fetch.timeout.ms = 6 > reconnect.backoff.ms = 50 > sasl.kerberos.ticket.renew.window.factor = 0.8 > bootstrap.servers = [a.b.c.d:6667] > retry.backoff.ms = 100 > sasl.kerberos.kinit.cmd = /usr/bin/kinit > buffer.memory = 33554432 > timeout.ms = 3 > key.serializer = class > org.apache.kafka.common.serialization.StringSerializer > sasl.kerberos.service.name = null > sasl.kerberos.ticket.renew.jitter = 0.05 > ssl.keystore.type = JKS > ssl.trustmanager.algorithm = PKIX > block.on.buffer.full = false > ssl.key.password = null > max.block.ms = 6 > sasl.kerberos.min.time.before.relogin = 6 > connections.max.idle.ms = 54 > ssl.truststore.password = null > max.in.flight.requests.per.connection = 5 > metrics.num.samples = 2 > client.id = > ssl.endpoint.identification.algorithm = null > ssl.protocol = TLS > request.timeout.ms = 3 > ssl.provider = null > ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] > acks = all > batch.size = 16384 > ssl.keystore.location = null > receive.buffer.bytes = 32768 > ssl.cipher.suites = null > ssl.truststore.type = JKS > security.protocol = PLAINTEXT > retries = 0 > max.request.size = 1048576 > value.serializer = class > org.apache.kafka.common.serialization.ByteArraySerializer > ssl.truststore.location = null > ssl.keystore.password = null > ssl.keymanager.algorithm = SunX509 > metrics.sample.window.ms = 3 > partitioner.class = class > org.apache.kafka.clients.producer.internals.DefaultPartitioner > send.buffer.bytes = 131072 > linger.ms = 0 > 2016-10-21 01:15:51,742 (lifecycleSupervisor-1-6) [DEBUG - > org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added > sensor with name bufferpool-wait-time > 2016-10-21 01:15:51,743 (lifecycleSupervisor-1-6) [DEBUG - > org.apache.kafka.common.metrics.Metrics.sensor(Metrics.java:201)] Added > sensor with name buffer-exhausted-records > 2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [INFO - > org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:613)] > Closing the Kafka producer with timeoutMillis = 0 ms. > 2016-10-21 01:15:51,744 (lifecycleSupervisor-1-6) [DEBUG - > org.apache.kafka.clients.producer.KafkaProducer.close(KafkaProducer.java:654)] > The Kafka p
[jira] [Updated] (FLUME-3142) Adding HBase2 sink
[ https://issues.apache.org/jira/browse/FLUME-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3142: Labels: 1.9 (was: ) > Adding HBase2 sink > -- > > Key: FLUME-3142 > URL: https://issues.apache.org/jira/browse/FLUME-3142 > Project: Flume > Issue Type: New Feature >Reporter: Miklos Csanady >Assignee: Miklos Csanady >Priority: Minor > Labels: 1.9 > Fix For: 1.8.0 > > > HBase sink use some api calls which are deprecated in HBase 1.0.0 > As a preparation for HBase upgrade, these should be replaced. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-3142) Adding HBase2 sink
[ https://issues.apache.org/jira/browse/FLUME-3142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3142: Fix Version/s: (was: 1.8.0) > Adding HBase2 sink > -- > > Key: FLUME-3142 > URL: https://issues.apache.org/jira/browse/FLUME-3142 > Project: Flume > Issue Type: New Feature >Reporter: Miklos Csanady >Assignee: Miklos Csanady >Priority: Minor > Labels: 1.9 > > HBase sink use some api calls which are deprecated in HBase 1.0.0 > As a preparation for HBase upgrade, these should be replaced. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLUME-2853) Allow for YAML configuration files
[
https://issues.apache.org/jira/browse/FLUME-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2853:
Labels: 1.9 (was: )
> Allow for YAML configuration files
> --
>
> Key: FLUME-2853
> URL: https://issues.apache.org/jira/browse/FLUME-2853
> Project: Flume
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 1.8.0
>Reporter: Christopher White
>Priority: Minor
> Labels: 1.9
>
> Allow for YAML formatted configuration files
> (http://www.yaml.org/spec/1.2/spec.html).
> This provides:
> * A more condensed format than properties files
> * Less 'typo' prone for repetitive common prefixes
> * Ability to define a value once and reuse via references (see [spec -
> Structures - Example 2.10|http://www.yaml.org/spec/1.2/spec.html#id2760395]
> For example compare the following properties file and potential YAML
> equivalent:
> {code:title=agent.properties}
> host1.sources = source1
> host1.channels = channel1
> host1.sinks = sink1
> host1.sources.source1.type = seq
> host1.sources.source1.channels = channel1
> host1.channels.channel1.type = memory
> host1.channels.channel1.capacity = 1
> host1.sinks.sink1.type = null
> host1.sinks.sink1.channel = channel1
> {code}
> {code:title=agent.yaml}
> host1:
> sources:
> _: source1
> source1:
> type: seq
> channels: channel1
>
> channels:
> _: channel1
> channel1:
> type: memory
> capacity: 1
>
> sinks:
> _: sink1
> sink1:
> type: null
> channel: channel1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-2853) Allow for YAML configuration files
[
https://issues.apache.org/jira/browse/FLUME-2853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Szabo updated FLUME-2853:
Fix Version/s: (was: 1.8.0)
> Allow for YAML configuration files
> --
>
> Key: FLUME-2853
> URL: https://issues.apache.org/jira/browse/FLUME-2853
> Project: Flume
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 1.8.0
>Reporter: Christopher White
>Priority: Minor
> Labels: 1.9
>
> Allow for YAML formatted configuration files
> (http://www.yaml.org/spec/1.2/spec.html).
> This provides:
> * A more condensed format than properties files
> * Less 'typo' prone for repetitive common prefixes
> * Ability to define a value once and reuse via references (see [spec -
> Structures - Example 2.10|http://www.yaml.org/spec/1.2/spec.html#id2760395]
> For example compare the following properties file and potential YAML
> equivalent:
> {code:title=agent.properties}
> host1.sources = source1
> host1.channels = channel1
> host1.sinks = sink1
> host1.sources.source1.type = seq
> host1.sources.source1.channels = channel1
> host1.channels.channel1.type = memory
> host1.channels.channel1.capacity = 1
> host1.sinks.sink1.type = null
> host1.sinks.sink1.channel = channel1
> {code}
> {code:title=agent.yaml}
> host1:
> sources:
> _: source1
> source1:
> type: seq
> channels: channel1
>
> channels:
> _: channel1
> channel1:
> type: memory
> capacity: 1
>
> sinks:
> _: sink1
> sink1:
> type: null
> channel: channel1
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (FLUME-3115) Upgrade netty library dependency
[ https://issues.apache.org/jira/browse/FLUME-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Szabo updated FLUME-3115: Fix Version/s: (was: 1.8.0) 1.8.1 > Upgrade netty library dependency > > > Key: FLUME-3115 > URL: https://issues.apache.org/jira/browse/FLUME-3115 > Project: Flume > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Attila Simon >Assignee: Ferenc Szabo >Priority: Critical > Labels: dependency > Fix For: 1.8.1 > > > ||Group||Artifact||Version used||Upgrade target|| > |io.netty|netty|3.2.2.Final, 3.9.4.Final|4.1.12.Final| > Note: This artifact was moved to: > - New Group io.netty > - New Artifactnetty-all > Security vulnerability: http://www.cvedetails.com/cve/CVE-2014-3488/ > Please do: > - double check the newest version. > - consider to remove a dependency if better alternative is available. > - check whether the lib change would introduce a backward incompatibility (in > which case please add this label `breaking_change` and fix version should be > the next major) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
