[
https://issues.apache.org/jira/browse/FLUME-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240821#comment-13240821
]
[email protected] commented on FLUME-1037:
------------------------------------------------------
bq. On 2012-03-28 21:43:06, Hari Shreedharan wrote:
bq. > flume-ng-core/src/main/java/org/apache/flume/event/EventHelper.java,
line 47
bq. > <https://reviews.apache.org/r/4497/diff/2/?file=96930#file96930line47>
bq. >
bq. > Maybe log this?
Logging this isn't needed... this is not an exceptional case (just an empty
body). This is just working around a bug in HexDump where it improperly handles
a zero-length array.
bq. On 2012-03-28 21:43:06, Hari Shreedharan wrote:
bq. > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java,
line 130
bq. > <https://reviews.apache.org/r/4497/diff/2/?file=96931#file96931line130>
bq. >
bq. > If there is no "port", context.getInteger() will return null here -
causing autoboxing of a null value - ending in NullPointerException. Either
port has to be an Integer or you must check for the null value before
assignment.
This comes after the line:
Configurables.ensureRequiredNonNull(context, "bind", "port");
So by the time it reaches this statement, "port" is guaranteed not to be null.
bq. On 2012-03-28 21:43:06, Hari Shreedharan wrote:
bq. > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java,
line 315
bq. > <https://reviews.apache.org/r/4497/diff/2/?file=96931#file96931line315>
bq. >
bq. > Why would charsRead be 0, if the client sent a line longer than
maxLineLength? Shouldn't it be > maxLineLength?
No, the condition here is that we didn't read any additional data, the buffer
is full, and we searched and didn't find any newlines. Therefore we conclude
that the client sent a longer line than can fit into our buffer. I'll add this
explanation in a comment and update the patch.
bq. On 2012-03-28 21:43:06, Hari Shreedharan wrote:
bq. > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java,
line 328
bq. > <https://reviews.apache.org/r/4497/diff/2/?file=96931#file96931line328>
bq. >
bq. > nit: Shouldn't this go at the beginning of the loop? Otherwise
doesn't the first iteration of the loop just do nothing? Since the buffer was
just created, it is obviously empty, so the first run of the loop seems like a
no-op.
You are right; this ordering is a remnant from a work-in-progress incarnation
of the implementation. I will reorder these steps to be more intuitive.
- Mike
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4497/#review6490
-----------------------------------------------------------
On 2012-03-27 09:19:51, Mike Percy wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4497/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-27 09:19:51)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. The NetcatSource does not create one event per newline. Instead, it
creates one event per TCP session.
bq.
bq. I have addressed multiple issues with NetcatSource in this patch:
bq.
bq. 1. Properly process a new event per newline.
bq. 2. Enforce a maximum length per line, to ensure clients cannot run the
server out of memory. (This is now configurable; the default is 512 characters.)
bq. 3. Properly flush responses to the client.
bq. 4. Increment counters for successful processing and failure.
bq. 5. Use shutdownNow() when shutting down the server, otherwise an open
client connection will cause the server to hang on shutdown.
bq. 6. Made the inner classes of NetcatSource private; I believe making them
public was unintentional.
bq. 7. Corrected unit test so it now sends a newline with each request. Also
now checks for "OK" response from server.
bq. 8. Attempting to sneak in a minor fix for the EventHelper.dumpEvent()
method into this patch, since I'm testing with LoggerSink and it's bugging me
when it throws an exception on a zero-length body.
bq.
bq.
bq. This addresses bug FLUME-1037.
bq. https://issues.apache.org/jira/browse/FLUME-1037
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/main/java/org/apache/flume/event/EventHelper.java
a326a70
bq. flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
a841b0e
bq.
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java
fb2a960
bq.
bq. Diff: https://reviews.apache.org/r/4497/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Unit tests pass. Did a bunch of manual testing using the nc tool with a
Netcat source and a Logger sink.
bq.
bq.
bq. Thanks,
bq.
bq. Mike
bq.
bq.
> NETCAT handler theads terminate under stress test
> -------------------------------------------------
>
> Key: FLUME-1037
> URL: https://issues.apache.org/jira/browse/FLUME-1037
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Environment: [CentOS 6.2 64-bit]
> Reporter: Will McQueen
> Assignee: Mike Percy
>
> Steps:
> 1. Use a props file such as the following:
> \# a = agent
> \# r = source
> \# c = channel
> \# k = sink
> a1.sources = r1
> a1.channels = c1
> a1.sinks = k1
> \# ===SOURCES===
> a1.sources.r1.type = NETCAT
> a1.sources.r1.channels = c1
> a1.sources.r1.bind = localhost
> a1.sources.r1.port = 1473
> \# ===CHANNELS===
> a1.channels.c1.type = MEMORY
> \# ===SINKS===
> a1.sinks.k1.type = NULL
> a1.sinks.k1.channel = c1
> 2. Set the FLUME_CONF_DIR to point to your conf dir
> [will@localhost flume-1.2.0-incubating-SNAPSHOT]$ export
> FLUME_CONF_DIR=/home/will/git/apache/flume/flume-1.2.0-incubating-SNAPSHOT/conf
> 3. Create a flume-env.sh file
> [will@localhost flume-1.2.0-incubating-SNAPSHOT]$ cp
> conf/flume-env.sh.template conf/flume-env.sh
> 4. Adjust the memory size within flume-env.sh (this file will be
> automatically sourced when calling bin/flume-ng, but only if you've specified
> the FLUME_CONF_DIR env var)
> (here, I went to the extreme and I set the min and max heap to 1GB. I also
> specified a YourKit profiler agent)
> Sample contents of flume-env.sh:
> export JAVA_OPTS="-Xms1024m -Xmx1024m
> -agentpath:/home/will/tools/yjp-10.0.6/bin/linux-x86-64/libyjpagent.so=tracing,noj2ee"
> 5. Run the flume NG agent:
> bin/flume-ng node --conf conf --conf-file conf/a1.properties --name a1
> 6. Open-up 10 terminal windows (on the same host) to connect to the netcat
> server port. Sent continuous output in each terminal. I chose to use the
> command:
> yes | nc localhost 1473
> The "yes" unix cmd will continuously output 'y' char, followed by newline
> char. If you use YourKit and go into the Threads view, you'll see that after
> a while (possibly need to wait up to 10 mins) after a netcat handler thread
> has continuously been alternating between Runnable and Blocked states
> (blocking due to org.apache.klog4j.Category.log(..), but that's beside the
> point), the netcat handler thread enters a continuous wait state for exactly
> 1 minute, and then terminates (while its associated 'yes | nc localhost 1473'
> command is still running).
> I haven't done further analysis. My first thought was a thread safety issue.
> Note that there are no property file reconfigurations done during this test
> -- I leave the props file alone.
> I welcome your ideas/comments. I initially ran this test with the default
> -Xmx20m but it ran out of memory. For a future test I might lower the Xmx/Xms
> from 1GB to maybe 128MB.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
