[
https://issues.apache.org/jira/browse/FLUME-3037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15783135#comment-15783135
]
Tristan Stevens commented on FLUME-3037:
----------------------------------------
The reason this happens is because you're not waiting for the responses to come
back in. By default, the netcat source will send an "OK" message back for every
event. You're killing the connection before that response can be sent, which is
causing the processing of further messages to fail.
To fix this, you need the following change to your flume.conf:
{{a1.sources.r1.ack-every-event=false}}
This eliminates the requirement for an "OK" to be sent and therefore stops the
failure.
> can't get events correctly while flume working with python socket or telnet
> ---------------------------------------------------------------------------
>
> Key: FLUME-3037
> URL: https://issues.apache.org/jira/browse/FLUME-3037
> Project: Flume
> Issue Type: Question
> Components: Sinks+Sources
> Affects Versions: v1.7.0
> Environment: ubuntu 14&java 1.8
> centos &java 1.7
> Reporter: finetu
> Assignee: Tristan Stevens
> Labels: beginner
>
> I want to use flume for collecting logs in python scripts, so I follow user
> guide to config flume with netcat source, then I use telnet and nc for test,
> it works well.
> config code :
> a1.sources = r1
> a1.sinks = k1
> a1.channels = c1
> # Describe/configure the source
> a1.sources.r1.type = netcat
> a1.sources.r1.bind = localhost
> a1.sources.r1.port = 44444
> a1.sinks.k1.type = logger
> a1.channels.c1.type = memory
> a1.channels.c1.capacity = 1000
> a1.channels.c1.transactionCapacity = 100
> a1.sources.r1.channels = c1
> a1.sinks.k1.channel = c1
> Then I use python to connect flume, and send some words to it like this:
> import socket
> def netcat(hostname, port):
> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> s.connect((hostname, port))
> s.send("test words 1\n")
> s.send("test words 2\n")
> s.send("test words 3\n")
> s.send("test words 4\n")
> s.shutdown(socket.SHUT_WR)
> s.close()
> if __name__ == "__main__":
> netcat("127.0.0.1",44444)
> problem happens, flume can only receive 2 rows.
> flume logs:
> 2016-12-28 16:44:32,248 (lifecycleSupervisor-1-0) [INFO -
> org.apache.flume.source.NetcatSource.start(NetcatSource.java:169)] Created
> serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
> 2016-12-28 16:44:41,814 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO
> - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
> headers:{} body: 74 65 73 74 20 77 6F 72 64 73 20 31 test words 1
> }
> 2016-12-28 16:44:41,815 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO
> - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: {
> headers:{} body: 74 65 73 74 20 77 6F 72 64 73 20 32 test words 2
> }
> I got same result on both ubuntu&java1.8 and centos&java 1.7 and the same
> result with telnet model in python.
> Is there anything wrong with config or python scripts ?
> I think the issue-2611 is similar to this one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
