I had found the issue reason, it's caused by Python stdout buffer strategy.
the standard input and output streams are fully buffered if and only if the
streams do not to refer to an interactive device.
-- from libc stdio
manual
So if you want to get Python stdout content while using exec source, should
call sys.stdout.flush() after print someting...
Other ways:
1. python -u , Force stdin, stdout and stderr to be totally unbuffered.
2. declare sys.stdout as sys.stdout = os.fdopen(sys.stdout.fileno(), ‘w’, 0)
2014-10-10
shengyi.pan
发件人：Roshan Naik <[email protected]>
发送时间：2014-10-01 03:46
主题：Re: The flume exec source can not get content while using python command
收件人："[email protected]"<[email protected]>
抄送：
i use python with exec source quite often. see if putting a full path to
python helps... /usr/bin/python or whatever it is on ur system.
-roshan
On Thu, Sep 18, 2014 at 5:30 PM, shengyi.pan <[email protected]> wrote:
hi, all:
I want to use exec source under Flume 1.5.0. Follow the flume user guide, tail
command is running successfully on my computer, the flume.conf is :
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /var/log/secure
When I want to run Python script, change the flume.conf as follows:
a1.sources.r1.type = exec
a1.sources.r1.command = python /home/test.py
The test.py is very simpe, just print string line periodically
cat test_flume.py
#!/usr/bin/python
import sys
import time
sys.stdout.write("gogogo\n")
while True:
sys.stdout.write("gogogo\n")
time.sleep(3)
Start the flume with: nohup bin/flume-ng agent --conf ./conf/ -f
./conf/flume.conf -n a1 &
The flume is running and the python subprocess is existed, but no data is
catched from command : python /home/test.py, looks like Flume cann't read
anything from Python stdout.
Can someone help me ? T_T Thank you very much!
2014-09-18
shengyi.pan
2014-09-19
shengyi.pan
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader of
this message is not the intended recipient, you are hereby notified that any
printing, copying, dissemination, distribution, disclosure or forwarding of
this communication is strictly prohibited. If you have received this
communication in error, please contact the sender immediately and delete it
from your system. Thank You.
