I am using Flume 1.3.1 and ran the example before sending.
I don't have access to a windows system at the moment so can't test.
This is output at my end on Flume Agent console
2013-06-21 13:03:54,913 (conf-file-poller-0) [INFO -
org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
Starting Source src1
2013-06-21 13:03:54,927 (lifecycleSupervisor-1-2) [INFO -
org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2013-06-21 13:03:54,958 (lifecycleSupervisor-1-2) [INFO -
org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
2013-06-21 13:03:55,022 (lifecycleSupervisor-1-2) [INFO -
org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started
SocketConnector@0.0.0.0:9001
2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
Event: { headers:{timestamp=434324343, host=random_host.example.co
m} body: 72 61 6E 64 6F 6D 5F 62 6F 64 79 random_body }
2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
[INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
Event: { headers:{namenode=namenode.example.com, datanode=
random_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F 6D
5F 62 6F really_random_bo }
Flume dev's can comment further on this.
On Fri, Jun 21, 2013 at 12:33 PM, Nickolay Kolev <lem0...@gmail.com> wrote:
> Hi Ashish,
>
> Thanks for pointing me that error. I am trying to read the code and this
> is the correct full class name. (last time I wrote java code was in 1998
> and my knowledge are a lot out of date)
>
> Unfortunately the result is the same.
> >curl -X POST -H "Content-Type: application/json" -d "[{ \"headers\" : {
> \"timestamp\" : \"434324343\", \"host\" : \"random_host.example.com\" },
> \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\" },
> \"body\" : \"really_random_body\"}]" http://localhost:9001
>
> curl: (7) Failed connect to localhost:9001; No error
>
> Could be the reason if this errors the fact that I use Windows? I see in
> wiki that Windows is not officially supported.
>
> Yesterday my first test was with netcat source and it worked as expected.
>
>
> best regards,
> nickolay kolev
>
>
> On Fri, Jun 21, 2013 at 9:48 AM, Ashish <paliwalash...@gmail.com> wrote:
>
>> update the handler config as
>>
>> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>>
>> and try. I used your source config and it worked for me. Let us know if
>> it work
>>
>>
>> On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <lem0...@gmail.com>wrote:
>>
>>> Hi Hari and the all members of this list,
>>>
>>> Thanks for pointing me that error. This is actually my second attempt to
>>> use Apache Flume - with avro client. The symptoms are the same - the is no
>>> listener on the specified port and I am not able to connect.
>>>
>>> I started to evaluate Flume just yesterday and many things are new to me
>>> and probably I do many mistakes. I want to point that I use Flume 1.3.1
>>> compiled by me if this is important to be mentioned. I have tested too with
>>> the official binary from Flume web site but the results are the same.
>>>
>>> Here is what I do to test HTTP source:
>>>
>>> my config file (httppost.conf):
>>> agent1.sources = r1
>>> agent1.channels = ch1
>>> agent1.sinks = k1
>>>
>>> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>>> agent1.sources.r1.port = 9001
>>> agent1.sources.r1.channels = ch1
>>> #agent1.sources.r1.handler = org.example.rest.RestHandler
>>> agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
>>> agent1.sources.r1.handler.nickname = json handler
>>>
>>> agent1.sinks.k1.type = file_roll
>>> agent1.sinks.k1.channel = ch1
>>>
>>> agent1.sinks.k1.sink.directory =
>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>>>
>>> agent1.channels.ch1.type = memory
>>> agent1.channels.ch1.capacity = 10000
>>> agent1.channels.ch1.transactionCapacity = 1000
>>>
>>> my server bat file:
>>>
>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>> set JAVA_OPTS=-Xmx60m
>>> set CONF=%FLUME_HOME%\conf\httppost.conf
>>> set AGENT=agent1
>>>
>>> %JAVA% %JAVA_OPTS%
>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>>>
>>> How I test:
>>>
>>> server console:
>>> >"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m
>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties
>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*"
>>> org.apache.flume.node.Application -f
>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1
>>>
>>> client console:
>>> >C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H
>>> "Content-Type: application/json" -d "[{ \"headers\" : { \"timestamp\" :
>>> \"434324343\", \"host\" : \"random_host.example.co
>>> m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
>>> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\"
>>> }, \"body\" : \"really_random_body\"
>>> }]" http://localhost:9001
>>> curl: (7) Failed connect to localhost:9001; No error
>>>
>>> If I understand correct the documentation I should expect that on the
>>> specified port (9001) it must be listener that will accept POST requests?
>>> Is this correct?
>>> Do I miss some important aspects that are critical to run Flume?
>>>
>>> Thanks in advance for your help.
>>>
>>>
>>> best regards,
>>> nickolay kolev
>>>
>>>
>>>
>>> On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan <
>>> hshreedha...@cloudera.com> wrote:
>>>
>>>> Does not look like you are using the http source at all. Your source
>>>> type needs to be HTTP
>>>>
>>>>
>>>> Cheers,
>>>> Hari
>>>>
>>>> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:
>>>>
>>>> Hi all,
>>>> I am new to flume and all that logging stuff and probably many things
>>>> are unclear to me despite I read the docs.
>>>> I want to run Flume on windows server to create proof of concept
>>>> solution. I was able to compile it successfully. I was able to make
>>>> successful test with netcat.
>>>> Next test that i wanted to do is with http source but unfortunately it
>>>> did not pass.
>>>> Here is what I do:
>>>>
>>>> my server config (httppost.conf):
>>>> agent1.sources = r1
>>>> agent1.channels = memoryChannel
>>>> agent1.sinks = k1
>>>>
>>>> agent1.sources.r1.type = avro
>>>> agent1.sources.r1.bind = localhost
>>>> agent1.sources.r1.port = 9003
>>>>
>>>> agent1.sinks.k1.type = file_roll
>>>> agent1.sinks.k1.sink.directory =
>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>>>>
>>>> agent1.channels.memoryChannel.type = memory
>>>> agent1.channels.memoryChannel.capacity = 10000
>>>> agent1.channels.memoryChannel.transactionCapacity = 1000
>>>>
>>>>
>>>> agent1.sources.r1.channels = memoryChannel
>>>> agent1.sinks.k1.channel = memoryChannel
>>>>
>>>> I run the server with this bat file:
>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>> set JAVA_OPTS=-Xmx60m
>>>> set CONF=%FLUME_HOME%\conf\httppost.conf
>>>> set AGENT=agent1
>>>>
>>>> %JAVA% %JAVA_OPTS%
>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>>>>
>>>> netstat does not show any listener on port 9003? Is this normal or
>>>> there must be a listener?
>>>>
>>>> I run avro client with this bat file:
>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>> set JAVA_OPTS=-Xmx60m
>>>> set AVRO_HOST=localhost
>>>> set AVRO_PORT=9003
>>>> set SENDFILE=%FLUME_HOME%\logs\data.log
>>>> set HEADFILE=%FLUME_HOME%\conf\header.txt
>>>>
>>>> %JAVA% %JAVA_OPTS%
>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>>>> "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H
>>>> %AVRO_HOST% -p %AVRO_PORT% -F %SENDFILE% --headerFile %HEADFILE%
>>>>
>>>>
>>>> Then I get this output:
>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"
>>>> -Xmx60m
>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties
>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*"
>>>> org.apache.flume.client.avro.AvroCLIClient -H localhost -p 9003 -F
>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile
>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
>>>> 2013-06-20 17:55:58,285 (main) [DEBUG -
>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>>>> Inserting Header Key [filename] header value [datalog]
>>>> 2013-06-20 17:55:58,287 (main) [DEBUG -
>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>>>> Inserting Header Key [host] header value [zhouhh]
>>>> 2013-06-20 17:55:58,295 (main) [DEBUG -
>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
>>>> Batch size string = 5
>>>> 2013-06-20 17:55:59,358 (main) [ERROR -
>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)]
>>>> Unable to open connection to Flume. Exception follows.
>>>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost,
>>>> port: 9003 }: RPC connection error
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>>>> at
>>>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>>>> at
>>>> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>>>> at
>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>>>> Caused by: java.io.IOException: Error connecting to localhost/
>>>> 127.0.0.1:9003
>>>> at
>>>> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>>>> at
>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>>>> at
>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>>>> at
>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>>>> ... 5 more
>>>> Caused by: java.net.ConnectException: Connection refused: no further
>>>> information
>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>>>> at
>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> 2013-06-20 17:55:59,369 (main) [DEBUG -
>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)]
>>>> Exiting
>>>>
>>>>
>>>> My question is what I am doing wrong and what I need to test in order
>>>> to fix this situatioin.
>>>>
>>>> Thanks in advance.
>>>>
>>>> best regards,
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> thanks
>> ashish
>>
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
