Hi Ashish,
I works, I have used notepad++ and it have put some dummy symbols and the
config was not parsed correct.
Now I can confirm that this setup works on Windows.
Thanks all for the patience and sorry for too many email. Hope this can be
useful for someone that will do similar tests.
best regards,
nickolay koelv
On Fri, Jun 21, 2013 at 10:40 AM, Ashish <paliwalash...@gmail.com> wrote:
> I am using Flume 1.3.1 and ran the example before sending.
>
> I don't have access to a windows system at the moment so can't test.
>
> This is output at my end on Flume Agent console
>
> 2013-06-21 13:03:54,913 (conf-file-poller-0) [INFO -
> org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:152)]
> Starting Source src1
> 2013-06-21 13:03:54,927 (lifecycleSupervisor-1-2) [INFO -
> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2013-06-21 13:03:54,958 (lifecycleSupervisor-1-2) [INFO -
> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] jetty-6.1.26
> 2013-06-21 13:03:55,022 (lifecycleSupervisor-1-2) [INFO -
> org.mortbay.log.Slf4jLog.info(Slf4jLog.java:67)] Started
> SocketConnector@0.0.0.0:9001
> 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
> [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
> Event: { headers:{timestamp=434324343, host=random_host.example.co
> m} body: 72 61 6E 64 6F 6D 5F 62 6F 64 79 random_body }
> 2013-06-21 13:09:36,992 (SinkRunner-PollingRunner-DefaultSinkProcessor)
> [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)]
> Event: { headers:{namenode=namenode.example.com, datanode=
> random_datanode.example.com} body: 72 65 61 6C 6C 79 5F 72 61 6E 64 6F 6D
> 5F 62 6F really_random_bo }
>
> Flume dev's can comment further on this.
>
>
> On Fri, Jun 21, 2013 at 12:33 PM, Nickolay Kolev <lem0...@gmail.com>wrote:
>
>> Hi Ashish,
>>
>> Thanks for pointing me that error. I am trying to read the code and this
>> is the correct full class name. (last time I wrote java code was in 1998
>> and my knowledge are a lot out of date)
>>
>> Unfortunately the result is the same.
>> >curl -X POST -H "Content-Type: application/json" -d "[{ \"headers\" : {
>> \"timestamp\" : \"434324343\", \"host\" : \"random_host.example.com\" },
>> \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
>> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\"
>> }, \"body\" : \"really_random_body\"}]" http://localhost:9001
>>
>> curl: (7) Failed connect to localhost:9001; No error
>>
>> Could be the reason if this errors the fact that I use Windows? I see in
>> wiki that Windows is not officially supported.
>>
>> Yesterday my first test was with netcat source and it worked as expected.
>>
>>
>> best regards,
>> nickolay kolev
>>
>>
>> On Fri, Jun 21, 2013 at 9:48 AM, Ashish <paliwalash...@gmail.com> wrote:
>>
>>> update the handler config as
>>>
>>> agent1.sources.r1.handler = org.apache.flume.source.http.JSONHandler
>>>
>>> and try. I used your source config and it worked for me. Let us know if
>>> it work
>>>
>>>
>>> On Fri, Jun 21, 2013 at 12:04 PM, Nickolay Kolev <lem0...@gmail.com>wrote:
>>>
>>>> Hi Hari and the all members of this list,
>>>>
>>>> Thanks for pointing me that error. This is actually my second attempt
>>>> to use Apache Flume - with avro client. The symptoms are the same - the is
>>>> no listener on the specified port and I am not able to connect.
>>>>
>>>> I started to evaluate Flume just yesterday and many things are new to
>>>> me and probably I do many mistakes. I want to point that I use Flume 1.3.1
>>>> compiled by me if this is important to be mentioned. I have tested too with
>>>> the official binary from Flume web site but the results are the same.
>>>>
>>>> Here is what I do to test HTTP source:
>>>>
>>>> my config file (httppost.conf):
>>>> agent1.sources = r1
>>>> agent1.channels = ch1
>>>> agent1.sinks = k1
>>>>
>>>> agent1.sources.r1.type = org.apache.flume.source.http.HTTPSource
>>>> agent1.sources.r1.port = 9001
>>>> agent1.sources.r1.channels = ch1
>>>> #agent1.sources.r1.handler = org.example.rest.RestHandler
>>>> agent1.sources.r1.handler = org.apache.flume.http.JSONHandler
>>>> agent1.sources.r1.handler.nickname = json handler
>>>>
>>>> agent1.sinks.k1.type = file_roll
>>>> agent1.sinks.k1.channel = ch1
>>>>
>>>> agent1.sinks.k1.sink.directory =
>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>>>>
>>>> agent1.channels.ch1.type = memory
>>>> agent1.channels.ch1.capacity = 10000
>>>> agent1.channels.ch1.transactionCapacity = 1000
>>>>
>>>> my server bat file:
>>>>
>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>> set JAVA_OPTS=-Xmx60m
>>>> set CONF=%FLUME_HOME%\conf\httppost.conf
>>>> set AGENT=agent1
>>>>
>>>> %JAVA% %JAVA_OPTS%
>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n %AGENT%
>>>>
>>>> How I test:
>>>>
>>>> server console:
>>>> >"C:\Java\jdk1.6.0_45\bin\java.exe" -Xmx60m
>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\log4j.properties
>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\\lib\*"
>>>> org.apache.flume.node.Application -f
>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\\conf\httppost.conf -n agent1
>>>>
>>>> client console:
>>>> >C:\Projects\nickyk\apache-flume-1.3.1-bin>curl -X POST -H
>>>> "Content-Type: application/json" -d "[{ \"headers\" : { \"timestamp\" :
>>>> \"434324343\", \"host\" : \"random_host.example.co
>>>> m\" }, \"body\" : \"random_body\" }, { \"headers\" : { \"namenode\" : \"
>>>> namenode.example.com\", \"datanode\" : \"random_datanode.example.com\"
>>>> }, \"body\" : \"really_random_body\"
>>>> }]" http://localhost:9001
>>>> curl: (7) Failed connect to localhost:9001; No error
>>>>
>>>> If I understand correct the documentation I should expect that on the
>>>> specified port (9001) it must be listener that will accept POST requests?
>>>> Is this correct?
>>>> Do I miss some important aspects that are critical to run Flume?
>>>>
>>>> Thanks in advance for your help.
>>>>
>>>>
>>>> best regards,
>>>> nickolay kolev
>>>>
>>>>
>>>>
>>>> On Thu, Jun 20, 2013 at 9:07 PM, Hari Shreedharan <
>>>> hshreedha...@cloudera.com> wrote:
>>>>
>>>>> Does not look like you are using the http source at all. Your source
>>>>> type needs to be HTTP
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Hari
>>>>>
>>>>> On Thursday, June 20, 2013 at 8:57 AM, Nickolay Kolev wrote:
>>>>>
>>>>> Hi all,
>>>>> I am new to flume and all that logging stuff and probably many things
>>>>> are unclear to me despite I read the docs.
>>>>> I want to run Flume on windows server to create proof of concept
>>>>> solution. I was able to compile it successfully. I was able to make
>>>>> successful test with netcat.
>>>>> Next test that i wanted to do is with http source but unfortunately it
>>>>> did not pass.
>>>>> Here is what I do:
>>>>>
>>>>> my server config (httppost.conf):
>>>>> agent1.sources = r1
>>>>> agent1.channels = memoryChannel
>>>>> agent1.sinks = k1
>>>>>
>>>>> agent1.sources.r1.type = avro
>>>>> agent1.sources.r1.bind = localhost
>>>>> agent1.sources.r1.port = 9003
>>>>>
>>>>> agent1.sinks.k1.type = file_roll
>>>>> agent1.sinks.k1.sink.directory =
>>>>> C:\\Projects\\nickyk\\apache-flume-1.3.1-bin\\log
>>>>>
>>>>> agent1.channels.memoryChannel.type = memory
>>>>> agent1.channels.memoryChannel.capacity = 10000
>>>>> agent1.channels.memoryChannel.transactionCapacity = 1000
>>>>>
>>>>>
>>>>> agent1.sources.r1.channels = memoryChannel
>>>>> agent1.sinks.k1.channel = memoryChannel
>>>>>
>>>>> I run the server with this bat file:
>>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin\
>>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>>> set JAVA_OPTS=-Xmx60m
>>>>> set CONF=%FLUME_HOME%\conf\httppost.conf
>>>>> set AGENT=agent1
>>>>>
>>>>> %JAVA% %JAVA_OPTS%
>>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>>>>> "%FLUME_HOME%\lib\*" org.apache.flume.node.Application -f %CONF% -n
>>>>> %AGENT%
>>>>>
>>>>> netstat does not show any listener on port 9003? Is this normal or
>>>>> there must be a listener?
>>>>>
>>>>> I run avro client with this bat file:
>>>>> set FLUME_HOME=C:\Projects\nickyk\apache-flume-1.3.1-bin
>>>>> set JAVA="C:\Java\jdk1.6.0_45\bin\java.exe"
>>>>> set JAVA_OPTS=-Xmx60m
>>>>> set AVRO_HOST=localhost
>>>>> set AVRO_PORT=9003
>>>>> set SENDFILE=%FLUME_HOME%\logs\data.log
>>>>> set HEADFILE=%FLUME_HOME%\conf\header.txt
>>>>>
>>>>> %JAVA% %JAVA_OPTS%
>>>>> -Dlog4j.configuration=file:///%FLUME_HOME%\conf\log4j.properties -cp
>>>>> "%FLUME_HOME%\lib\*" org.apache.flume.client.avro.AvroCLIClient -H
>>>>> %AVRO_HOST% -p %AVRO_PORT% -F %SENDFILE% --headerFile %HEADFILE%
>>>>>
>>>>>
>>>>> Then I get this output:
>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin>"C:\Java\jdk1.6.0_45\bin\java.exe"
>>>>> -Xmx60m
>>>>> -Dlog4j.configuration=file:///C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\log4j.properties
>>>>> -cp "C:\Projects\nickyk\apache-flume-1.3.1-bin\lib\*"
>>>>> org.apache.flume.client.avro.AvroCLIClient -H localhost -p 9003 -F
>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\logs\data.log --headerFile
>>>>> C:\Projects\nickyk\apache-flume-1.3.1-bin\conf\header.txt
>>>>> 2013-06-20 17:55:58,285 (main) [DEBUG -
>>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>>>>> Inserting Header Key [filename] header value [datalog]
>>>>> 2013-06-20 17:55:58,287 (main) [DEBUG -
>>>>> org.apache.flume.client.avro.AvroCLIClient.parseHeaders(AvroCLIClient.java:101)]
>>>>> Inserting Header Key [host] header value [zhouhh]
>>>>> 2013-06-20 17:55:58,295 (main) [DEBUG -
>>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
>>>>> Batch size string = 5
>>>>> 2013-06-20 17:55:59,358 (main) [ERROR -
>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:79)]
>>>>> Unable to open connection to Flume. Exception follows.
>>>>> org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost,
>>>>> port: 9003 }: RPC connection error
>>>>> at
>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)
>>>>> at
>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)
>>>>> at
>>>>> org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)
>>>>> at
>>>>> org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)
>>>>> at
>>>>> org.apache.flume.client.avro.AvroCLIClient.run(AvroCLIClient.java:180)
>>>>> at
>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:71)
>>>>> Caused by: java.io.IOException: Error connecting to localhost/
>>>>> 127.0.0.1:9003
>>>>> at
>>>>> org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
>>>>> at
>>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:203)
>>>>> at
>>>>> org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:152)
>>>>> at
>>>>> org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)
>>>>> ... 5 more
>>>>> Caused by: java.net.ConnectException: Connection refused: no further
>>>>> information
>>>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>> at
>>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)
>>>>> at
>>>>> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)
>>>>> at
>>>>> org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> 2013-06-20 17:55:59,369 (main) [DEBUG -
>>>>> org.apache.flume.client.avro.AvroCLIClient.main(AvroCLIClient.java:84)]
>>>>> Exiting
>>>>>
>>>>>
>>>>> My question is what I am doing wrong and what I need to test in order
>>>>> to fix this situatioin.
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>> best regards,
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> thanks
>>> ashish
>>>
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>
>>
>>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
