Re: Flume ERROR

Fri, 07 Aug 2015 07:13:36 -0700 


Hi,

Flush the firewall rules and start flume.

What OS are you using?

Thanks
Anand.

On 08/07/2015 07:24 PM, Nikhil Gs wrote:

Hello AnandKumar,
The issue is again repeating. Yesterday the files were deleting from spool but the data was not available in the destination Hbase tables, but now this morning what I have noticed is the log is same.
Earlier, as you have suggested, "verify the firewall settings. It blocks the connection it seems." Can you be more specific, I mean how can verify the firewall settings. Any suggestion? 

Thanks in advance.

8:45:01.040 AM  ERROR   org.apa
che.flume.SinkRunner    
Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
        at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
        at 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { 
host:sdldalplhdw02.suddenlink.cequel3.com  
<http://sdldalplhdw02.suddenlink.cequel3.com>, port: 40002 }: RPC connection 
error
        at 
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:182)
        at 
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
        at 
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
        at 
org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:89)
        at org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
        at 
org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:211)
        at 
org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:272)
        at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:349)
        ... 3 more
Caused by: java.io.IOException: Error connecting 
tosdldalplhdw02.suddenlink.cequel3.com/10.48.210.244:40002  
<http://sdldalplhdw02.suddenlink.cequel3.com/10.48.210.244:40002>
        at 
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
        at 
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
        at 
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:155)
        at 
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:168)
        ... 10 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148)
        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104)
        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
        at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        ... 1 more


Regards,
Nik.
On Fri, Aug 7, 2015 at 8:28 AM, Nikhil Gs <[email protected] <mailto:[email protected]>> wrote: 

    Hello Anandkumar,

    Thank you for your time and also for the reply.

    Now, flume acts differently. I don't see any log errors, in fact
    my log now looks as below;

    2015-08-06 22:51:14,426 INFO
org.apache.flume.client.avro.ReliableSpoolingFileEv entReader: Preparing to delete file 
    /home/a_nikhil.gopishetti/pnm/GVLLCMTK03-675 79944.pf
    <http://79944.pf>
    2015-08-06 23:12:15,026 INFO
org.apache.flume.client.avro.ReliableSpoolingFileEv entReader: Preparing to delete file 
    /home/a_nikhil.gopishetti/pnm/KNTNCMTK01.SUD DENLINK.NET-7682.pf
    <http://DENLINK.NET-7682.pf>
    2015-08-06 23:12:22,030 INFO
org.apache.flume.client.avro.ReliableSpoolingFileEv entReader: Preparing to delete file 
    /home/a_nikhil.gopishetti/pnm/KNTNCMTK01.SUD DENLINK.NET-7682.pf
    <http://DENLINK.NET-7682.pf>
    2015-08-06 23:13:47,570 INFO
org.apache.flume.client.avro.ReliableSpoolingFileEv entReader: Preparing to delete file /home/a_nikhil.gopishetti/pnm/NBRNCMTK01-340 17296.pf.filepart 
    2015-08-06 23:13:57,076 INFO
org.apache.flume.client.avro.ReliableSpoolingFileEv entReader: Preparing to delete file 
    /home/a_nikhil.gopishetti/pnm/NBRNCMTK01-675 84072.pf
    <http://84072.pf>
    2015-08-06 23:14:03,581 INFO
org.apache.flume.client.avro.ReliableSpoolingFileEv entReader: Preparing to delete file 
    /home/a_nikhil.gopishetti/pnm/NBRNCMTK01-675 79944.pf
    <http://79944.pf>
    2015-08-06 23:23:23,348 INFO
org.apache.flume.client.avro.ReliableSpoolingFileEv entReader: Preparing to delete file /home/a_nikhil.gopishetti/pnm/PMLECMTK01.SUD DENLINK.NET-8426.pf.filepart 


    So the files are deleting. So when I go back to my destination
    i.e. Hbase tables, I don't see any data. Earlier, a couple of days
    back, I could clearly see that my data was deleting from the spool
    directory and it was getting into my Hbase table. But now I don't
    see any data in my destination tables. I don't know why. Any
    suggestions.

    Thanks in advance for your time and reply.

    Regards,
    Nik.

    On Thu, Aug 6, 2015 at 11:43 PM, Anandkumar Lakshmanan
    <[email protected] <mailto:[email protected]>> wrote:

        Hi Nik,

        Please verify the firewall settings. It blocks the connection
        it seems.

        Thanks
        Anand.



        On 08/07/2015 02:07 AM, Nikhil Gs wrote:

        Hello Team,

        Facing the below error very alternatively even though worked
        with different port numbers. I have pasted my flume config
        file along with the error.

        Thanks in advance.


        Below is my flume configuration

        #################################################

        # Please paste flume.conf here. Example:

        # Sources, channels, and sinks are defined per
        # agent name, in this case 'pnmtest2'.
        pnmtest2.sources  = SPOOL
        pnmtest2.channels = MemChanneltest2
        pnmtest2.sinks    = AVRO

        # For each source, channel, and sink, set
        # standard properties.
        pnmtest2.sources.SPOOL.type      = spooldir
        pnmtest2.sources.SPOOL.spoolDir      = /home/a.nikhill/pnm
        pnmtest2.sources.SPOOL.ignorePattern = \.*tmp$
        pnmtest2.sources.SPOOL.channels      = MemChanneltest2
        pnmtest2.sources.SPOOL.fileHeader    = true
        pnmtest2.sources.SPOOL.deletePolicy  = immediate
        pnmtest2.sources.SPOOL.consumeOrder  = oldest
        pnmtest2.sources.SPOOL.batchSize     = 100

        pnmtest2.sources.SPOOL.interceptors = time
        pnmtest2.sources.SPOOL.interceptors.time.type =
        org.apache.flume.interceptor.TimestampInterceptor$Builder
        pnmtest2.sources.SPOOL.deserializer  =
        com.sudnline.flume.WholeFileDeserializer$Builder

        pnmtest2.sinks.AVRO.type   = avro
        pnmtest2.sinks.AVRO.channel  = MemChanneltest2
        pnmtest2.sinks.AVRO.hostname =
        sdldalplhdw02.sudnline.cequel3.com
        <http://sdldalplhdw02.suddenlink.cequel3.com/>
        pnmtest2.sinks.AVRO.port     = 40002
        pnmtest2.sinks.AVRO.batch-size = 100
        pnmtest2.sinks.AVRO.connect-timeout = 40000


        # pnmtest2.sinks.HDFS.type     = hdfs
        # pnmtest2.sinks.HDFS.channel    = MemChannel2
        # pnmtest2.sinks.HDFS.hdfs.path = /user/flume/poll/%Y/%m/%d/%H/
        # pnmtest2.sinks.HDFS.hdfs.fileType = DataStream
        # pnmtest2.sinks.HDFS.hdfs.writeFormat = Text
        # pnmtest2.sinks.HDFS.hdfs.batchSize = 100
        # pnmtest2.sinks.HDFS.hdfs.rollSize = 0
        # pnmtest2.sinks.HDFS.hdfs.rollCount = 1000
        # pnmtest2.sinks.HDFS.hdfs.rollInterval = 600

        # Other properties are specific to each type of
        # source, channel, or sink. In this case, we
        # specify the capacity of the memory channel.

        #pnmtest2.channels.MemChanneltest1.capacity = 10000
        #pnmtest2.channels.MemChanneltest1.type   = memory

        pnmtest2.channels.MemChanneltest2.capacity = 1000000
        pnmtest2.channels.MemChanneltest2.type   = memory


        Below is my error.

                ERROR   org.apache.flume.SinkRunner     

                Unable to deliver event. Exception follows.
                org.apache.flume.EventDeliveryException: Failed to send events
                        at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:392)
                        at 
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
                        at 
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
                        at java.lang.Thread.run(Thread.java:745)
                Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { 
host:sdldalplhdw02.suddenlink.cequel3.com  
<http://sdldalplhdw02.suddenlink.cequel3.com/>, port: 40002 }: RPC connection 
error
                        at 
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:182)
                        at 
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:121)
                        at 
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:638)
                        at 
org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:89)
                        at 
org.apache.flume.sink.AvroSink.initializeRpcClient(AvroSink.java:127)
                        at 
org.apache.flume.sink.AbstractRpcSink.createConnection(AbstractRpcSink.java:211)
                        at 
org.apache.flume.sink.AbstractRpcSink.verifyConnection(AbstractRpcSink.java:272)
                        at 
org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:349)
                        ... 3 more
                Caused by: java.io.IOException: Error connecting 
tosdldalplhdw02.suddenlink.cequel3.com/10.48.210.244:40002  
<http://sdldalplhdw02.suddenlink.cequel3.com/10.48.210.244:40002>
                        at 
org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:292)
                        at 
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:206)
                        at 
org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:155)
                        at 
org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:168)
                        ... 10 more
                Caused by: java.net.ConnectException: Connection refused
                        at sun.nio.ch.SocketChannelImpl.checkConnect(Native 
Method)
                        at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
                        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:148)
                        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:104)
                        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:78)
                        at 
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
                        at 
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
                        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)




        Regards,
        Nik.

Reply via email to