This also came up here: http://mail-archives.apache.org/mod_mbox/flume-user/201207.mbox/%3CCAFukC=6C=jBayVSVAQdiJA+6kWUQ46_uXbDWBbUjDxb+U=a...@mail.gmail.com%3E
I think it would be great if we could get a standalone python client working. Brock On Mon, Jul 30, 2012 at 3:34 AM, Juhani Connolly <[email protected]> wrote: > I'm playing around with making a standalone tail client in python(so that I > can access inode data) that tracks position in a file and then sends it > across avro to an avro sink. > > However I'm having issues with the avro part of this and wondering if anyone > more familiar with it could help. > > I took the flume.avdl file and converted it using "java -jar > ~/Downloads/avro-tools-1.6.3.jar idl flume.avdl flume.avpr" > > I then run it through a simple test program to see if its sending the data > correctly and it sends from the python client fine, but the sink end OOM's > because presumably the wire format is wrong: > > 2012-07-30 17:22:57,565 INFO ipc.NettyServer: [id: 0x5fc6e818, > /172.22.114.32:55671 => /172.28.19.112:41414] OPEN > 2012-07-30 17:22:57,565 INFO ipc.NettyServer: [id: 0x5fc6e818, > /172.22.114.32:55671 => /172.28.19.112:41414] BOUND: /172.28.19.112:41414 > 2012-07-30 17:22:57,565 INFO ipc.NettyServer: [id: 0x5fc6e818, > /172.22.114.32:55671 => /172.28.19.112:41414] CONNECTED: > /172.22.114.32:55671 > 2012-07-30 17:22:57,646 WARN ipc.NettyServer: Unexpected exception from > downstream. > java.lang.OutOfMemoryError: Java heap space > at java.util.ArrayList.<init>(ArrayList.java:112) > at > org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154) > at > org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) > at > org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351) > at > org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282) > at > org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > 2012-07-30 17:22:57,647 INFO ipc.NettyServer: [id: 0x5fc6e818, > /172.22.114.32:55671 :> /172.28.19.112:41414] DISCONNECTED > 2012-07-30 17:22:57,647 INFO ipc.NettyServer: [id: 0x5fc6e818, > /172.22.114.32:55671 :> /172.28.19.112:41414] UNBOUND > 2012-07-30 17:22:57,647 INFO ipc.NettyServer: [id: 0x5fc6e818, > /172.22.114.32:55671 :> /172.28.19.112:41414] CLOSED > > I've dumped the test program and its output > > http://pastebin.com/1DtXZyTu > http://pastebin.com/T9kaqKHY -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
