Hi, Daria The lost tuples may going into two places: 1) message_queue in netty-client, which will cause memory leak; 2) netty internal buffer, if connection lose, all tuples in it get lost.
so, check your worker log to see if there is any connection lost error Regards 2014-05-16 12:18 GMT+08:00 李家宏 <jh.li...@gmail.com>: > I am running into the same issue. Where do the lost tuples gone ? If they > were queueing in the transport layer, the memory usage should keep > increasing, but I didn't see any noticeable memory leaks. > > Does storm have the guarantee all tuples sent from task A to task B will > be received by task B ? Moreover, are they in order ? > > Can anybody give any idea on this issue > > > 2014-04-02 20:56 GMT+08:00 Daria Mayorova <d.mayor...@gmail.com>: > > Hi everyone, >> >> We are having some issues with the Storm topology. The problem is that >> some tuples are being lost somewhere in the topology. Just after the >> topology is deployed, it goes pretty well, but after several hours it >> starts to loose a significant amount of tuples. >> >> From what we've found out from the logs, the thing is that the tuples >> exit one bolt/spout, and never enter the next bolt. >> >> Here is some info about the topology: >> >> - The version is 0.9.1, and netty is used as transport >> - The spout is extending BaseRichSpout, and the bolts extend >> BaseBasicBolt >> - The spout is using Kestrel message queue >> - The cluster consists of 2 nodes: zookeeper, nimbus and ui are >> running on one node, and the workers run on another node. I am attaching >> the content of the config files below. We have also tried running the >> workers on another node (the same where nimbus and zookeeper are), and >> also >> on both nodes, but the behavior is the same. >> >> According to the Storm UI there are no Failed tuples. Can anybody give >> any idea of what might be the reason of the tuples getting lost? >> >> Thanks. >> >> *Storm config (storm.yaml)* >> (In case both nodes have workers running, the configuration is the same >> on both nodes, just the "storm.local.hostname" parameter changes) >> >> storm.zookeeper.servers: >> - "zkserver1" >> nimbus.host: "nimbusserver" >> storm.local.dir: "/mnt/storm" >> supervisor.slots.ports: >> - 6700 >> - 6701 >> - 6702 >> - 6703 >> storm.local.hostname: "storm1server" >> >> nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" >> ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >> supervisor.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" >> worker.childopts: "-Xmx3548m -Djava.net.preferIPv4Stack=true" >> >> storm.cluster.mode: "distributed" >> storm.local.mode.zmq: false >> storm.thrift.transport: >> "backtype.storm.security.auth.SimpleTransportPlugin" >> >> storm.messaging.transport: "backtype.storm.messaging.netty.Context" >> >> storm.messaging.netty.server_worker_threads: 1 >> storm.messaging.netty.client_worker_threads: 1 >> storm.messaging.netty.buffer_size: 5242880 #5MB buffer >> storm.messaging.netty.max_retries: 30 >> storm.messaging.netty.max_wait_ms: 1000 >> storm.messaging.netty.min_wait_ms: 100 >> >> *Zookeeper config (zoo.cfg):* >> tickTime=2000 >> initLimit=10 >> syncLimit=5 >> dataDir=/var/zookeeper >> clientPort=2181 >> autopurge.purgeInterval=24 >> autopurge.snapRetainCount=5 >> server.1=localhost:2888:3888 >> >> *Topology configuration* passed to the StormSubmitter: >> Config conf = new Config(); >> conf.setNumAckers(6); >> conf.setNumWorkers(4); >> conf.setMaxSpoutPending(100); >> >> >> Best regards, >> Daria Mayorova >> > > > > -- > > ====================================================== > > Gvain > > Email: jh.li...@gmail.com > -- ====================================================== Gvain Email: jh.li...@gmail.com