config trident to emit no batch when no data found in kafka
hi, i am using trident, and found that when no data found in kafka, spout still emit empty batch can this be turned off? thx!
Re: Field Grouping question on optimization
You may want to look at partial key grouping feature to reduce hotspots, cases where key cardinality makes a bolt instance straggle. This feature is in 0.9.4 -Rajiv > On Apr 8, 2015, at 11:52 AM, Kashyap Mhaisekar wrote: > > Hi, > My topology is like the following -> > Spout -> Bolt A -> Bolt B -> Bolt C -> Bolt D > > the groupings between Bolt C -> Bolt D is a field grouping as Bolt D is doing > an aggregation while everything else is a shuffleGrouping. > > Use Case: > If the spout emits 100K tuples such that the emits all are grouped on the > same field, then Bolt D will need to take all the load and hence becomes very > slow. IN this case, increasing the no. of instances of Bolt D will not help > as the grouping is for the instance of Bolt D. > > Question: How can this be optimized? > > Did anyone face such a use case? Please recommend. > > Thanks > Kashyap >
Field Grouping question on optimization
Hi, My topology is like the following -> *Spout -> Bolt A -> Bolt B -> Bolt C -> Bolt D* the groupings between* Bolt C -> Bolt D* is a field grouping as Bolt D is doing an aggregation while everything else is a shuffleGrouping. Use Case: If the spout emits 100K tuples such that the emits all are grouped on the same field, then Bolt D will need to take all the load and hence becomes very slow. IN this case, increasing the no. of instances of Bolt D will not help as the grouping is for the instance of Bolt D. Question: How can this be optimized? Did anyone face such a use case? Please recommend. Thanks Kashyap
Re: Opening connection to Random IPs
Hi Jens, Thanks for your reply. We are not running supervisor in LXC / Docker containers. Its running from a physical Linux machine. - Anil On 8 Apr 2015 23:46, "Jens-U. Mozdzen" wrote: > Hi Anil, > > Zitat von Anil Kumar : > >> Hi Guys, >> >> We have problem while submitting topology where it is trying to open up >> client connection to random IPs like 70.127.0.0, 90.127.0.0 etc. >> > > are you running your Storm supervisors within Docker / LXC containers? > > Regards, > Jens > >
Re: Opening connection to Random IPs
Hi Anil, Zitat von Anil Kumar : Hi Guys, We have problem while submitting topology where it is trying to open up client connection to random IPs like 70.127.0.0, 90.127.0.0 etc. are you running your Storm supervisors within Docker / LXC containers? Regards, Jens
Opening connection to Random IPs
Hi Guys, We have problem while submitting topology where it is trying to open up client connection to random IPs like 70.127.0.0, 90.127.0.0 etc. Could you please help us in finding out why it is trying to do this ? Extract from our logs are: 6845 [main-SendThread(90.127.0.0:2000)] INFO org.apache.storm.zookeeper.ClientCnxn - Opening socket connection to server 90.127.0.0/90.127.0.0:2000. Will n ot attempt to authenticate using SASL (unknown error) 6851 [main-SendThread(90.127.0.0:2000)] WARN org.apache.storm.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connecti on and attempting reconnect java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_11] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) ~[na:1.8.0_11] at org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[storm-core-0.9.3.jar:0.9.3] at org.apache.storm.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) ~[storm-core-0.9.3.jar:0.9.3] - Anil
