Storm is not your bottleneck. Check your Storm code to 1) ensure you're parallelizing your writes and 2) you're batching writes to your external resources if possible. Some quick napkin math shows you only doing 110 writes/s, which seems awfully low.
Michael Rose (@Xorlev <https://twitter.com/xorlev>) Senior Platform Engineer, FullContact <http://www.fullcontact.com/> [email protected] On Mon, Sep 22, 2014 at 8:05 PM, Kushan Maskey < [email protected]> wrote: > I am trying to load 20 M records into Cassandra database through > Kafka-Storm. I am able to post all the data in 5 mins into Kafka. But > reading it from storm and inserting into Cassandra, Couch and Solr is kind > of very slow. It has been running for past 5 hours and so far only 2 > Million records. > > How do I make the storm perform faster? Coz in this pace it will take > couple of days to load all the data. > > -- > Kushan Maskey > >
