Hi Naveen, I had similar situation. Two things you can do : 1. Decouple file reading from cache streaming, so that both can be handled in separate threads asynchronously.
2. Once you have data from csv in collection, use use parallelStreams to add data in streamer with multiple threads. Thanks, Gaurav On 09-Mar-2018 3:05 PM, "Naveen" <[email protected]> wrote: > Hi DH > > I am not using any custom streamReciever, my requirement is very simple. > Have huge data in CSV, reading line by line and parsing the line and > populating the POJO and using the DataStreamer to load data into cache. > > while (sc.hasNextLine()) { > ct++; > String line = sc.nextLine(); > String[] tokens = line.split(Constants.Delimter, > -1); > aASSOCIATED_PARTIES = new ASSOCIATED_PARTIES(); > > aASSOCIATED_PARTIES.setASSOCIATED_PARTY_ID(tokens[ > 1]); > aASSOCIATED_PARTIES.setUPDATEDBY(tokens[3]); > > > streamer.perNodeBufferSize(Constants.perNodeBufferSize); > > streamer.perNodeParallelOperations(Constants.perNodeParallelOperations); > > streamer.addData(tokens[0], aASSOCIATED_PARTIES); > } > > As you mentioned, I have made sure that one DataStreamer for each client, > after this change, it stopped failing. > Any ways we can improve this performance. > > Thanks > Naveen > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
