Vladimir Ozerov created IGNITE-6699:
---------------------------------------

             Summary: Optimize client-side data streamer performance
                 Key: IGNITE-6699
                 URL: https://issues.apache.org/jira/browse/IGNITE-6699
             Project: Ignite
          Issue Type: Task
      Security Level: Public (Viewable by anyone)
          Components: streaming
    Affects Versions: 2.3
            Reporter: Vladimir Ozerov
             Fix For: 2.4


Currently if a user has several server nodes and a single client node with 
single thread pushing data to streamer, he will not be able to load data at 
maximum speed. On the other hand, if he start several data loading threads, 
throughput will increase. 

One of root causes of this is bad data streamer design. Method 
{{IgniteDataStreamer.addData(K, V)}} returns new feature for every operation, 
this is too fine grained approach. Also it generates a lot of garbage and 
causes contention on streamer internals. 

Proposed implementation flow:
1) Compare performance of {{addData(K, V)}} vs {{addData(Collection)}} methods 
from one thread in distributed environment. The latter should show considerably 
higher throughput.
2) Users should receive per-batch features, rather than per-key. 
3) Try caching thread data in some collection until it is large enough to avoid 
contention and unnecessary allocations.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to