Re: Loading Data from RDMS to ignite using Data Streamer

[nithin91]

Hi

Currently i am able to load 5.5 million data from Oracle Table to Ignite in
1 hr using JDBC Pojo Configuration.So i was curious that how much time will
it take to load the data using data streamers as a result of which i tried
to load 10,000 records using the Data Streamers and it took almost half an
hour.
So wanted to know whether technique i implemented for  Data Streamers to
load Data from Oracle to Ignite is correct or wrong. PFB for the psuedo code
i implemented for  Data Streamers to load Data from Oracle to Ignite.
Using this  method it is taking lot of time which is as expected as we are
looping through each
and every record .

try (IgniteDataStreamer stmr = ignite.dataStreamer("CacheName")) {
// Stream entries.


while(rs.hasNext()){-->rs is JDBC result set

Key keyobj=new Key(rs.getString(1));
Value valueobj=new
Value(rs.getString(2),rs.getString(3),rs.getString(4));
stmr.addData(keyobj , valueobj);
}
}

Can you please share a sample code streaming data  from any RDBMS to ignite.
Also can you please let me know if an external streamer is needed to stream
data from RDBMS to ignite



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Loading Data from RDMS to ignite using Data Streamer

[akurbanov]

Since this feels like as a performance question, I would start with
identifying the bottleneck in this case.

How much data are you going to load from Oracle?

What is consuming the most time while streaming, reading data from Oracle or
streaming itself? How high the resource usage while streaming (CPU/memory
consumption)? This will define what should be tuned/changed.

What current numbers are for loadCache(null) vs DataStreamer?

Best regards,
Anton



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Loading Data from RDMS to ignite using Data Streamer

[nithin91]

Hi ,

I am new to Ignite. i am able to load data from oracle into ignite using
Cache Pojo Store method ignite.cache.("cachename").loadcache(null) . But in
the documentation it was mentioned that Data Steamers will provide better 
performance when loading data into ignite cache.

*Documentation Reference:*

https://apacheignite.readme.io/docs/data-streamers

Following is the example provided for Data Streamers in the documentation

// Get the data streamer reference and stream data.
try (IgniteDataStreamer stmr =
ignite.dataStreamer("myStreamCache")) {
// Stream entries.
for (int i = 0; i < 10; i++)
stmr.addData(i, Integer.toString(i));
}

I have following questions by taking  this example as reference to stream
data from oracle table to ignite 
cache.

1. Is an external streamer needed to stream data from oracle to ignite to
use data streaming technique
as mentioned in above example.

2. If an external streamer is not required, then do we need to loop through
the jdbc result set and add data to the cache.Please find below for the
pseudo code implementation of the same.

try (IgniteDataStreamer stmr = ignite.dataStreamer("CacheName")) {
// Stream entries.


while(rs.hasNext()){-->rs is JDBC result set

Key keyobj=new Key(rs.getString(1));
Value valueobj=new
Value(rs.getString(2),rs.getString(3),rs.getString(4));
stmr.addData(keyobj , valueobj);
}
}

When i tried to load data into ignite cache from oracle using this  method
it is taking lot of time which is as expected as we are looping through each
and every record.Is this right method of implementing Data Streamers to load
data from Oracle to Ignite.If this is not right way,can anyone share sample
code to stream data from any RDBMS table to Ignite cache.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/