Hi, I have a big database table(1 million plus records) in oracle. I need to query records based on input numbers. For this use case, I am doing below steps
I am creating two data frames. DF1 = I am computing this DF1 using sql query. It has one million + records. DF2 = I have a list of numbers. I am converting list of input numbers to data-frame I am converting DF1 and DF2 to register temp table and forming sql query. It will return input number of records Steps :- DF1.registerTempTable("E1") DF2.registerTempTable("E2") DF3 = sqlContext.sql(select * from E1, E2 where E1.id = E2.id) DF3.map(row => (row(0),row(1),row(2))).saveToCassandra(keyspace, table1) *query :-* How D3 records will fetch? Is DF1 load entire table data(1 million plus records) into memory when joining with DF2 ? (Or) It will fetch only DF2 matched records from oracle and load into memory. Please clarify and let me know my approach is correct Regards, Rajesh