Hello! Then you will have to simplify your query or split it into parts to avoid pulling too many rows into RAM at the same time. Note that other databases also have this issue, but with Apache Ignite it's more severe since it would not hold temporary mid-query tables on disk.
As far as I know spark has a mode where it does joins on its side, and you can probably sidestep Ignite's limitations that way. Regards, -- Ilya Kasnacheev пн, 17 дек. 2018 г. в 17:26, yangjiajun <[email protected]>: > Hello! Thanks for your reply! > > Unfortunately,the above query is a sub query of a big query.So I think I > can't overcome it.How about integration with spark?Will spark deal with > such > queries in a smart way? > > > ilya.kasnacheev wrote > > Hello! > > > > The recommendation here is to use Lazy SQL Queries. > > > https://apacheignite-sql.readme.io/docs/performance-and-debugging#section-result-set-lazy-loading > > Then, be careful to process data in small batches and avoid holding on to > > large subsets of result set. > > > > Regards, > > -- > > Ilya Kasnacheev > > > > > > пн, 17 дек. 2018 г. в 16:08, yangjiajun < > > > 1371549332@ > > >>: > > > >> Hello. > >> > >> I do a simple sql join operation on a ignite node which is version 2.6 > >> and > >> has 22GB heap memory and 22 GB off-heap memory.I also enable > >> persistence.The > >> query exhausted ignite's heap memory quickly.How can I overcome this > >> situation. > >> > >> Here is my query: > >> explain > >> SELECT > >> t2.*, > >> t1.SHIPDOMESTICFLAG, > >> t1.RESOURCEALLOCATIONTYPE, > >> t1.URGENTFLAG, > >> t1.EXPEDITEDFLAG, > >> CONCAT( '%', > >> t2.DELIVERY_CENTER_CODE, > >> '%' ) AS DELIVERY_CENTER_CODE_1 > >> FROM > >> TABLE_6978_R_1_1 t1,TABLE_6930_R_1_1 t2 > >> WHERE > >> t2.ORIGINAL_CONTRACT_NUM = t1.ORIGINALCONTRACTNUM > >> > >> And here is the query plan: > >> SELECT > >> T2.CONTRACT_NUMBER, > >> T2.ORDER_ASSEMBLY_DEPT, > >> T2.ORDER_NUMBER, > >> T2.ORIGINAL_CONTRACT_NUM, > >> T2.SHIPMENT_BATCH_DESC, > >> T2.EPD, > >> T2.REQUEST_PACK_DATE, > >> T2.ORGANIZATION_ID, > >> T2.CPP_STATUS, > >> T2.BG_TYPE, > >> T2.PSD, > >> T2.ORDER_PRIORITY, > >> T2.ADVANCED_ARRIVAL_FLAG, > >> T2.ITEM_ATTR_FLAG, > >> T2.ITEM_CODE, > >> T2.ORDERED_QUANTITY, > >> T2.DELIVERY_CENTER_CODE, > >> T2.OTO_CREATION_DATE, > >> T2.ITEMID, > >> T2.LINEID, > >> T2.ROMA3C_BSP_NATIVE_COLUMN, > >> T1.SHIPDOMESTICFLAG, > >> T1.RESOURCEALLOCATIONTYPE, > >> T1.URGENTFLAG, > >> T1.EXPEDITEDFLAG, > >> CONCAT('%', T2.DELIVERY_CENTER_CODE, '%') AS DELIVERY_CENTER_CODE_1 > >> FROM PUBLIC.TABLE_6978_R_1_1 T1 > >> /* PUBLIC.TABLE_6978_R_1_1.__SCAN_ */ > >> INNER JOIN PUBLIC.TABLE_6930_R_1_1 T2 > >> /* PUBLIC.TABLE_6930_R_1_1_ORIGINAL_CONTRACT_NUM_IDX: > >> ORIGINAL_CONTRACT_NUM = T1.ORIGINALCONTRACTNUM */ > >> ON 1=1 > >> WHERE T2.ORIGINAL_CONTRACT_NUM = T1.ORIGINALCONTRACTNUM > >> > >> > >> Table t1 has about 1.5 million records.Table t2 has 6w records. > >> > >> > >> > >> -- > >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ > >> > > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
