I have a table of size 5GB, and want to load selective rows into dataframe instead of loading the entire table in memory,
For me memory is a constraint hence , and i would like to peridically load few set of rows and perform dataframe operations on it, , for the "dbtable" is there a way to perform select * from master_schema where 'TID' = '100_0'; which can load only this to memory as dataframe . Currently I'm using code as below val df = sqlContext.read .format("jdbc") .option("url", url) .option("dbtable", "master_schema").load() Thansk, Sujeet