You can pass a ParametersProvider to the jdbc input format in order to parallelize the fetch. Of course you don't have to kill the mysql server with too many request in parallel so you'll probably put a limit to the parallelism of the input format.
On Tue, 4 Dec 2018, 17:31 miki haiat <miko5...@gmail.com wrote: > HI , > I want to query some sql table that contains ~80m rows. > > There is a few ways to do that and i wonder what is the best way to do > that . > > > 1. Using JDBCINPUTFORMAT -> convert to dataset and output it without > doing any logic in the dataset, passing the full query in the > JDBCINPUTFORMAT set query parameters. > 2. Using JDBCINPUTFORMAT select all the data from table then > desirelaze it ->convert to dataset and preforming logic. > > > Or something else that is much efficient ? > > Thanks, > > Miki > >
