Hello! I just wanted to add that you can't create tables in parallel. Table creation is sequential since it involves a Partition Map Exchange, which is a blocking operation.
It can become a bottleneck quickly. Regards, -- Ilya Kasnacheev пт, 15 мая 2020 г. в 07:24, adipro <aditya.har...@zohocorp.com>: > We created a task which crawls website. We have an SQL table, where the > application writes data while crawling website. But the application has > several threads running in parallel, each thread working on a certain > website. The amount of data that is being generated on that SQL table is > huge and we want to create a separate SQL table for each thread. > > Now if we crawl 100s of websites in parallel, we'll create 100s of threads. > So can we create 100s of SQL tables? And right after the crawling is done > we'll delete/destroy those SQL tables. > > I am asking this question because, we have a limit for now like each > website > should be crawled for "n" number of pages. If that "n" increases the amount > of data one SQL table holds increases linearly and it's not ideal for > future > purposes. > > Does Ignite hold connections to 100s of SQL tables in parallel assuming > that > ulimit is set to whatever is required? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
