Hi Vishal, what you said "1 core launching 2 thread" could be the view from system level, right? In yarn mode, what application got is vCore, so carbon should not take that as a physical core.
On 2020/04/16 16:15:23, Kumar Vishal <kumarvishal1...@gmail.com> wrote: > Hi Manhua, > In addition to what Ajantha said. All the configuration are exposed to the > user. > And by default no of threads is 2, so in 1 core launching 2 thread is okay. > > -Regarda > Kumar Vishal > > On Wed, 15 Apr 2020 at 9:55 PM, Ajantha Bhat <ajanthab...@gmail.com> wrote: > > > Hi Manhua, > > > > For only No sort and Local sort, we don't follow spark task launch logic. > > we have our own logic of one node one task. And inside that task we can > > control resource by configuration (carbon.number.of.cores.while.loading) > > > > As you pointed in the above mail, *N * C is controlled by configuration* > > and the default value of C is 2. > > *I see over use cluster problem only if you configure it badly.* > > > > Do you have any suggestion to the change design? Feel free to raise a > > discussion and work on it. > > > > Thanks, > > Ajantha > > > > On Tue, Apr 14, 2020 at 6:06 PM Liang Chen <chenliang6...@gmail.com> > > wrote: > > > > > OK, thank you feedbacked this issue, let us look into it. > > > > > > Regards > > > Liang > > > > > > > > > Manhua Jiang wrote > > > > Hi All, > > > > Recently, I found carbon over-use cluster resources. Generally the > > design > > > > of carbon work flow does not act as common spark task which only do one > > > > small work in one thread, but the task has its mind/logic. > > > > > > > > For example, > > > > 1.launch carbon with --num-executors=1 but set > > > > carbon.number.of.cores.while.loading=10; > > > > 2.no_sort table with multi-block input, N Iterator > > > > <CarbonRowBatch> > > > > for example, carbon will start N tasks in parallel. And in each task > > the > > > > CarbonFactDataHandlerColumnar has model.getNumberOfCores() (let's say > > C) > > > > in ProducerPool. Totally launch N*C threads; ==>This is the case makes > > me > > > > take this as serious problem. To many threads stucks the executor to > > send > > > > heartbeat and be killed. > > > > > > > > So, the over-use is related to usage of threadpool. > > > > > > > > This would affect the cluster overall resource usage and may lead to > > > wrong > > > > performance results. > > > > > > > > I hope this get your notice while fixing or writing new codes. > > > > > > > > > > > > > > > > > > -- > > > Sent from: > > > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/ > > > > > >
