Re: [Proposal] Proposal to change default value of two parameters for data loading

2018-10-17 Thread Jacky Li
+1 > 在 2018年10月15日,下午9:03,xuchuanyin 写道: > > Hi, all: > > About a year ago, we introduced 'multiple dirs for temp data' to solve disk > hotspot problem in data loading. > > This feature enables carbon randomly pick one of the local directories > configured in yarn-local-dirs when it writes a

Re: [Proposal] Proposal to change default value of two parameters for data loading

2018-10-15 Thread xuchuanyin
Yes, it needs further modification to meet the requirement -- an additional property is needed to handle this, we can configure multiple directories there. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Proposal] Proposal to change default value of two parameters for data loading

2018-10-15 Thread xm_zzc
Hi chuanyin: +1 for this. One question: these two parameters just support for on-yarn mode, right? Can it support to config other path instead of /tmp path when user run app without on-yarn mode? -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

[Proposal] Proposal to change default value of two parameters for data loading

2018-10-15 Thread xuchuanyin
Hi, all: About a year ago, we introduced 'multiple dirs for temp data' to solve disk hotspot problem in data loading. This feature enables carbon randomly pick one of the local directories configured in yarn-local-dirs when it writes any temp files to disk (for example: sort temp files and fact