The configuration is 16GB ram and 1TB HD. have a single node Spark cluster. Even after setting driver memory to 5g and executor memory to 3g, I get this error. The size of the data set is 350 KB and the set that it works well is hardly few KBs.
On Mon, Feb 2, 2015 at 1:18 PM, Arush Kharbanda <ar...@sigmoidanalytics.com> wrote: > Hi Deep, > > What is your configuration and what is the size of the 2 data sets? > > Thanks > Arush > > On Mon, Feb 2, 2015 at 11:56 AM, Deep Pradhan <pradhandeep1...@gmail.com> > wrote: > >> I did not check the console because once the job starts I cannot run >> anything else and have to force shutdown the system. I commented parts of >> codes and I tested. I doubt it is because of union. So, I want to change it >> to something else and see if the problem persists. >> >> Thank you >> >> On Mon, Feb 2, 2015 at 11:53 AM, Jerry Lam <chiling...@gmail.com> wrote: >> >>> Hi Deep, >>> >>> How do you know the cluster is not responsive because of "Union"? >>> Did you check the spark web console? >>> >>> Best Regards, >>> >>> Jerry >>> >>> >>> On Mon, Feb 2, 2015 at 1:21 AM, Deep Pradhan <pradhandeep1...@gmail.com> >>> wrote: >>> >>>> The cluster hangs. >>>> >>>> On Mon, Feb 2, 2015 at 11:25 AM, Jerry Lam <chiling...@gmail.com> >>>> wrote: >>>> >>>>> Hi Deep, >>>>> >>>>> what do you mean by stuck? >>>>> >>>>> Jerry >>>>> >>>>> >>>>> On Mon, Feb 2, 2015 at 12:44 AM, Deep Pradhan < >>>>> pradhandeep1...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> Is there any better operation than Union. I am using union and the >>>>>> cluster is getting stuck with a large data set. >>>>>> >>>>>> Thank you >>>>>> >>>>> >>>>> >>>> >>> >> > > > -- > > [image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com> > > *Arush Kharbanda* || Technical Teamlead > > ar...@sigmoidanalytics.com || www.sigmoidanalytics.com >
