Fwd: Problem in Understanding concept of Physical Cores

Regards Aniruddh -- Forwarded message -- From: Aniruddh Sharma Date: Fri, Jul 17, 2015 at 6:05 PM Subject: Re: Problem in Understanding concept of Physical Cores To: user Dear Community Request to help on below queries they are unanswered. Thanks and Regards Aniruddh On Wed

Re: Problem in Understanding concept of Physical Cores

Dear Community Request to help on below queries they are unanswered. Thanks and Regards Aniruddh On Wed, Jul 15, 2015 at 12:37 PM, Aniruddh Sharma wrote: > Hi TD, > > Request your guidance on below 5 queries. Following is the context of them > that I would use to evaluate b

Re: Problem in Understanding concept of Physical Cores

this both in Yarn and Stand Alone mode. Thanks and Regards Aniruddh On Fri, Jul 10, 2015 at 11:45 AM, Aniruddh Sharma wrote: > Hi TD, > > Thanks for elaboration. I have further doubts based on further test that > I did after your guidance > > Case 1: Standalone Spark-- > I

Re: How to speed up Spark process

Hi Deepak Not 100% sure , but please try increasing (--executor-cores ) to twice the number of your physical cores on your machine. Thanks and Regards Aniruddh On Tue, Jul 14, 2015 at 9:49 AM, ÐΞ€ρ@Ҝ (๏̯͡๏) wrote: > Its been 30 minutes and still the partitioner has not completed yet, its > eve

Re: spark streaming doubt

Hi Sushant/Cody, For question 1 , following is my understanding ( I am not 100% sure and this is only my understanding, I have asked this question in another words to TD for confirmation which is not confirmed as of now). Following is my understanding. In accordance with tasks created in proporti

Re: Problem in Understanding concept of Physical Cores

ating system underneath has only one core (physical or virtual), > then those two thread will be multiplexing the resources of that core. > > > > On Thu, Jul 9, 2015 at 1:41 AM, Aniruddh Sharma > wrote: > >> Thanks for revert.I st

Number of Threads in Executor to process Tasks

Hi I am new to Spark. I am confused between correlation in threads and physical cores. As per my understanding, according to number of partitions in data set, number of tasks is created. For example I have a machine which has 10 physical cores and I have data set which has 100 partitions then in

Re: Problem in Understanding concept of Physical Cores

ses as many threads as the number of cores that VM has. > Between 1 and 2 VM cores, the behavior was as expected. > Test 2: When you specified master as local[2], it used two threads. > > HTH > > TD > > On Wed, Jul 8, 2015 at 4:21 AM, Aniruddh Sharma > wrote: > >> Hi &

Out of Memory Errors on less number of cores in proportion to Partitions in Data

Hi, I am new to Spark. I have done following tests and I am confused in conclusions. I have 2 queries. Following is the detail of test Test 1) Used 11 Node Cluster where each machine has 64 GB RAM and 4 physical cores. I ran a ALS algorithm using MilLib on 1.6 GB data set. I ran 10 executors and

Problem in Understanding concept of Physical Cores

Hi I am new to Spark. Following is the problem that I am facing Test 1) I ran a VM on CDH distribution with only 1 core allocated to it and I ran simple Streaming example in spark-shell with sending data on port and trying to read it. With 1 core allocated to this nothing happens in my strea