On Sat, Mar 14, 2020 at 5:56 PM Andrew Melo wrote:
> Sorry, I'm from a completely different field, so I've inherited a completely
> different vocabulary. So thanks for bearing with me :)
>
> I think from reading your response, maybe the confusion is that HTCondor is a
> completely different
Hi Sean
On Fri, Mar 13, 2020 at 6:46 PM Sean Owen wrote:
> Do you really need a new cluster per user? and if so, why specify N
> workers > M machines? I am not seeing a need for that. I don't even
> think 2 workers on the same host makes sense, as they are both
> managing the same resources; it
Do you really need a new cluster per user? and if so, why specify N
workers > M machines? I am not seeing a need for that. I don't even
think 2 workers on the same host makes sense, as they are both
managing the same resources; it only exists for test purposes AFAICT.
What you are trying to do
Hi Xingbo, Sean,
On Fri, Mar 13, 2020 at 12:31 PM Xingbo Jiang wrote:
> Andrew, could you provide more context of your use case please? Is it like
> you deploy homogeneous containers on hosts with available resources, and
> each container launches one worker? Or you deploy workers directly on
Andrew, could you provide more context of your use case please? Is it like
you deploy homogeneous containers on hosts with available resources, and
each container launches one worker? Or you deploy workers directly on hosts
thus you could have multiple workers from the same application on the same
You have multiple workers in one Spark (standalone) app? this wouldn't
prevent N apps from each having a worker on a machine.
On Fri, Mar 13, 2020 at 11:51 AM Andrew Melo wrote:
>
> Hello,
>
> On Fri, Feb 28, 2020 at 13:21 Xingbo Jiang wrote:
>>
>> Hi all,
>>
>> Based on my experience, there is
Hello,
On Fri, Feb 28, 2020 at 13:21 Xingbo Jiang wrote:
> Hi all,
>
> Based on my experience, there is no scenario that necessarily requires
> deploying multiple Workers on the same node with Standalone backend. A
> worker should book all the resources reserved to Spark on the host it is
>
Hi Prashant,
I guess you are referring to the local-cluster mode? AFAIK the
local-cluster mode has not been mentioned at all in the user guide, thus it
should only be used in Spark tests. Also, there are a few differences
between having multiple workers on the same node and having one worker on
It was by design, one could run multiple workers on his laptop for trying
out or testing spark in distributed mode, one could launch multiple workers
and see how resource offers and requirements work. Certainly, I have not
commonly seen, starting multiple workers on the same node as a practice so
Thanks Sean for your input, I really think it could simplify Spark
Standalone backend a lot by only allowing a single worker on the same host,
also I can confirm this deploy model can satisfy all the workloads deployed
on Standalone backend AFAIK.
Regarding the case multiple distinct Spark
I'll admit, I didn't know you could deploy multiple workers per
machine. I agree, I don't see the use case for it? multiple executors,
yes of course. And I guess you could imagine multiple distinct Spark
clusters running a worker on one machine. I don't have an informed
opinion therefore, but
Hi all,
Based on my experience, there is no scenario that necessarily requires
deploying multiple Workers on the same node with Standalone backend. A
worker should book all the resources reserved to Spark on the host it is
launched, then it can allocate those resources to one or more executors
12 matches
Mail list logo
