Re: Is it able to specify the percentage of memory instead of # of MB
Sumit, Yes, that's the reason. Best, Siyuan On Tue, Dec 2, 2014 at 5:33 PM, Sumit Mohanty smoha...@hortonworks.com wrote: Siyuan, can you clarify the underlying requirement that requires an application to specify a percentage of available memory? One, as Steve suggested, could be to ask for 51% of available on node to guarantee that only one instance can be active on the node. Were there other reasons? -Sumit On Tue, Dec 2, 2014 at 5:29 PM, Ted Yu yuzhih...@gmail.com wrote: I created SLIDER-691 for this feature. Cheers On Tue, Dec 2, 2014 at 7:09 AM, Steve Loughran ste...@hortonworks.com wrote: I don't even think we can ask yarn for the percentage of memory I do understand what you are trying to do: ask for 51%. We could do something where we can query YARN for the maximum amount an app can ask for (the AM finds this out when registered), and the capacity could be derived as a percentage of that. Rather than percentage of available RAM, it's just a fraction of what you are allowed. On 2 December 2014 at 01:49, Sumit Mohanty smoha...@hortonworks.com wrote: That is not possible. The number provided (# of MB) is used to ask for container from YARN and YARN only accepts numbers. % of total on each node also becomes an issue when nodes can have varying value of total memory available. This is common for heterogeneous clusters where hosts may have different value for available memory and core, etc. On Mon, Dec 1, 2014 at 1:39 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Is it able to specify the percentage of memory instead of # of MB
I don't even think we can ask yarn for the percentage of memory I do understand what you are trying to do: ask for 51%. We could do something where we can query YARN for the maximum amount an app can ask for (the AM finds this out when registered), and the capacity could be derived as a percentage of that. Rather than percentage of available RAM, it's just a fraction of what you are allowed. On 2 December 2014 at 01:49, Sumit Mohanty smoha...@hortonworks.com wrote: That is not possible. The number provided (# of MB) is used to ask for container from YARN and YARN only accepts numbers. % of total on each node also becomes an issue when nodes can have varying value of total memory available. This is common for heterogeneous clusters where hosts may have different value for available memory and core, etc. On Mon, Dec 1, 2014 at 1:39 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Is it able to specify the percentage of memory instead of # of MB
I created SLIDER-691 for this feature. Cheers On Tue, Dec 2, 2014 at 7:09 AM, Steve Loughran ste...@hortonworks.com wrote: I don't even think we can ask yarn for the percentage of memory I do understand what you are trying to do: ask for 51%. We could do something where we can query YARN for the maximum amount an app can ask for (the AM finds this out when registered), and the capacity could be derived as a percentage of that. Rather than percentage of available RAM, it's just a fraction of what you are allowed. On 2 December 2014 at 01:49, Sumit Mohanty smoha...@hortonworks.com wrote: That is not possible. The number provided (# of MB) is used to ask for container from YARN and YARN only accepts numbers. % of total on each node also becomes an issue when nodes can have varying value of total memory available. This is common for heterogeneous clusters where hosts may have different value for available memory and core, etc. On Mon, Dec 1, 2014 at 1:39 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Is it able to specify the percentage of memory instead of # of MB
That is not possible. The number provided (# of MB) is used to ask for container from YARN and YARN only accepts numbers. % of total on each node also becomes an issue when nodes can have varying value of total memory available. This is common for heterogeneous clusters where hosts may have different value for available memory and core, etc. On Mon, Dec 1, 2014 at 1:39 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi guys, For each component instance, Is there a way to specify the % of total memory of each node instead of # of MB? Best, Siyuan -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
