How Sliders configure configuration file of regionserver of HBase
Hi Team, The template of regionserver of HBase application uses following scripts to configure configuraiton file of regionserver of HBase. I did not find where the rs_hosts is defined in any file in Slider HBase package. How a template is designed is mentioned a little at the end of http://slider.incubator.apache.org/docs/slider_specs/application_package.html, but cannot help much. {% for host in rs_hosts %}{{host}} {% endfor %} Does anyone know where rs_hosts is defined? I am now writing a package for a HPC applicaiton and need to know host list of all the instances of a component to prepare configuration file of the application. Thanks, Yong
Re: How Sliders configure configuration file of regionserver of HBase
bq. host list of all the instances of a component Are you using node labels feature of YARN ? If not, the list would comprise of all the nodes in your YARN cluster. Cheers On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng wrote: > Hi Team, > > The template of regionserver of HBase application uses following scripts to > configure configuraiton file of regionserver of HBase. I did not find where > the rs_hosts is defined in any file in Slider HBase package. How a template > is designed is mentioned a little at the end of > > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > , > but cannot help much. > > {% for host in rs_hosts %}{{host}} > {% endfor %} > > Does anyone know where rs_hosts is defined? I am now writing a package for > a HPC applicaiton and need to know host list of all the instances of a > component to prepare configuration file of the application. > > Thanks, > > Yong >
Re: How Sliders configure configuration file of regionserver of HBase
ah, rs_hosts means all the nodes managed by YARN. I know a little about labels feature of YARN which adds a tag to host and allow yarn API client to filter hosts according to the tag. I don't think it helps my case. Taking a typical master/slave cluster as an example. To support active-passive fail over, we need define multiple instances for master role (unfortunately it does not use Zookeeper as the cluster is implemented 10+ years ago before Zookeeper). In slave node, a list of mast host are specified in conf file, so that slave node could contact master. To deploy such cluster, we need know host list of master and configure them in the conf file of slave. Changing membership of master need restart whole cluster so we will not support flex up/down role of master. How can I know the host list of master role in Slider script or template? Thanks, Yong On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu wrote: > bq. host list of all the instances of a component > > Are you using node labels feature of YARN ? > If not, the list would comprise of all the nodes in your YARN cluster. > > Cheers > > On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng wrote: > > > Hi Team, > > > > The template of regionserver of HBase application uses following scripts > to > > configure configuraiton file of regionserver of HBase. I did not find > where > > the rs_hosts is defined in any file in Slider HBase package. How a > template > > is designed is mentioned a little at the end of > > > > > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > > , > > but cannot help much. > > > > {% for host in rs_hosts %}{{host}} > > {% endfor %} > > > > Does anyone know where rs_hosts is defined? I am now writing a package > for > > a HPC applicaiton and need to know host list of all the instances of a > > component to prepare configuration file of the application. > > > > Thanks, > > > > Yong > > >
Re: How Sliders configure configuration file of regionserver of HBase
What's the timeframe for this migration ? If time permits, you should consider rewriting master failover logic in your application. Using zookeeper is a common technique. HBase, e.g., registers active master under a well known zookeeper znode. This is much more flexible than the way you described. Again, knowing the host list of master role in Slider script or template is not supported by either Slider or YARN. Cheers On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng wrote: > ah, rs_hosts means all the nodes managed by YARN. > > I know a little about labels feature of YARN which adds a tag to host and > allow yarn API client to filter hosts according to the tag. I don't think > it helps my case. > > Taking a typical master/slave cluster as an example. To support > active-passive fail over, we need define multiple instances for master role > (unfortunately it does not use Zookeeper as the cluster is implemented 10+ > years ago before Zookeeper). In slave node, a list of mast host are > specified in conf file, so that slave node could contact master. > > To deploy such cluster, we need know host list of master and configure them > in the conf file of slave. Changing membership of master need restart whole > cluster so we will not support flex up/down role of master. > > How can I know the host list of master role in Slider script or template? > > Thanks, > > Yong > > On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu wrote: > > > bq. host list of all the instances of a component > > > > Are you using node labels feature of YARN ? > > If not, the list would comprise of all the nodes in your YARN cluster. > > > > Cheers > > > > On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng > wrote: > > > > > Hi Team, > > > > > > The template of regionserver of HBase application uses following > scripts > > to > > > configure configuraiton file of regionserver of HBase. I did not find > > where > > > the rs_hosts is defined in any file in Slider HBase package. How a > > template > > > is designed is mentioned a little at the end of > > > > > > > > > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > > > , > > > but cannot help much. > > > > > > {% for host in rs_hosts %}{{host}} > > > {% endfor %} > > > > > > Does anyone know where rs_hosts is defined? I am now writing a package > > for > > > a HPC applicaiton and need to know host list of all the instances of a > > > component to prepare configuration file of the application. > > > > > > Thanks, > > > > > > Yong > > > > > >
Re: How Sliders configure configuration file of regionserver of HBase
Thanks Ted. >> knowing the host list of master role in Slider script or template is not supported by either Slider or YARN. [Yong] If there is only one instance of master role, at least I could know its host name by COMPONENT_HOST in slave hosts, right? If so, I could work it around to just configure one master. Yong On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu wrote: > What's the timeframe for this migration ? > > If time permits, you should consider rewriting master failover logic in > your application. > Using zookeeper is a common technique. HBase, e.g., registers active master > under a well known zookeeper znode. > > This is much more flexible than the way you described. > > Again, knowing the host list of master role in Slider script or template is > not supported by either Slider or YARN. > > Cheers > > On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng wrote: > > > ah, rs_hosts means all the nodes managed by YARN. > > > > I know a little about labels feature of YARN which adds a tag to host and > > allow yarn API client to filter hosts according to the tag. I don't think > > it helps my case. > > > > Taking a typical master/slave cluster as an example. To support > > active-passive fail over, we need define multiple instances for master > role > > (unfortunately it does not use Zookeeper as the cluster is implemented > 10+ > > years ago before Zookeeper). In slave node, a list of mast host are > > specified in conf file, so that slave node could contact master. > > > > To deploy such cluster, we need know host list of master and configure > them > > in the conf file of slave. Changing membership of master need restart > whole > > cluster so we will not support flex up/down role of master. > > > > How can I know the host list of master role in Slider script or template? > > > > Thanks, > > > > Yong > > > > On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu wrote: > > > > > bq. host list of all the instances of a component > > > > > > Are you using node labels feature of YARN ? > > > If not, the list would comprise of all the nodes in your YARN cluster. > > > > > > Cheers > > > > > > On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng > > wrote: > > > > > > > Hi Team, > > > > > > > > The template of regionserver of HBase application uses following > > scripts > > > to > > > > configure configuraiton file of regionserver of HBase. I did not find > > > where > > > > the rs_hosts is defined in any file in Slider HBase package. How a > > > template > > > > is designed is mentioned a little at the end of > > > > > > > > > > > > > > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > > > > , > > > > but cannot help much. > > > > > > > > {% for host in rs_hosts %}{{host}} > > > > {% endfor %} > > > > > > > > Does anyone know where rs_hosts is defined? I am now writing a > package > > > for > > > > a HPC applicaiton and need to know host list of all the instances of > a > > > > component to prepare configuration file of the application. > > > > > > > > Thanks, > > > > > > > > Yong > > > > > > > > > >
Re: How Sliders configure configuration file of regionserver of HBase
I’m confused - if certain hosts were labeled as MASTER, wouldn’t you know a priori the list of master hosts? On Dec 31, 2014, at 12:31 PM, Yong Feng wrote: > Thanks Ted. > >>> knowing the host list of master role in Slider script or template is not > supported by either Slider or YARN. > [Yong] If there is only one instance of master role, at least I could know > its host name by COMPONENT_HOST in slave hosts, right? If so, I could work > it around to just configure one master. > > Yong > > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu wrote: > >> What's the timeframe for this migration ? >> >> If time permits, you should consider rewriting master failover logic in >> your application. >> Using zookeeper is a common technique. HBase, e.g., registers active master >> under a well known zookeeper znode. >> >> This is much more flexible than the way you described. >> >> Again, knowing the host list of master role in Slider script or template is >> not supported by either Slider or YARN. >> >> Cheers >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng wrote: >> >>> ah, rs_hosts means all the nodes managed by YARN. >>> >>> I know a little about labels feature of YARN which adds a tag to host and >>> allow yarn API client to filter hosts according to the tag. I don't think >>> it helps my case. >>> >>> Taking a typical master/slave cluster as an example. To support >>> active-passive fail over, we need define multiple instances for master >> role >>> (unfortunately it does not use Zookeeper as the cluster is implemented >> 10+ >>> years ago before Zookeeper). In slave node, a list of mast host are >>> specified in conf file, so that slave node could contact master. >>> >>> To deploy such cluster, we need know host list of master and configure >> them >>> in the conf file of slave. Changing membership of master need restart >> whole >>> cluster so we will not support flex up/down role of master. >>> >>> How can I know the host list of master role in Slider script or template? >>> >>> Thanks, >>> >>> Yong >>> >>> On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu wrote: >>> bq. host list of all the instances of a component Are you using node labels feature of YARN ? If not, the list would comprise of all the nodes in your YARN cluster. Cheers On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng >>> wrote: > Hi Team, > > The template of regionserver of HBase application uses following >>> scripts to > configure configuraiton file of regionserver of HBase. I did not find where > the rs_hosts is defined in any file in Slider HBase package. How a template > is designed is mentioned a little at the end of > > >>> >> http://slider.incubator.apache.org/docs/slider_specs/application_package.html > , > but cannot help much. > > {% for host in rs_hosts %}{{host}} > {% endfor %} > > Does anyone know where rs_hosts is defined? I am now writing a >> package for > a HPC applicaiton and need to know host list of all the instances of >> a > component to prepare configuration file of the application. > > Thanks, > > Yong > >>> >> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: How Sliders configure configuration file of regionserver of HBase
Hi John I do expect the hosts allocated to master role of my slider application are labeled as appname_rolename, but the question is who will add such label and how? and how to get the hosts with the label? Thanks, Yong On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron wrote: > I’m confused - if certain hosts were labeled as MASTER, wouldn’t you know > a priori the list of master hosts? > > > On Dec 31, 2014, at 12:31 PM, Yong Feng wrote: > > > Thanks Ted. > > > >>> knowing the host list of master role in Slider script or template is > not > > supported by either Slider or YARN. > > [Yong] If there is only one instance of master role, at least I could > know > > its host name by COMPONENT_HOST in slave hosts, right? If so, I could > work > > it around to just configure one master. > > > > Yong > > > > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu wrote: > > > >> What's the timeframe for this migration ? > >> > >> If time permits, you should consider rewriting master failover logic in > >> your application. > >> Using zookeeper is a common technique. HBase, e.g., registers active > master > >> under a well known zookeeper znode. > >> > >> This is much more flexible than the way you described. > >> > >> Again, knowing the host list of master role in Slider script or > template is > >> not supported by either Slider or YARN. > >> > >> Cheers > >> > >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng > wrote: > >> > >>> ah, rs_hosts means all the nodes managed by YARN. > >>> > >>> I know a little about labels feature of YARN which adds a tag to host > and > >>> allow yarn API client to filter hosts according to the tag. I don't > think > >>> it helps my case. > >>> > >>> Taking a typical master/slave cluster as an example. To support > >>> active-passive fail over, we need define multiple instances for master > >> role > >>> (unfortunately it does not use Zookeeper as the cluster is implemented > >> 10+ > >>> years ago before Zookeeper). In slave node, a list of mast host are > >>> specified in conf file, so that slave node could contact master. > >>> > >>> To deploy such cluster, we need know host list of master and configure > >> them > >>> in the conf file of slave. Changing membership of master need restart > >> whole > >>> cluster so we will not support flex up/down role of master. > >>> > >>> How can I know the host list of master role in Slider script or > template? > >>> > >>> Thanks, > >>> > >>> Yong > >>> > >>> On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu wrote: > >>> > bq. host list of all the instances of a component > > Are you using node labels feature of YARN ? > If not, the list would comprise of all the nodes in your YARN cluster. > > Cheers > > On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng > >>> wrote: > > > Hi Team, > > > > The template of regionserver of HBase application uses following > >>> scripts > to > > configure configuraiton file of regionserver of HBase. I did not find > where > > the rs_hosts is defined in any file in Slider HBase package. How a > template > > is designed is mentioned a little at the end of > > > > > > >>> > >> > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > > , > > but cannot help much. > > > > {% for host in rs_hosts %}{{host}} > > {% endfor %} > > > > Does anyone know where rs_hosts is defined? I am now writing a > >> package > for > > a HPC applicaiton and need to know host list of all the instances of > >> a > > component to prepare configuration file of the application. > > > > Thanks, > > > > Yong > > > > >>> > >> > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
Re: How Sliders configure configuration file of regionserver of HBase
Jon, Ted Happy new year. After read more, I understand knowing the host list or single host of master role in Slider script or template is not supported by either Slider or YARN. So what I can do is a) pre-define a group of hosts with label of master b) specify label of master for master role in app resource specfication c) get host list with label of master in template/scripts when deploying slave role However I did not find doc how to get host list with label of master in Slider template/scripts like rs_hosts as follows {% for host in rs_hosts %}{{host}} {% endfor %} Any suggestion? Thanks, Yong On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng wrote: > Hi John > > I do expect the hosts allocated to master role of my slider application > are labeled as appname_rolename, but the question is who will add such > label and how? and how to get the hosts with the label? > > Thanks, > > Yong > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron wrote: > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t you know >> a priori the list of master hosts? >> >> >> On Dec 31, 2014, at 12:31 PM, Yong Feng wrote: >> >> > Thanks Ted. >> > >> >>> knowing the host list of master role in Slider script or template is >> not >> > supported by either Slider or YARN. >> > [Yong] If there is only one instance of master role, at least I could >> know >> > its host name by COMPONENT_HOST in slave hosts, right? If so, I could >> work >> > it around to just configure one master. >> > >> > Yong >> > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu wrote: >> > >> >> What's the timeframe for this migration ? >> >> >> >> If time permits, you should consider rewriting master failover logic in >> >> your application. >> >> Using zookeeper is a common technique. HBase, e.g., registers active >> master >> >> under a well known zookeeper znode. >> >> >> >> This is much more flexible than the way you described. >> >> >> >> Again, knowing the host list of master role in Slider script or >> template is >> >> not supported by either Slider or YARN. >> >> >> >> Cheers >> >> >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng >> wrote: >> >> >> >>> ah, rs_hosts means all the nodes managed by YARN. >> >>> >> >>> I know a little about labels feature of YARN which adds a tag to host >> and >> >>> allow yarn API client to filter hosts according to the tag. I don't >> think >> >>> it helps my case. >> >>> >> >>> Taking a typical master/slave cluster as an example. To support >> >>> active-passive fail over, we need define multiple instances for master >> >> role >> >>> (unfortunately it does not use Zookeeper as the cluster is implemented >> >> 10+ >> >>> years ago before Zookeeper). In slave node, a list of mast host are >> >>> specified in conf file, so that slave node could contact master. >> >>> >> >>> To deploy such cluster, we need know host list of master and configure >> >> them >> >>> in the conf file of slave. Changing membership of master need restart >> >> whole >> >>> cluster so we will not support flex up/down role of master. >> >>> >> >>> How can I know the host list of master role in Slider script or >> template? >> >>> >> >>> Thanks, >> >>> >> >>> Yong >> >>> >> >>> On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu wrote: >> >>> >> bq. host list of all the instances of a component >> >> Are you using node labels feature of YARN ? >> If not, the list would comprise of all the nodes in your YARN >> cluster. >> >> Cheers >> >> On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng >> >>> wrote: >> >> > Hi Team, >> > >> > The template of regionserver of HBase application uses following >> >>> scripts >> to >> > configure configuraiton file of regionserver of HBase. I did not >> find >> where >> > the rs_hosts is defined in any file in Slider HBase package. How a >> template >> > is designed is mentioned a little at the end of >> > >> > >> >> >>> >> >> >> http://slider.incubator.apache.org/docs/slider_specs/application_package.html >> > , >> > but cannot help much. >> > >> > {% for host in rs_hosts %}{{host}} >> > {% endfor %} >> > >> > Does anyone know where rs_hosts is defined? I am now writing a >> >> package >> for >> > a HPC applicaiton and need to know host list of all the instances of >> >> a >> > component to prepare configuration file of the application. >> > >> > Thanks, >> > >> > Yong >> > >> >> >>> >> >> >> >> >> -- >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to >> which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified >> that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly pro
Re: How Sliders configure configuration file of regionserver of HBase
Yong: Let me discuss with slider / YARN developers and get back to you. Happy new year. On Thu, Jan 1, 2015 at 7:13 PM, Yong Feng wrote: > Jon, Ted > > Happy new year. > > After read more, I understand knowing the host list or single host of > master role in Slider script or template is not supported by either Slider > or YARN. > > So what I can do is > > a) pre-define a group of hosts with label of master > b) specify label of master for master role in app resource specfication > c) get host list with label of master in template/scripts when deploying > slave role > > However I did not find doc how to get host list with label of master in > Slider template/scripts like rs_hosts as follows > > {% for host in rs_hosts %}{{host}} > {% endfor %} > > Any suggestion? > > Thanks, > > Yong > > > > > > > On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng wrote: > > > Hi John > > > > I do expect the hosts allocated to master role of my slider application > > are labeled as appname_rolename, but the question is who will add such > > label and how? and how to get the hosts with the label? > > > > Thanks, > > > > Yong > > > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron > wrote: > > > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t you > know > >> a priori the list of master hosts? > >> > >> > >> On Dec 31, 2014, at 12:31 PM, Yong Feng wrote: > >> > >> > Thanks Ted. > >> > > >> >>> knowing the host list of master role in Slider script or template is > >> not > >> > supported by either Slider or YARN. > >> > [Yong] If there is only one instance of master role, at least I could > >> know > >> > its host name by COMPONENT_HOST in slave hosts, right? If so, I could > >> work > >> > it around to just configure one master. > >> > > >> > Yong > >> > > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu wrote: > >> > > >> >> What's the timeframe for this migration ? > >> >> > >> >> If time permits, you should consider rewriting master failover logic > in > >> >> your application. > >> >> Using zookeeper is a common technique. HBase, e.g., registers active > >> master > >> >> under a well known zookeeper znode. > >> >> > >> >> This is much more flexible than the way you described. > >> >> > >> >> Again, knowing the host list of master role in Slider script or > >> template is > >> >> not supported by either Slider or YARN. > >> >> > >> >> Cheers > >> >> > >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng > >> wrote: > >> >> > >> >>> ah, rs_hosts means all the nodes managed by YARN. > >> >>> > >> >>> I know a little about labels feature of YARN which adds a tag to > host > >> and > >> >>> allow yarn API client to filter hosts according to the tag. I don't > >> think > >> >>> it helps my case. > >> >>> > >> >>> Taking a typical master/slave cluster as an example. To support > >> >>> active-passive fail over, we need define multiple instances for > master > >> >> role > >> >>> (unfortunately it does not use Zookeeper as the cluster is > implemented > >> >> 10+ > >> >>> years ago before Zookeeper). In slave node, a list of mast host are > >> >>> specified in conf file, so that slave node could contact master. > >> >>> > >> >>> To deploy such cluster, we need know host list of master and > configure > >> >> them > >> >>> in the conf file of slave. Changing membership of master need > restart > >> >> whole > >> >>> cluster so we will not support flex up/down role of master. > >> >>> > >> >>> How can I know the host list of master role in Slider script or > >> template? > >> >>> > >> >>> Thanks, > >> >>> > >> >>> Yong > >> >>> > >> >>> On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu > wrote: > >> >>> > >> bq. host list of all the instances of a component > >> > >> Are you using node labels feature of YARN ? > >> If not, the list would comprise of all the nodes in your YARN > >> cluster. > >> > >> Cheers > >> > >> On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng > >> >>> wrote: > >> > >> > Hi Team, > >> > > >> > The template of regionserver of HBase application uses following > >> >>> scripts > >> to > >> > configure configuraiton file of regionserver of HBase. I did not > >> find > >> where > >> > the rs_hosts is defined in any file in Slider HBase package. How a > >> template > >> > is designed is mentioned a little at the end of > >> > > >> > > >> > >> >>> > >> >> > >> > http://slider.incubator.apache.org/docs/slider_specs/application_package.html > >> > , > >> > but cannot help much. > >> > > >> > {% for host in rs_hosts %}{{host}} > >> > {% endfor %} > >> > > >> > Does anyone know where rs_hosts is defined? I am now writing a > >> >> package > >> for > >> > a HPC applicaiton and need to know host list of all the instances > of > >> >> a > >> > component to prepare configuration file of the application. > >> > > >> > Thanks, > >> > > >> > Yong > >> > > >> > >> >>> > >>
Re: How Sliders configure configuration file of regionserver of HBase
Thanks Ted. Basically what we are doing is to ask HPC workload and BigData workload share the same infrastructure by Slider. The current way to fit Hadoop into HPC community is either on demand provisioning Hadoop cluster by HPC scheduler (for example HOD or myHadoop) or enhancing Hadoop as HPC workload scheduler directly (quite a lot papers). Besides them, we actually could ask Yarn/Slider as resource manager to manage both HPC wlm and BigData wlm. Mesos/Marathon is another choice however I prefer Yarn/Slider due to its BigData DNA. Looking forward to getting more support from Slider and Yarn team. Thanks, Yong On Thu, Jan 1, 2015 at 10:31 PM, Ted Yu wrote: > Yong: > Let me discuss with slider / YARN developers and get back to you. > > Happy new year. > > On Thu, Jan 1, 2015 at 7:13 PM, Yong Feng wrote: > > > Jon, Ted > > > > Happy new year. > > > > After read more, I understand knowing the host list or single host of > > master role in Slider script or template is not supported by either > Slider > > or YARN. > > > > So what I can do is > > > > a) pre-define a group of hosts with label of master > > b) specify label of master for master role in app resource specfication > > c) get host list with label of master in template/scripts when deploying > > slave role > > > > However I did not find doc how to get host list with label of master in > > Slider template/scripts like rs_hosts as follows > > > > {% for host in rs_hosts %}{{host}} > > {% endfor %} > > > > Any suggestion? > > > > Thanks, > > > > Yong > > > > > > > > > > > > > > On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng > wrote: > > > > > Hi John > > > > > > I do expect the hosts allocated to master role of my slider application > > > are labeled as appname_rolename, but the question is who will add such > > > label and how? and how to get the hosts with the label? > > > > > > Thanks, > > > > > > Yong > > > > > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron > > wrote: > > > > > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t you > > know > > >> a priori the list of master hosts? > > >> > > >> > > >> On Dec 31, 2014, at 12:31 PM, Yong Feng > wrote: > > >> > > >> > Thanks Ted. > > >> > > > >> >>> knowing the host list of master role in Slider script or template > is > > >> not > > >> > supported by either Slider or YARN. > > >> > [Yong] If there is only one instance of master role, at least I > could > > >> know > > >> > its host name by COMPONENT_HOST in slave hosts, right? If so, I > could > > >> work > > >> > it around to just configure one master. > > >> > > > >> > Yong > > >> > > > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu > wrote: > > >> > > > >> >> What's the timeframe for this migration ? > > >> >> > > >> >> If time permits, you should consider rewriting master failover > logic > > in > > >> >> your application. > > >> >> Using zookeeper is a common technique. HBase, e.g., registers > active > > >> master > > >> >> under a well known zookeeper znode. > > >> >> > > >> >> This is much more flexible than the way you described. > > >> >> > > >> >> Again, knowing the host list of master role in Slider script or > > >> template is > > >> >> not supported by either Slider or YARN. > > >> >> > > >> >> Cheers > > >> >> > > >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng > > >> wrote: > > >> >> > > >> >>> ah, rs_hosts means all the nodes managed by YARN. > > >> >>> > > >> >>> I know a little about labels feature of YARN which adds a tag to > > host > > >> and > > >> >>> allow yarn API client to filter hosts according to the tag. I > don't > > >> think > > >> >>> it helps my case. > > >> >>> > > >> >>> Taking a typical master/slave cluster as an example. To support > > >> >>> active-passive fail over, we need define multiple instances for > > master > > >> >> role > > >> >>> (unfortunately it does not use Zookeeper as the cluster is > > implemented > > >> >> 10+ > > >> >>> years ago before Zookeeper). In slave node, a list of mast host > are > > >> >>> specified in conf file, so that slave node could contact master. > > >> >>> > > >> >>> To deploy such cluster, we need know host list of master and > > configure > > >> >> them > > >> >>> in the conf file of slave. Changing membership of master need > > restart > > >> >> whole > > >> >>> cluster so we will not support flex up/down role of master. > > >> >>> > > >> >>> How can I know the host list of master role in Slider script or > > >> template? > > >> >>> > > >> >>> Thanks, > > >> >>> > > >> >>> Yong > > >> >>> > > >> >>> On Tue, Dec 30, 2014 at 9:08 PM, Ted Yu > > wrote: > > >> >>> > > >> bq. host list of all the instances of a component > > >> > > >> Are you using node labels feature of YARN ? > > >> If not, the list would comprise of all the nodes in your YARN > > >> cluster. > > >> > > >> Cheers > > >> > > >> On Tue, Dec 30, 2014 at 4:43 PM, Yong Feng < > fengyong...@gmail.com> > > >> >>> wrote: > > >> > > >> > Hi T
Re: How Sliders configure configuration file of regionserver of HBase
Thanks for the background information. Adding Wangda who worked on node labels feature. On Fri, Jan 2, 2015 at 9:04 AM, Yong Feng wrote: > Thanks Ted. > > Basically what we are doing is to ask HPC workload and BigData workload > share the same infrastructure by Slider. The current way to fit Hadoop into > HPC community is either on demand provisioning Hadoop cluster by HPC > scheduler (for example HOD or myHadoop) or enhancing Hadoop as HPC workload > scheduler directly (quite a lot papers). > > Besides them, we actually could ask Yarn/Slider as resource manager to > manage both HPC wlm and BigData wlm. Mesos/Marathon is another choice > however I prefer Yarn/Slider due to its BigData DNA. > > Looking forward to getting more support from Slider and Yarn team. > > Thanks, > > Yong > > > On Thu, Jan 1, 2015 at 10:31 PM, Ted Yu wrote: > > > Yong: > > Let me discuss with slider / YARN developers and get back to you. > > > > Happy new year. > > > > On Thu, Jan 1, 2015 at 7:13 PM, Yong Feng wrote: > > > > > Jon, Ted > > > > > > Happy new year. > > > > > > After read more, I understand knowing the host list or single host of > > > master role in Slider script or template is not supported by either > > Slider > > > or YARN. > > > > > > So what I can do is > > > > > > a) pre-define a group of hosts with label of master > > > b) specify label of master for master role in app resource specfication > > > c) get host list with label of master in template/scripts when > deploying > > > slave role > > > > > > However I did not find doc how to get host list with label of master in > > > Slider template/scripts like rs_hosts as follows > > > > > > {% for host in rs_hosts %}{{host}} > > > {% endfor %} > > > > > > Any suggestion? > > > > > > Thanks, > > > > > > Yong > > > > > > > > > > > > > > > > > > > > > On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng > > wrote: > > > > > > > Hi John > > > > > > > > I do expect the hosts allocated to master role of my slider > application > > > > are labeled as appname_rolename, but the question is who will add > such > > > > label and how? and how to get the hosts with the label? > > > > > > > > Thanks, > > > > > > > > Yong > > > > > > > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron > > > wrote: > > > > > > > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t you > > > know > > > >> a priori the list of master hosts? > > > >> > > > >> > > > >> On Dec 31, 2014, at 12:31 PM, Yong Feng > > wrote: > > > >> > > > >> > Thanks Ted. > > > >> > > > > >> >>> knowing the host list of master role in Slider script or > template > > is > > > >> not > > > >> > supported by either Slider or YARN. > > > >> > [Yong] If there is only one instance of master role, at least I > > could > > > >> know > > > >> > its host name by COMPONENT_HOST in slave hosts, right? If so, I > > could > > > >> work > > > >> > it around to just configure one master. > > > >> > > > > >> > Yong > > > >> > > > > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu > > wrote: > > > >> > > > > >> >> What's the timeframe for this migration ? > > > >> >> > > > >> >> If time permits, you should consider rewriting master failover > > logic > > > in > > > >> >> your application. > > > >> >> Using zookeeper is a common technique. HBase, e.g., registers > > active > > > >> master > > > >> >> under a well known zookeeper znode. > > > >> >> > > > >> >> This is much more flexible than the way you described. > > > >> >> > > > >> >> Again, knowing the host list of master role in Slider script or > > > >> template is > > > >> >> not supported by either Slider or YARN. > > > >> >> > > > >> >> Cheers > > > >> >> > > > >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng < > fengyong...@gmail.com> > > > >> wrote: > > > >> >> > > > >> >>> ah, rs_hosts means all the nodes managed by YARN. > > > >> >>> > > > >> >>> I know a little about labels feature of YARN which adds a tag to > > > host > > > >> and > > > >> >>> allow yarn API client to filter hosts according to the tag. I > > don't > > > >> think > > > >> >>> it helps my case. > > > >> >>> > > > >> >>> Taking a typical master/slave cluster as an example. To support > > > >> >>> active-passive fail over, we need define multiple instances for > > > master > > > >> >> role > > > >> >>> (unfortunately it does not use Zookeeper as the cluster is > > > implemented > > > >> >> 10+ > > > >> >>> years ago before Zookeeper). In slave node, a list of mast host > > are > > > >> >>> specified in conf file, so that slave node could contact master. > > > >> >>> > > > >> >>> To deploy such cluster, we need know host list of master and > > > configure > > > >> >> them > > > >> >>> in the conf file of slave. Changing membership of master need > > > restart > > > >> >> whole > > > >> >>> cluster so we will not support flex up/down role of master. > > > >> >>> > > > >> >>> How can I know the host list of master role in Slider script or > > > >> template? > > > >> >>> > > > >> >>> Thanks, > > > >> >>> > > >
Re: How Sliders configure configuration file of regionserver of HBase
YARN-3003 was logged. But it is not known when it will be implemented. Yong: Please consider using zookeeper for recoding master nodes. This makes your solution workable on Slider 0.60 Cheers On Fri, Jan 2, 2015 at 10:33 AM, Wangda Tan wrote: > Currently, YARN doesn't have direct method to do that, but you can get a > label-to-nodes mapping using YarnClient.getNodeToLabels(). > > We can consider to add a method label-to-nodes mapping in future release > if necessary. > > Thanks, > Wangda > > On Fri, Jan 2, 2015 at 9:11 AM, Ted Yu wrote: > >> Thanks for the background information. >> >> Adding Wangda who worked on node labels feature. >> >> On Fri, Jan 2, 2015 at 9:04 AM, Yong Feng wrote: >> >>> Thanks Ted. >>> >>> Basically what we are doing is to ask HPC workload and BigData workload >>> share the same infrastructure by Slider. The current way to fit Hadoop >>> into >>> HPC community is either on demand provisioning Hadoop cluster by HPC >>> scheduler (for example HOD or myHadoop) or enhancing Hadoop as HPC >>> workload >>> scheduler directly (quite a lot papers). >>> >>> Besides them, we actually could ask Yarn/Slider as resource manager to >>> manage both HPC wlm and BigData wlm. Mesos/Marathon is another choice >>> however I prefer Yarn/Slider due to its BigData DNA. >>> >>> Looking forward to getting more support from Slider and Yarn team. >>> >>> Thanks, >>> >>> Yong >>> >>> >>> On Thu, Jan 1, 2015 at 10:31 PM, Ted Yu wrote: >>> >>> > Yong: >>> > Let me discuss with slider / YARN developers and get back to you. >>> > >>> > Happy new year. >>> > >>> > On Thu, Jan 1, 2015 at 7:13 PM, Yong Feng >>> wrote: >>> > >>> > > Jon, Ted >>> > > >>> > > Happy new year. >>> > > >>> > > After read more, I understand knowing the host list or single host of >>> > > master role in Slider script or template is not supported by either >>> > Slider >>> > > or YARN. >>> > > >>> > > So what I can do is >>> > > >>> > > a) pre-define a group of hosts with label of master >>> > > b) specify label of master for master role in app resource >>> specfication >>> > > c) get host list with label of master in template/scripts when >>> deploying >>> > > slave role >>> > > >>> > > However I did not find doc how to get host list with label of master >>> in >>> > > Slider template/scripts like rs_hosts as follows >>> > > >>> > > {% for host in rs_hosts %}{{host}} >>> > > {% endfor %} >>> > > >>> > > Any suggestion? >>> > > >>> > > Thanks, >>> > > >>> > > Yong >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng >>> > wrote: >>> > > >>> > > > Hi John >>> > > > >>> > > > I do expect the hosts allocated to master role of my slider >>> application >>> > > > are labeled as appname_rolename, but the question is who will add >>> such >>> > > > label and how? and how to get the hosts with the label? >>> > > > >>> > > > Thanks, >>> > > > >>> > > > Yong >>> > > > >>> > > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron >> > >>> > > wrote: >>> > > > >>> > > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t >>> you >>> > > know >>> > > >> a priori the list of master hosts? >>> > > >> >>> > > >> >>> > > >> On Dec 31, 2014, at 12:31 PM, Yong Feng >>> > wrote: >>> > > >> >>> > > >> > Thanks Ted. >>> > > >> > >>> > > >> >>> knowing the host list of master role in Slider script or >>> template >>> > is >>> > > >> not >>> > > >> > supported by either Slider or YARN. >>> > > >> > [Yong] If there is only one instance of master role, at least I >>> > could >>> > > >> know >>> > > >> > its host name by COMPONENT_HOST in slave hosts, right? If so, I >>> > could >>> > > >> work >>> > > >> > it around to just configure one master. >>> > > >> > >>> > > >> > Yong >>> > > >> > >>> > > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu >>> > wrote: >>> > > >> > >>> > > >> >> What's the timeframe for this migration ? >>> > > >> >> >>> > > >> >> If time permits, you should consider rewriting master failover >>> > logic >>> > > in >>> > > >> >> your application. >>> > > >> >> Using zookeeper is a common technique. HBase, e.g., registers >>> > active >>> > > >> master >>> > > >> >> under a well known zookeeper znode. >>> > > >> >> >>> > > >> >> This is much more flexible than the way you described. >>> > > >> >> >>> > > >> >> Again, knowing the host list of master role in Slider script or >>> > > >> template is >>> > > >> >> not supported by either Slider or YARN. >>> > > >> >> >>> > > >> >> Cheers >>> > > >> >> >>> > > >> >> On Tue, Dec 30, 2014 at 7:50 PM, Yong Feng < >>> fengyong...@gmail.com> >>> > > >> wrote: >>> > > >> >> >>> > > >> >>> ah, rs_hosts means all the nodes managed by YARN. >>> > > >> >>> >>> > > >> >>> I know a little about labels feature of YARN which adds a tag >>> to >>> > > host >>> > > >> and >>> > > >> >>> allow yarn API client to filter hosts according to the tag. I >>> > don't >>> > > >> think >>> > > >> >>> it helps my case. >>> > > >> >>> >>> > > >> >>> Taking a typical mast
Re: How Sliders configure configuration file of regionserver of HBase
Thanks Ted. I will check if I could get some resources to contribute. Yong On Sat, Jan 3, 2015 at 2:13 PM, Ted Yu wrote: > YARN-3003 was logged. > But it is not known when it will be implemented. > > Yong: > Please consider using zookeeper for recoding master nodes. This makes your > solution workable on Slider 0.60 > > Cheers > > On Fri, Jan 2, 2015 at 10:33 AM, Wangda Tan wrote: > > > Currently, YARN doesn't have direct method to do that, but you can get a > > label-to-nodes mapping using YarnClient.getNodeToLabels(). > > > > We can consider to add a method label-to-nodes mapping in future release > > if necessary. > > > > Thanks, > > Wangda > > > > On Fri, Jan 2, 2015 at 9:11 AM, Ted Yu wrote: > > > >> Thanks for the background information. > >> > >> Adding Wangda who worked on node labels feature. > >> > >> On Fri, Jan 2, 2015 at 9:04 AM, Yong Feng > wrote: > >> > >>> Thanks Ted. > >>> > >>> Basically what we are doing is to ask HPC workload and BigData workload > >>> share the same infrastructure by Slider. The current way to fit Hadoop > >>> into > >>> HPC community is either on demand provisioning Hadoop cluster by HPC > >>> scheduler (for example HOD or myHadoop) or enhancing Hadoop as HPC > >>> workload > >>> scheduler directly (quite a lot papers). > >>> > >>> Besides them, we actually could ask Yarn/Slider as resource manager to > >>> manage both HPC wlm and BigData wlm. Mesos/Marathon is another choice > >>> however I prefer Yarn/Slider due to its BigData DNA. > >>> > >>> Looking forward to getting more support from Slider and Yarn team. > >>> > >>> Thanks, > >>> > >>> Yong > >>> > >>> > >>> On Thu, Jan 1, 2015 at 10:31 PM, Ted Yu wrote: > >>> > >>> > Yong: > >>> > Let me discuss with slider / YARN developers and get back to you. > >>> > > >>> > Happy new year. > >>> > > >>> > On Thu, Jan 1, 2015 at 7:13 PM, Yong Feng > >>> wrote: > >>> > > >>> > > Jon, Ted > >>> > > > >>> > > Happy new year. > >>> > > > >>> > > After read more, I understand knowing the host list or single host > of > >>> > > master role in Slider script or template is not supported by either > >>> > Slider > >>> > > or YARN. > >>> > > > >>> > > So what I can do is > >>> > > > >>> > > a) pre-define a group of hosts with label of master > >>> > > b) specify label of master for master role in app resource > >>> specfication > >>> > > c) get host list with label of master in template/scripts when > >>> deploying > >>> > > slave role > >>> > > > >>> > > However I did not find doc how to get host list with label of > master > >>> in > >>> > > Slider template/scripts like rs_hosts as follows > >>> > > > >>> > > {% for host in rs_hosts %}{{host}} > >>> > > {% endfor %} > >>> > > > >>> > > Any suggestion? > >>> > > > >>> > > Thanks, > >>> > > > >>> > > Yong > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > On Wed, Dec 31, 2014 at 5:18 PM, Yong Feng > >>> > wrote: > >>> > > > >>> > > > Hi John > >>> > > > > >>> > > > I do expect the hosts allocated to master role of my slider > >>> application > >>> > > > are labeled as appname_rolename, but the question is who will add > >>> such > >>> > > > label and how? and how to get the hosts with the label? > >>> > > > > >>> > > > Thanks, > >>> > > > > >>> > > > Yong > >>> > > > > >>> > > > On Wed, Dec 31, 2014 at 3:57 PM, Jon Maron < > jma...@hortonworks.com > >>> > > >>> > > wrote: > >>> > > > > >>> > > >> I’m confused - if certain hosts were labeled as MASTER, wouldn’t > >>> you > >>> > > know > >>> > > >> a priori the list of master hosts? > >>> > > >> > >>> > > >> > >>> > > >> On Dec 31, 2014, at 12:31 PM, Yong Feng > >>> > wrote: > >>> > > >> > >>> > > >> > Thanks Ted. > >>> > > >> > > >>> > > >> >>> knowing the host list of master role in Slider script or > >>> template > >>> > is > >>> > > >> not > >>> > > >> > supported by either Slider or YARN. > >>> > > >> > [Yong] If there is only one instance of master role, at least > I > >>> > could > >>> > > >> know > >>> > > >> > its host name by COMPONENT_HOST in slave hosts, right? If so, > I > >>> > could > >>> > > >> work > >>> > > >> > it around to just configure one master. > >>> > > >> > > >>> > > >> > Yong > >>> > > >> > > >>> > > >> > On Tue, Dec 30, 2014 at 10:59 PM, Ted Yu > > >>> > wrote: > >>> > > >> > > >>> > > >> >> What's the timeframe for this migration ? > >>> > > >> >> > >>> > > >> >> If time permits, you should consider rewriting master > failover > >>> > logic > >>> > > in > >>> > > >> >> your application. > >>> > > >> >> Using zookeeper is a common technique. HBase, e.g., registers > >>> > active > >>> > > >> master > >>> > > >> >> under a well known zookeeper znode. > >>> > > >> >> > >>> > > >> >> This is much more flexible than the way you described. > >>> > > >> >> > >>> > > >> >> Again, knowing the host list of master role in Slider script > or > >>> > > >> template is > >>> > > >> >> not supported by either Slider or YARN. > >>> > > >> >> > >>> > > >> >> Cheers > >>> > > >> >> > >>> > >