Current EDP config-hints are not only plugin specific. Several types of jobs must have certain key/values and without it job will fail. For instance, MapReduce (former Jar) job type requires Mapper/Reducer classes parameters to be set[1]. Moreover, for such kind of jobs we already have separated configuration defaults [2]. Also initial versions of patch implementing config-hints contained plugin-independent defaults for all each job types [3]. I remember we postponed decision about which configs are commmon for all plugins and agreed to show users all vanilla-specific defaults. That's why now we have several TODOs in the code about config-hints should be plugin-specific.
So I propose to leave config-hints REST call in EDP internal and make it plugin-independent (or job-specific) by removing of parsing all vanilla-specific defaults and define small list of configs which is definitely common for each type of jobs. The first things come to mind: - For MapReduce jobs it's already defined in [1] - Configs like number of map and reduce tasks are common for all type of jobs - At least user always has an ability to set any key/value(s) as params/arguments for job [1] http://docs.openstack.org/developer/savanna/userdoc/edp.html#workflow [2] https://github.com/openstack/savanna/blob/master/savanna/service/edp/resources/mapred-job-config.xml [3] https://review.openstack.org/#/c/45419/10 Regards, Alexander Ignatov On 20 Jan 2014, at 22:04, Matthew Farrellee <m...@redhat.com> wrote: > On 01/20/2014 12:50 PM, Andrey Lazarev wrote: >> Inlined. >> >> >> On Mon, Jan 20, 2014 at 8:15 AM, Matthew Farrellee <m...@redhat.com >> <mailto:m...@redhat.com>> wrote: >> >> (inline, trying to make this readable by a text-only mail client >> that doesn't use tabs to indicate quoting) >> >> On 01/20/2014 02:50 AM, Andrey Lazarev wrote: >> >> ------ >> FIX - @rest.get('/jobs/config-hints/____<job_type>') - >> should move to >> GET /plugins/<plugin_name>/<____plugin_version>, similar to >> get_node_processes >> and get_required_image_tags >> ------ >> Not sure if it should be plugin specific right now. EDP >> uses it >> to show some >> configs to users in the dashboard. it's just a cosmetic >> thing. >> Also when user >> starts define some configs for some job he might not define >> cluster yet and >> thus plugin to run this job. I think we should leave it >> as is >> and leave only >> abstract configs like Mapper/Reducer class and allow >> users to >> apply any >> key/value configs if needed. >> >> >> FYI, the code contains comments suggesting it should be >> plugin specific. >> >> >> https://github.com/openstack/____savanna/blob/master/savanna/____service/edp/workflow_creator/____workflow_factory.py#L179 >> >> <https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179> >> >> >> <https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179 >> >> <https://github.com/openstack/savanna/blob/master/savanna/service/edp/workflow_creator/workflow_factory.py#L179>> >> >> IMHO, the EDP should have no plugin specific dependencies. >> >> If it currently does, we should look into why and see if we >> can't >> eliminate this entirely. >> >> [AL] EDP uses plugins in two ways: >> 1. for HDFS user >> 2. for config hints >> I think both items should not be plugin specific on EDP API >> level. But >> implementation should go to plugin and call plugin API for result. >> >> >> In fact they are both plugin specific. The user is forced to click >> through a plugin selection (when launching a job on transient >> cluster) or the plugin selection has already occurred (when >> launching a job on an existing cluster). >> >> Since the config is something that is plugin specific, you might not >> have hbase hints from vanilla but you would from hdp, and you >> already have plugin information whenever you ask for a hint, my view >> that this be under the /plugins namespace is growing stronger. >> >> >> [AL] Disagree. They are plugin specific, but EDP itself could have >> additional plugin-independent logic inside. Now config hints return EDP >> properties (like mapred.input.dir) as well as plugin-specific >> properties. Placing it under /plugins namespace will give a vision that >> it is fully plugin specific. >> >> I like to see EDP API fully plugin independent and in one workspace. If >> core side needs some information internally it can easily go into the >> plugin. > > I'm not sure if we're disagreeing. We may, in fact, be in violent agreement. > > The EDP API is fully plugin independent, and should stay that way as a > project goal. config-hints is extra data that the horizon app can use to help > give users suggestions about what config they may want to optionally add to > their job. Those config options are independent of the job and specific to > the cluster where the job will run, which is the purview of the plugin. > > Moving config-hints out of the EDP API will make this even more clear. > > Best, > > > matt > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
