Additionally, I think that we should explicitly specify the need to ensure that all outputs doesn't contain any sensitive information like credentials.
On Tue, Jan 28, 2014 at 4:06 PM, Alexander Ignatov <aigna...@mirantis.com>wrote: > "EDP internal" I meant current EDP specific code. And since job configs > are job-specific > I'd prefer to have urls containing /jobs or at least /edp for that. > > Regards, > Alexander Ignatov > > > > On 24 Jan 2014, at 23:20, Matthew Farrellee <m...@redhat.com> wrote: > > > what do you consider "EDP internal", and how does it relate to the v1.1 > or v2 API? > > > > i'm ok with making it plugin independent. i'd just suggest moving it out > of /jobs and to something like /extra/config-hints/{type}, maybe along with > /extra/validations/config. > > > > best, > > > > > > matt > > > > On 01/22/2014 06:25 AM, Alexander Ignatov wrote: > >> Current EDP config-hints are not only plugin specific. Several types of > jobs > >> must have certain key/values and without it job will fail. For instance, > >> MapReduce (former Jar) job type requires Mapper/Reducer classes > parameters > >> to be set[1]. Moreover, for such kind of jobs we already have separated > >> configuration defaults [2]. Also initial versions of patch implementing > >> config-hints contained plugin-independent defaults for all each job > types [3]. > >> I remember we postponed decision about which configs are commmon for all > >> plugins and agreed to show users all vanilla-specific defaults. That's > why now > >> we have several TODOs in the code about config-hints should be > plugin-specific. > >> > >> So I propose to leave config-hints REST call in EDP internal and make it > >> plugin-independent (or job-specific) by removing of parsing all > vanilla-specific > >> defaults and define small list of configs which is definitely common > for each type of jobs. > >> The first things come to mind: > >> - For MapReduce jobs it's already defined in [1] > >> - Configs like number of map and reduce tasks are common for all type > of jobs > >> - At least user always has an ability to set any key/value(s) as > params/arguments for job > >> > >> > >> [1] > http://docs.openstack.org/developer/savanna/userdoc/edp.html#workflow > >> [2] > https://github.com/openstack/savanna/blob/master/savanna/service/edp/resources/mapred-job-config.xml > >> [3] https://review.openstack.org/#/c/45419/10 > >> > >> Regards, > >> Alexander Ignatov > >> > >> > >> > >> On 20 Jan 2014, at 22:04, Matthew Farrellee <m...@redhat.com> wrote: > >> > >>> On 01/20/2014 12:50 PM, Andrey Lazarev wrote: > >>>> Inlined. > >>>> > >>>> > >>>> On Mon, Jan 20, 2014 at 8:15 AM, Matthew Farrellee <m...@redhat.com > >>>> <mailto:m...@redhat.com>> wrote: > >>>> > >>>> (inline, trying to make this readable by a text-only mail client > >>>> that doesn't use tabs to indicate quoting) > >>>> > >>>> On 01/20/2014 02:50 AM, Andrey Lazarev wrote: > >>>> > >>>> ------ > >>>> FIX - @rest.get('/jobs/config-hints/____<job_type>') - > >>>> should move to > >>>> GET /plugins/<plugin_name>/<____plugin_version>, > similar to > >>>> get_node_processes > >>>> and get_required_image_tags > >>>> ------ > >>>> Not sure if it should be plugin specific right now. > EDP > >>>> uses it > >>>> to show some > >>>> configs to users in the dashboard. it's just a > cosmetic > >>>> thing. > >>>> Also when user > >>>> starts define some configs for some job he might not > define > >>>> cluster yet and > >>>> thus plugin to run this job. I think we should leave > it > >>>> as is > >>>> and leave only > >>>> abstract configs like Mapper/Reducer class and allow > >>>> users to > >>>> apply any > >>>> key/value configs if needed. > >>>> > >>>> > >>>> FYI, the code contains comments suggesting it should be > >>>> plugin specific. > >>>> > >>>> > https://github.com/openstack/____savanna/blob/master/savanna/____service/edp/workflow_creator/____workflow_factory.py#L179 > >>>> < > https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179 > > > >>>> > >>>> < > https://github.com/openstack/__savanna/blob/master/savanna/__service/edp/workflow_creator/__workflow_factory.py#L179 > >>>> < > https://github.com/openstack/savanna/blob/master/savanna/service/edp/workflow_creator/workflow_factory.py#L179 > >> > >>>> > >>>> IMHO, the EDP should have no plugin specific dependencies. > >>>> > >>>> If it currently does, we should look into why and see if > we > >>>> can't > >>>> eliminate this entirely. > >>>> > >>>> [AL] EDP uses plugins in two ways: > >>>> 1. for HDFS user > >>>> 2. for config hints > >>>> I think both items should not be plugin specific on EDP API > >>>> level. But > >>>> implementation should go to plugin and call plugin API for > result. > >>>> > >>>> > >>>> In fact they are both plugin specific. The user is forced to click > >>>> through a plugin selection (when launching a job on transient > >>>> cluster) or the plugin selection has already occurred (when > >>>> launching a job on an existing cluster). > >>>> > >>>> Since the config is something that is plugin specific, you might > not > >>>> have hbase hints from vanilla but you would from hdp, and you > >>>> already have plugin information whenever you ask for a hint, my > view > >>>> that this be under the /plugins namespace is growing stronger. > >>>> > >>>> > >>>> [AL] Disagree. They are plugin specific, but EDP itself could have > >>>> additional plugin-independent logic inside. Now config hints return > EDP > >>>> properties (like mapred.input.dir) as well as plugin-specific > >>>> properties. Placing it under /plugins namespace will give a vision > that > >>>> it is fully plugin specific. > >>>> > >>>> I like to see EDP API fully plugin independent and in one workspace. > If > >>>> core side needs some information internally it can easily go into the > >>>> plugin. > >>> > >>> I'm not sure if we're disagreeing. We may, in fact, be in violent > agreement. > >>> > >>> The EDP API is fully plugin independent, and should stay that way as a > project goal. config-hints is extra data that the horizon app can use to > help give users suggestions about what config they may want to optionally > add to their job. Those config options are independent of the job and > specific to the cluster where the job will run, which is the purview of the > plugin. > >>> > >>> Moving config-hints out of the EDP API will make this even more clear. > >>> > >>> Best, > >>> > >>> > >>> matt > >>> > >>> _______________________________________________ > >>> OpenStack-dev mailing list > >>> OpenStack-dev@lists.openstack.org > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Sincerely yours, Sergey Lukjanov Savanna Technical Lead Mirantis Inc.
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev