from:"david.blaisonneau"

Re: [opnfv-tech-discuss] [infra][pharos]Is PDF feature supported by installer project now?

2017-09-19 Thread david.blaisonneau


Hi,

Thanks Alexandru, you clearly resume the discussion.

I wrote that IDF for baremetal xci with the need to bring more 
flexibility to PDF (and then SDF) that are quite rigid by design (but 
must stay as it, as they are reference for every one). It seems that we 
are several to have the same need, so if we came to a consensus on that 
extra file, i would propose several rules:


1. this file is an optional extension of the PDF, linked to a pod, and
   contains only informations required by installers for a specific pod
2. No overlap with PDF and SDF
3. this file is managed by installers, to avoid pod relative
   configurations in the installer itself.
4. Use a generic section if several installers agree on the same data
   (that may be moved to PDF or SDF if a consensus is reached)
5. the smaller, the better

In my opinion, this file would bring not only flexibility for 
installers, but could also be used as an incubator for new needs to be 
pushed in PDF and SDF.


I will split the associated patch [3] to only talk about the IDF. Please 
feel free to review it.


BR,

David


[3] https://gerrit.opnfv.org/gerrit/#/c/42233/

--
David BLAISONNEAU
NFV platforms DevOPS - OPNFV contributor
Orange - IMT / OLN / CNC / NCA / SINA

tel: +33 (0)2 96 07 27 93
email: david.blaisonn...@orange.com
2, avenue Pierre Marzin 22307 LANNION Cedex - France

Le 19/09/2017 à 10:21, Ulrich Kleber a écrit :


Hi,

Thanks, Alexandru, I think you solved my action from yesterday’s Infra 
call :-).


I think the idf as you proposed it is a good thing.

Just make sure, we don’t need to duplicate information too often (e.g. 
where there is identical information for all pods in a lab).


The sdf template exists since a few months, but I didn’t get enough 
feedback from the installer teams.


We have to make sure that not only the sdf can be easily consumed by 
the installers, but also provides all data the installers need in an 
independent way


and there is no gap then between the pdf&idf and the sdf.

The sdf template is available since June. Please have a look and let 
me know if there are things missing.


At that time I couldn’t check against MCP, only against Fuel 
configuration files.


(see 
https://git.opnfv.org/octopus/tree/scenarios/templates/sdf-template.yaml )


Please other installer teams, also let me know about missing or wrong 
things.


Also feel free to propose patches ;-).

The scenario lifecycle document is also available in octopus repo (or 
http://artifacts.opnfv.org/octopus/docs/scenario-lifecycle/index.html 
) and


awaiting more feedback.

Cheers,

Uli

*From:*opnfv-tech-discuss-boun...@lists.opnfv.org 
[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] *On Behalf Of 
*Alexandru Avadanii

*Sent:* Monday, 18 September, 2017 19:00
*To:* Julien; hu.zhiji...@zte.com.cn; Chigang (Justin); 
bob.monk...@arm.com; mpolenc...@mirantis.com; tro...@redhat.com; 
narinder.gu...@canonical.com; jack.mor...@intel.com; 
agard...@linuxfoundation.org; fatih.degirme...@ericsson.com; 
opnfv-tech-discuss@lists.opnfv.org

*Cc:* Charalampos Kominos; Guillermo Herrero
*Subject:* Re: [opnfv-tech-discuss] [infra][pharos]Is PDF feature 
supported by installer project now?


Hi,

In today’s Infra meeting, we had very little time to discuss the 
specifics of using PDF in real-life use-cases, so I am continuing the 
discussion here.


Afaik, there are at least 2 installer projects actively working on 
using PDF:


1.Fuel/Armband based on MCP – Guillermo H. and I have submitted some 
patches covering PDFs for our PODs, as well as minor improvements to 
PDF parsing, and an installer adapter (draft);


2.Openstack Ansible (OSA) – David B. has submitted a few patches, 
touching the PDF design and possible extensions;


Both installers work well with PDF as an input, but we hit a few 
design quirks that I want to elaborate on:


1.PDF tooling and installer adapters should be publicly available, 
which we decided today to move to pharos git repo (see [1]), 
eventually split that repo into smaller repos;


2.PDF alone is not enough to describe installer-specific configuration 
that is scenario-indepedent, e.g.:


a.Both David and I noticed there is no sane way to map POD networks to 
bridge names on the jump host;


b.David proposed a new section should be added to describe this 
mapping; I found a way around it for Fuel, but it requires each bridge 
to have an IP addr in each subnet it covers [2];


c.For OSA, there are other things that can’t be described by PDF, like 
possible node roles (controller, compute, network etc.) which Fuel 
solves by assuming all nodes are the same, and the first 3 nodes in 
PDF should be controllers – this works, but it’s not the best approach 
for PODs with different blades;


Since SDF is not yet available, we want to propose a mechanism that 
allows describing this installer-specific configuration.


David proposed in [3] a set of changes on top of PDF: removing some 
useless entries from PDF, as well as addin

Re: [opnfv-tech-discuss] PDF feedback and questions from our experience with Fuel

2017-09-25 Thread david.blaisonneau


Hi Alexandru,

thanks for this big set of patches and a sum up of our last pdf discussions.

The part about net_config is quite confuse, specially because you sum up 
many problems already solved by net_config, but talking about PDF not 
having it, then talking about what add net_config... but it is a hard 
job to summerize those evolution and point that we would better 
understand each one by using PDF version id, and work on a reference model.


Generally it will be +1 :)

BR

David

--
David BLAISONNEAU
NFV platforms DevOPS - OPNFV contributor
Orange - IMT / OLN / CNC / NCA / SINA

tel: +33 (0)2 96 07 27 93
email:david.blaisonn...@orange.com
2, avenue Pierre Marzin 22307 LANNION Cedex - France

Le 25/09/2017 à 00:21, Alexandru Avadanii a écrit :

Hi,

A. Current Fuel PDF integration status

Fuel and Armband teams have been working on moving to the new PDF as a single 
input configuration file.
We have proposed a new installer adapter template for Fuel in Pharos [1], as 
well as new PDFs in securedlab for the PODs Fuel uses:
- lf-pod2 [2];
- arm-pod5 has been around for a while;
- ericsson-pod1 and zte-pod1 already have PDFs, but might require smalls 
updates to work with the Fuel installer adapter;

While working on the PDFs, we proposed some patches that should improve/extend 
the verify job for securedlab, use the Pharos git repo for PDF parsing, 
respectively some minor cleanup/code folding:
- securedlab verify job should switch to using Pharos installer adapters and 
generate_config [3];
- yamllint fixes and code folding for existing PDFs [4];
- add verify job summary, run the whole test matrix instead of bailing on the 
first error [5];
- extend verify with yamllint runs for PDF files, as well as output yaml 
file(s) generated by check_jinja2 [6];
- fix missing IPMI credentials for lf-pod4 (caught by linting the output yaml 
described above) [7];
- (unrelated to PDF, Fuel cleanup) remove old Fuel configuration files we no 
longer use [8];

B. PDF specification limitations for Fuel

Tbh, I have a hard time summarizing this, but I'll try.
Currently, we have some PDFs that define a 'net_config' section (global per 
PDF), while the spec and most PDFs don't.
This resulted from:
- the need to support multiple VLANs over the same physical interface;
- installers expecting a network-centric mapping between existing/to-be-created 
networks and available interfaces;
Looking at what Fuel expects as input, we'd like to be able to easily map IPs 
in a certain network (e.g. admin, mgmt, private, public) to a particular 
interface name.
But the PDF does not allow specifying interface names directly, as those depend 
not only on target OS, but also on OS config (e.g. net.ifnames=0 biosdevname=1 
vs net.ifnames=1 biosdevname=0).
Indexing interfaces is also fragile, as a bios upgrade might change PCI layout 
and therefore the NIC ordering (quite rare, but still).
What happens if we add a new interface that ends up at index 1, between 
existing interfaces 0 and 2?
The 'net_config' section is a compromise that solves part of the problem, but 
still does not provide a solid solution for the physical interface name 
mapping, as it also relies on interface index.
What if the controller nodes have a different set of interfaces than the 
compute nodes have?
Adding 'net_config' to PODs aligned with the current spec is also problematic, 
as it'd duplicate the VLAN information, making the whole thing even more 
fragile.



Tl;dr I submited a proof of concept refactor of net_config, which triesto align 
more with the current PDF spec, although it is not 100% compatible (we can't 
model multiple VLANs on the same physical interface with the current spec) [9].
Observations wrt this change:
- network-centric approach, installer-friendly;
- physical interface to virtual interface mapping is NOT 1:1 (multiple VLANs on 
same physical NIC);
- virtual inteface to network mapping is 1:1;
- network to IP mapping is 1:1;
- networks are global for the POD (including network to virtual network);
- network to physical inteface mapping is NOT global per POD, and should be 
overrideable on a per-node basis;
- features and speed params were silently discarded during net_config addition, 
bring them back for the physical intefaces;
- converting from current spec-compatible PDF to this proposed format should be 
trivial for PDF files, and should require very little work on the installer 
adapters;
Etc.

Please take some time to review this and let's come up with a better solution 
if we can find one, or at least align our current PDFs to one format or another.
The PoC does not solve the interface indexing issue, but at least it provides a 
mechanism that makes it per-node configurable.

C. PDF current implementation issues

This section covers some divergent aspects in the current PDFs in securedlab. 
The PDF spec is quite clear on some of these, so I don't understand why there 
are so many mutations in the wild.
If parsing the PDF is hard / do

Re: [opnfv-tech-discuss] PDF feedback and questions from our experience with Fuel

2017-09-26 Thread david.blaisonneau


Hi,

I hope I don't offend you about net_config, it is an hard subject, not 
easy to summarize, and born during summer holidays without everyone 
around the gerrit pit.


About network naming, I understand the need of anonymize the networks 
parts to not over specify the installer level, but by calling them 
'networkX' we will lost a precious information: 'how this network is 
prepared' by ops in network and security aspects. You may had already 
talk about that yesterday (sorry for missing it, job and child schedules 
collision), but simple networking questions must be answered when 
preparing a network mapping: Is this network NATed to outside ? does 
this network can talk to one other ? are some ip/ports opened at 
firewall level ? In Orange side, and for security aspects, we isolate 
some networks: Public can not talk to admin, storage is isolated and is 
not NATed to outside... Using 'admin', 'storage', 'public' naming give a 
part of the information, but 'networkX' don't, except if we are adding a 
description in the PDF like flow matrix, or we add external source, like 
the wiki. How do you think to handle that part ?


This raise a more general question, is there recommendations about 
network topology and security rules in OPNFV pods or each lab is follow 
its own rules ?


For C section, here are my answers:

   1. 'vlan: 0' vs 'vlan: native': OK for native if 0 have a special
   meaning
   2. 'disk_rotation: ssd' or 'disk_rotation: 15000' for SSD drives. Is
   disk_rotation enough to specify a disk performances? don't we also
   need the size of the cache ? we would better talk about bandwidth if
   we want also use it for ssd ? Maybe Bottlenecks project can help us.
   3. 'features: dpdk|sriov' vs 'features: dpdk, sriov': it seems to be
   a collision between PDF specs and the yaml one. If it is a list we
   should simplify the installer parsing by using a yaml list: [ x, y,
   z] or a dashed list
   4. 'features: null' vs 'features: ' vs omitting it altogether:
   'features: []' or omitting; the first is yaml compliant with a list,
   the second is to simplify the pdf.
   5. inconsistent node naming: in my opinion, this part shall be
   reserved for the ops, it is the only way for them to map a node with
   a physical server in the DC (except the ipmi address that is not the
   simple way to find a node). All values shall be authorized, but
   recommandation can be made on 'pod-')
   6. Address/netmask: isn't netmask information in address overlaping
   with net_config netmasks ?
   7. 'os' in POD nodes: Only in jumphost
   8. IPMI interface MAC on the 'interfaces' list: no, it overlaps with
   remote_management/mac_address and is not seen from the os.

Thanks again for this huge work of listing all pending points.

BR

David

 


--
David BLAISONNEAU
NFV platforms DevOPS - OPNFV contributor
Orange - IMT / OLN / CNC / NCA / SINA

tel: +33 (0)2 96 07 27 93
email: david.blaisonn...@orange.com
2, avenue Pierre Marzin 22307 LANNION Cedex - France

Le 25/09/2017 à 20:02, Alexandru Avadanii a écrit :


Hi,

The part about net_config was confusing because it was not (yet) part 
of the official PDF spec.


After today’s infra meeting, and some more discussion on IRC, it looks 
like net_config will become part of the spec, but we need to sort out 
some network naming issues first.


However, ‚admin’, ‚mgmt’ & co network names are not inline with the 
PDF idea of describing strictly the hardware, so they will most likely 
need to be mapped using the installer descriptor file companion for now.


I updated [9] to reflect this decision. It makes things a little bit 
more complicated for the installer adapters, but I think keeping PDF 
clean (with no higher-level specifics) is a good idea overall.


Sections C and D in my first mail still need to be sorted out, but 
they are mostly minor quirks or deviations from the spec, so they are 
not blocking the release of PDF.


BR,

Alex

*From:*david.blaisonn...@orange.com [mailto:david.blaisonn...@orange.com]
*Sent:* Monday, September 25, 2017 12:11 PM
*To:* Alexandru Avadanii; opnfv-tech-discuss@lists.opnfv.org
*Cc:* jack.mor...@intel.com; Guillermo Herrero
*Subject:* Re: PDF feedback and questions from our experience with Fuel

Hi Alexandru,

thanks for this big set of patches and a sum up of our last pdf 
discussions.


The part about net_config is quite confuse, specially because you sum 
up many problems already solved by net_config, but talking about PDF 
not having it, then talking about what add net_config... but it is a 
hard job to summerize those evolution and point that we would better 
understand each one by using PDF version id, and work on a reference 
model.


Generally it will be +1 :)

BR

David

--
David BLAISONNEAU
NFV platforms DevOPS - OPNFV contributor
Orange - IMT / OLN / CNC / NCA / SINA
tel: +33 (0)2 96 07 27 93
email:david.blaisonn...@orange.com 
2, avenue Pierre Marzin 22307 LANNION Cedex - France

Re: [opnfv-tech-discuss] [Infra] Infra WG Weekly Meeting Minutes - Oct. 9th

2017-10-16 Thread david.blaisonneau


Hi everyone,

I agree that the PDF/IDF definitions must be fixed ASAP.

 * *About net_config in IDF*: I think it is a bad idea. As i answer in
   the patch #42233 [2], if the L3 network is configured in the infra
   (firewall, router, ...), and not configurable by the installer, it
   must stay in the PDF. If the PDF leaves an empty L3 network section,
   it could be fixed by the installer, and configured in the IDF.
 * *About net_config as proposed by **Guillermo Herrero in #42893 [1]:*
   +1 with an official naming less openstack 'like'. For a better
   understanding and writing by infra ops, can we authorize comments
   describing what is this network in simple name (and openstack 'like'
   naming if the author like it)? We must consider that ops may not use
   the same convention in the data center, and pdf writting must not be
   painful.
 * *Jobs naming: *Today, jobs in IDF are also openstack centric and can
   only describe a ha or noha mode, but not both, I propose in
   #42233[2] that we use 'infra', 'worker' and 'infra_or_worker' as
   naming for main node's job. This will let SDF file describing what
   is the mapping  to openstack, k8s or any other vim naming.
 * *storage definition in shared IDF part [3]*: Julien propose to add
   the storage definition in IDF, I agree, but it must be consolidated.
   I will comment in the patch

BR

David

[1] https://gerrit.opnfv.org/gerrit/#/c/42893 

[2] 
https://gerrit.opnfv.org/gerrit/#/c/42233 



[3] 
https://gerrit.opnfv.org/gerrit/#/c/44603/1/labs/att/idf-pod1.yaml




--
David BLAISONNEAU
NFV platforms DevOPS - OPNFV contributor
Orange - IMT / OLN / CNC / NCA / SINA

tel: +33 (0)2 96 07 27 93
email: david.blaisonn...@orange.com
2, avenue Pierre Marzin 22307 LANNION Cedex - France

Le 16/10/2017 à 02:25, Julien a écrit :

Hi Alex,

It's a great news.
2 PODs in our lab reserved for MCP are blocked to deploy in master 
branch. Can you give me a PDF/IDF link which are used now, then we 
will submit PDF/IDF patches according to your version.


For **net_config**, I personally suggest to move it to IDF and get the 
template/format approved ASSP, for it is really important to us.


BR/Julien

Alexandru Avadanii >于2017年10月15日周日 下午9:44写道：


Hi,

Fuel has been using PDF and IDF for weeks now.

We still rely on net_config, which is out of spec, to map between
PDF networks and the network role within the installer.

Apart from net_config, we still need to sort out the mapping
between interfaces indexes in PDF and the interface names inside
OS (e.g. iface 0 = eno1, iface 2 = enp1s0f1).

For net_config, Guillermo proposed an alternative as a comment in
[1], which imo is closer to the hardware view of the setup
(especially the port_matrix part).

For iface name mapping, we will probably add it to IDF as a
Fuel-specific section for now (no PoC proposed yet).

BR,

Alex

[1] https://gerrit.opnfv.org/gerrit/#/c/42893/6/labs/lf/pod4.yaml

*From:*opnfv-tech-discuss-boun...@lists.opnfv.org

[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org
] *On Behalf Of
*Julien
*Sent:* Sunday, October 15, 2017 12:38 PM
*To:* Jack Morgan; opnfv-tech-discuss
*Subject:* Re: [opnfv-tech-discuss] [Infra] Infra WG Weekly
Meeting Minutes - Oct. 9th

Hi Jack and Infra team:

This week I have a chance to directly discuss with installer team:
Apex and Compass. I still can not contact Joid.

This is the status of PDF in each installer:

Apex: Not started, won't support in E release, and I have
submitted a Jira ticket in Apex.

Compass: Not started, and plan to support in F release

Daisy: PDF is supported and used in CI PODs for deployment, while
IDF is not used for now

Fuel: Started, not finished yet

Joid: No response in email and IRC(#opnfv-joid)

I would like to suggest to clear PDF/IDF patches in gerrit, let's
make an acceptable template and update them in the future, then
the lab owners can submit their PDF/IDF and used for deployment.

BR/Julien

Jack Morgan mailto:jack.mor...@intel.com>>于2017年10月10日周二 上午12:38写道：


October 9th

*Agenda:*

1.Status update on PDF

·complete discussion from last week as this is a deliverable
for Euphrates release

2.LaaS and Dashboard integration Lincoln Lavoie

Re: [opnfv-tech-discuss] [infra][pharos]Is PDF feature supported by installer project now?

Re: [opnfv-tech-discuss] PDF feedback and questions from our experience with Fuel

Re: [opnfv-tech-discuss] PDF feedback and questions from our experience with Fuel

Re: [opnfv-tech-discuss] [Infra] Infra WG Weekly Meeting Minutes - Oct. 9th

4 matches

Site Navigation

Mail list logo

Footer information