Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-18 Thread Jose Lausuch 
Srikanth,

That issue has been already fixed https://gerrit.opnfv.org/gerrit/#/c/45503/ 
<https://gerrit.opnfv.org/gerrit/#/c/45503/>

Yes, I think the plugfest is a good forum. Unfortunately I won’t be able to 
attend but many folks will be there. We can always start the discussion on any 
of the infra or weekly tech-discuss meetings.

Regards,
Jose





> On 18 Oct 2017, at 19:20, Srikanth Vavilapalli 
> <srikanth.vavilapa...@ericsson.com> wrote:
> 
> Hi Jose
>  
> Yes, the cleaner solution is to move the log collection or any deployer 
> specific actions to outside of test frameworks such that they can be run 
> against any OPNFV compliant platforms. We need to brainstorm in some forum on 
> the best way to achieve this. David, plz help us.
>  
> For the immediate need, we can add a fix in sdnvpn project such that  
> sdnvpn.lib.utils.get_nodes() returns an empty list of nodes rather than 
> throwing exception if the INSTALLER_TYPE is set to anything else other than 
> “apex” and “fuel”. So this is kind of disabling the sdnvpn LOG collection in 
> other installer types.
>  
> Plz let me know any comments or inputs…
>  
> Thanks
> Srikanth
>  
>  
>  
> From: Jose Lausuch [mailto:jalaus...@suse.com] 
> Sent: Tuesday, October 17, 2017 8:18 AM
> To: David McBride <dmcbr...@linuxfoundation.org>; Srikanth Vavilapalli 
> <srikanth.vavilapa...@ericsson.com>
> Cc: Tim Irnich <tim.irn...@ericsson.com>; Tim Rozet <tro...@redhat.com>; 
> opnfv-tech-discuss@lists.opnfv.org
> Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> IMO, the log collection topic is a CI/Releng function rather than the test 
> framework’s responsibility,  we should not discuss it only during the 
> Functest weekly since we would lose audience.
> @David, will you bring this discussion to the infra or technical weekly 
> meeting?
>  
> Thanks,
> Jose
>  
>  
>  
> 
>  
> On 17 Oct 2017, at 00:32, Srikanth Vavilapalli 
> <srikanth.vavilapa...@ericsson.com 
> <mailto:srikanth.vavilapa...@ericsson.com>> wrote:
>  
> Hi
>  
> Can we have a quick discussion in any weekly meeting (sdnvpn, functest?) on 
> this topic to pick the best approach to move forward? Also Xu Dan has raised 
> few other concerns on the location of log files and option for disabling the 
> gathering logs…etc.
>  
> Thanks
> Srikanth
>  
>  
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] On Behalf Of Jose Lausuch
> Sent: Friday, October 13, 2017 4:28 AM
> To: Tim Irnich <tim.irn...@ericsson.com <mailto:tim.irn...@ericsson.com>>
> Cc: opnfv-tech-discuss@lists.opnfv.org 
> <mailto:opnfv-tech-discuss@lists.opnfv.org>
> Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Hi Tim,
>  
> My only concern is that by implementing this function inside the test case, 
> we might run into exceptions raised if there is a problem with SSH keys or 
> whatever other problem accessing the nodes. 
>  
> Collecting logs when the test fails makes sense, but it doesn’t harm if we 
> collect it when it passes as well. Anyway, if we do it only when having 
> failures, I imagine that logic could also be implemented post-job by checking 
> the "criteria": “FAIL" value on the DB and executing some j-job to collect 
> the logs. This logs could be uploaded to artifact repo for further analysis 
> as we do for the Functest logs.
>  
> Regards,
> Jose
>  
>  
>  
> 
>  
> On 13 Oct 2017, at 09:27, Tim Irnich <tim.irn...@ericsson.com 
> <mailto:tim.irn...@ericsson.com>> wrote:
>  
> Jose, 
>  
> I see the point of avoiding log collection in the code for test execution to 
> avoid false negatives in case something with the log collection itself goes 
> wrong (although I’m wondering a bit if that cannot also be handled by 
> carefully implementing the test pass/fail criteria). 
>  
> There are however cases where we’ll want to collect logs only if a test 
> fails, in order to enable a first tier of offline troubleshooting. You’ll 
> remember the gather_logs function Niko implemented for the bgpvpn tests – 
> that is one example. Can that effectively be handled if we separate things as 
> you suggest?
>  
> Regards, Tim
>  
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-18 Thread Srikanth Vavilapalli 
Hi Jose

Yes, the cleaner solution is to move the log collection or any deployer 
specific actions to outside of test frameworks such that they can be run 
against any OPNFV compliant platforms. We need to brainstorm in some forum on 
the best way to achieve this. David, plz help us.

For the immediate need, we can add a fix in sdnvpn project such that  
sdnvpn.lib.utils.get_nodes() returns an empty list of nodes rather than 
throwing exception if the INSTALLER_TYPE is set to anything else other than 
“apex” and “fuel”. So this is kind of disabling the sdnvpn LOG collection in 
other installer types.

Plz let me know any comments or inputs…

Thanks
Srikanth



From: Jose Lausuch [mailto:jalaus...@suse.com]
Sent: Tuesday, October 17, 2017 8:18 AM
To: David McBride <dmcbr...@linuxfoundation.org>; Srikanth Vavilapalli 
<srikanth.vavilapa...@ericsson.com>
Cc: Tim Irnich <tim.irn...@ericsson.com>; Tim Rozet <tro...@redhat.com>; 
opnfv-tech-discuss@lists.opnfv.org
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

IMO, the log collection topic is a CI/Releng function rather than the test 
framework’s responsibility,  we should not discuss it only during the Functest 
weekly since we would lose audience.
@David, will you bring this discussion to the infra or technical weekly meeting?

Thanks,
Jose




On 17 Oct 2017, at 00:32, Srikanth Vavilapalli 
<srikanth.vavilapa...@ericsson.com<mailto:srikanth.vavilapa...@ericsson.com>> 
wrote:

Hi

Can we have a quick discussion in any weekly meeting (sdnvpn, functest?) on 
this topic to pick the best approach to move forward? Also Xu Dan has raised 
few other concerns on the location of log files and option for disabling the 
gathering logs…etc.

Thanks
Srikanth



From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Friday, October 13, 2017 4:28 AM
To: Tim Irnich <tim.irn...@ericsson.com<mailto:tim.irn...@ericsson.com>>
Cc: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi Tim,

My only concern is that by implementing this function inside the test case, we 
might run into exceptions raised if there is a problem with SSH keys or 
whatever other problem accessing the nodes.

Collecting logs when the test fails makes sense, but it doesn’t harm if we 
collect it when it passes as well. Anyway, if we do it only when having 
failures, I imagine that logic could also be implemented post-job by checking 
the "criteria": “FAIL" value on the DB and executing some j-job to collect the 
logs. This logs could be uploaded to artifact repo for further analysis as we 
do for the Functest logs.

Regards,
Jose




On 13 Oct 2017, at 09:27, Tim Irnich 
<tim.irn...@ericsson.com<mailto:tim.irn...@ericsson.com>> wrote:

Jose,

I see the point of avoiding log collection in the code for test execution to 
avoid false negatives in case something with the log collection itself goes 
wrong (although I’m wondering a bit if that cannot also be handled by carefully 
implementing the test pass/fail criteria).

There are however cases where we’ll want to collect logs only if a test fails, 
in order to enable a first tier of offline troubleshooting. You’ll remember the 
gather_logs function Niko implemented for the bgpvpn tests – that is one 
example. Can that effectively be handled if we separate things as you suggest?

Regards, Tim


From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Friday, October 13, 2017 09:16
To: xudan (N) <xuda...@huawei.com<mailto:xuda...@huawei.com>>; Morgan, Jack 
<jack.mor...@intel.com<mailto:jack.mor...@intel.com>>
Cc: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi,

When it comes to log collection, I strongly believe it should be done post-job 
in our CI pipeline, not as part of the test case. Users can always collect logs 
manually regardless of the installer tools they use…

And regarding making it automated in OPNFV after Functest/Yardstick execution, 
would it make sense to re-use the PDF (Pod Descriptor File)? Otherwise, we are 
duplicating config files like pod.yaml or new files with information about the 
servers…
@Jack: is there a possibility to include login information in the PDF (user, 
password, path to private key, …) of the nodes already deployed?

Regards,
Jose




On

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-17 Thread Jose Lausuch 
Hi,

We don´t need IPMI info, but access info (user, IPs, location of private key, 
..) Something similar to what Yardstick is using:
https://github.com/opnfv/yardstick/blob/master/etc/yardstick/nodes/pod.yaml.sample
 
<https://github.com/opnfv/yardstick/blob/master/etc/yardstick/nodes/pod.yaml.sample>

Not sure if that fits to the PDF...

Regards,
Jose

> On 18 Oct 2017, at 00:14, Jack Morgan <jack.mor...@intel.com> wrote:
> 
> Sorry, I was sick last week and just getting caught up on my email. The PDF 
> does have IPMI log in information. There was some concerns with this so Aric 
> is working on a way to secure the log in details. See patch 
> https://gerrit.opnfv.org/gerrit/#/c/44389/ 
> <https://gerrit.opnfv.org/gerrit/#/c/44389/>
> 
> On 10/13/2017 12:15 AM, Jose Lausuch wrote:
>> Hi,
>> 
>> When it comes to log collection, I strongly believe it should be done 
>> post-job in our CI pipeline, not as part of the test case. Users can always 
>> collect logs manually regardless of the installer tools they use… 
>> 
>> And regarding making it automated in OPNFV after Functest/Yardstick 
>> execution, would it make sense to re-use the PDF (Pod Descriptor File)? 
>> Otherwise, we are duplicating config files like pod.yaml or new files with 
>> information about the servers… 
>> @Jack: is there a possibility to include login information in the PDF (user, 
>> password, path to private key, …) of the nodes already deployed?
>> 
>> Regards,
>> Jose
>> 
>> 
>> 
>> 
>> 
>>> On 12 Oct 2017, at 03:50, xudan (N) <xuda...@huawei.com 
>>> <mailto:xuda...@huawei.com>> wrote:
>>> 
>>> Hi Srikanth,
>>>  
>>> I have one question. Are the paths of all these log files constant for 
>>> different environment (Apex, Fuel and commercial deployments)?
>>> If all paths for different deployments are the same, then using config file 
>>> to login and getting files can work.
>>> If not, there will be some errors even though it can login with the config 
>>> file.
>>>  
>>> One more question, what logs do SDNVPN get from all nodes? Are they useful 
>>> for users? If not, can we have an option to disable it?
>>>  
>>> Thanks
>>> Dan Xu
>>>  
>>> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
>>> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
>>> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
>>> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] On Behalf Of 
>>> limingjiang
>>> Sent: Thursday, October 12, 2017 9:24 AM
>>> To: Srikanth Vavilapalli
>>> Cc: opnfv-tech-discuss@lists.opnfv.org 
>>> <mailto:opnfv-tech-discuss@lists.opnfv.org>
>>> Subject: [opnfv-tech-discuss] 答复: [functest] [sdnvpn] Proposal for removing 
>>> installer dependent information in the test tools
>>>  
>>> Hi Srikanth,
>>>  
>>> Yardstick can use a global pod.yaml for test cases.
>>> Since each test case default use the “pod.yaml” located in 
>>> “/etc/yardstick/pod.yaml”. so if you put “pod.yaml” here, it can apply to 
>>> each test case.
>>> The picture you show below is how yardstick test suite customize the input 
>>> parameters, so it also support each test case with different “pod.yaml”
>>> if you give each test case different “pod.yaml”
>>>  
>>> BR,
>>> Rex
>>>  
>>> +---+
>>> 
>>> + Mingjiang Li (Rex) Mobile: +86 13761275017
>>> + Shanghai Institute, Huawei
>>> + No. 2222, Xinjinqiao Road, Pudong, Shanghai, 201206, P.R.China
>>> +---+
>>>  
>>> 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org 
>>> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
>>> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
>>> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] 代表 Srikanth Vavilapalli
>>> 发送时间: 2017年10月12日 9:03
>>> 收件人: Jose Lausuch; Georg Kunz
>>> 抄送: opnfv-tech-discuss@lists.opnfv.org 
>>> <mailto:opnfv-tech-discuss@lists.opnfv.org>
>>> 主题: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
>>> installer dependent information in the test tools
>>>  
>>> Thanks everyone for your inputs. 
>>>  
>>> So if Yardstick based approach is the preferred one, then I am thinking of

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-17 Thread Jack Morgan 

  
  
Sorry, I was sick last week and just getting caught up on my
email. The PDF does have IPMI log in information. There was some
concerns with this so Aric is working on a way to secure the log
in details. See patch https://gerrit.opnfv.org/gerrit/#/c/44389/


On 10/13/2017 12:15 AM, Jose Lausuch
  wrote:


  
  Hi,
  
  
  When it comes to log collection, I strongly believe
it should be done post-job in our CI pipeline, not as part of
the test case. Users can always collect logs manually regardless
of the installer tools they use… 
  
  
  And regarding making it automated in OPNFV after
Functest/Yardstick execution, would it make sense to re-use the
PDF (Pod Descriptor File)? Otherwise, we are duplicating config
files like pod.yaml or new files with information about the
servers… 
  @Jack: is there a possibility to include login
information in the PDF (user, password, path to private key, …)
of the nodes already deployed?
  
  
  

  

  Regards,
  Jose
  
  


  
  
  



  
On 12 Oct 2017, at 03:50, xudan (N) <xuda...@huawei.com> wrote:


  
Hi Srikanth,
 
I have one question. Are the
paths of all these log files constant for different
environment (Apex, Fuel and commercial deployments)?
If all paths for different
deployments are the same, then using config file to
login and getting files can work.
If not, there will be some
errors even though it can login with the config
file.
 
One more question, what logs
do SDNVPN get from all nodes? Are they useful for
users? If not, can we have an option to disable it?
 
Thanks
Dan Xu
 

  
From: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of limingjiang
Sent: Thursday,
October 12, 2017 9:24 AM
To: Srikanth
Vavilapalli
Cc: opnfv-tech-discuss@lists.opnfv.org
Subject: [opnfv-tech-discuss] 答复: [functest] [sdnvpn] Proposal for
removing installer dependent information in the
test tools
  

 
Hi Srikanth,
 
Yardstick can use a global pod.yaml for
test cases.
Since each test case default use the “pod.yaml” located in “/etc/yardstick/pod.yaml”. so if you put “pod.yaml” here, it
  can apply to each test case.
The picture you show below is how
yardstick test suite customize the input parameters,
so it also support each test case with different “pod.yaml”
  if you give each test case different “pod.yaml”
 
BR,
Rex
 

  +---+
  
  + Mingjiang Li (Rex) Mobile:
  +86 13761275017
  + Shanghai Institute,
  Huawei
  + No. ,
  Xinjinqiao Road, Pudong, Shanghai, 201206,
  P.R.China
  +---+

 

  
发件人: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Srikanth Vavilapalli
  发送时间: 2017年10月12日 9:03
收件人: Jose
  Lausuch; Georg Kunz
抄送: opnfv-tech-discuss@lists.opnfv.org
主题: Re:
  [opnfv-tech-discuss] [functest] [sdnvpn]
  Proposal for removing installer dependent
  information in the test tools
  

 
Thanks everyone for your
inputs. 
 
So if Yardstick based approach
is the preferr

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-17 Thread Jose Lausuch 
IMO, the log collection topic is a CI/Releng function rather than the test 
framework’s responsibility,  we should not discuss it only during the Functest 
weekly since we would lose audience.
@David, will you bring this discussion to the infra or technical weekly meeting?

Thanks,
Jose





> On 17 Oct 2017, at 00:32, Srikanth Vavilapalli 
> <srikanth.vavilapa...@ericsson.com> wrote:
> 
> Hi
>  
> Can we have a quick discussion in any weekly meeting (sdnvpn, functest?) on 
> this topic to pick the best approach to move forward? Also Xu Dan has raised 
> few other concerns on the location of log files and option for disabling the 
> gathering logs…etc.
>  
> Thanks
> Srikanth
>  
>  
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
> Sent: Friday, October 13, 2017 4:28 AM
> To: Tim Irnich <tim.irn...@ericsson.com>
> Cc: opnfv-tech-discuss@lists.opnfv.org
> Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Hi Tim,
>  
> My only concern is that by implementing this function inside the test case, 
> we might run into exceptions raised if there is a problem with SSH keys or 
> whatever other problem accessing the nodes. 
>  
> Collecting logs when the test fails makes sense, but it doesn’t harm if we 
> collect it when it passes as well. Anyway, if we do it only when having 
> failures, I imagine that logic could also be implemented post-job by checking 
> the "criteria": “FAIL" value on the DB and executing some j-job to collect 
> the logs. This logs could be uploaded to artifact repo for further analysis 
> as we do for the Functest logs.
>  
> Regards,
> Jose
>  
>  
>  
> 
>  
> On 13 Oct 2017, at 09:27, Tim Irnich <tim.irn...@ericsson.com 
> <mailto:tim.irn...@ericsson.com>> wrote:
>  
> Jose, 
>  
> I see the point of avoiding log collection in the code for test execution to 
> avoid false negatives in case something with the log collection itself goes 
> wrong (although I’m wondering a bit if that cannot also be handled by 
> carefully implementing the test pass/fail criteria). 
>  
> There are however cases where we’ll want to collect logs only if a test 
> fails, in order to enable a first tier of offline troubleshooting. You’ll 
> remember the gather_logs function Niko implemented for the bgpvpn tests – 
> that is one example. Can that effectively be handled if we separate things as 
> you suggest?
>  
> Regards, Tim
>  
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] On Behalf Of Jose Lausuch
> Sent: Friday, October 13, 2017 09:16
> To: xudan (N) <xuda...@huawei.com <mailto:xuda...@huawei.com>>; Morgan, Jack 
> <jack.mor...@intel.com <mailto:jack.mor...@intel.com>>
> Cc: opnfv-tech-discuss@lists.opnfv.org 
> <mailto:opnfv-tech-discuss@lists.opnfv.org>
> Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Hi,
>  
> When it comes to log collection, I strongly believe it should be done 
> post-job in our CI pipeline, not as part of the test case. Users can always 
> collect logs manually regardless of the installer tools they use… 
>  
> And regarding making it automated in OPNFV after Functest/Yardstick 
> execution, would it make sense to re-use the PDF (Pod Descriptor File)? 
> Otherwise, we are duplicating config files like pod.yaml or new files with 
> information about the servers… 
> @Jack: is there a possibility to include login information in the PDF (user, 
> password, path to private key, …) of the nodes already deployed?
>  
> Regards,
> Jose
>  
>  
>  
> 
>  
> On 12 Oct 2017, at 03:50, xudan (N) <xuda...@huawei.com 
> <mailto:xuda...@huawei.com>> wrote:
>  
> Hi Srikanth,
>  
> I have one question. Are the paths of all these log files constant for 
> different environment (Apex, Fuel and commercial deployments)?
> If all paths for different deployments are the same, then using config file 
> to login and getting files can work.
> If not, there will be some errors even though it can login with the config 
> file.
>  
> One more question, what logs do SDNVPN get from all nodes? Are they useful 
> for users? If not, can we have an option to disable it?
>  
> Thanks
> Dan Xu
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-16 Thread Srikanth Vavilapalli 
Hi

Can we have a quick discussion in any weekly meeting (sdnvpn, functest?) on 
this topic to pick the best approach to move forward? Also Xu Dan has raised 
few other concerns on the location of log files and option for disabling the 
gathering logs…etc.

Thanks
Srikanth



From: opnfv-tech-discuss-boun...@lists.opnfv.org 
[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Friday, October 13, 2017 4:28 AM
To: Tim Irnich <tim.irn...@ericsson.com>
Cc: opnfv-tech-discuss@lists.opnfv.org
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi Tim,

My only concern is that by implementing this function inside the test case, we 
might run into exceptions raised if there is a problem with SSH keys or 
whatever other problem accessing the nodes.

Collecting logs when the test fails makes sense, but it doesn’t harm if we 
collect it when it passes as well. Anyway, if we do it only when having 
failures, I imagine that logic could also be implemented post-job by checking 
the "criteria": “FAIL" value on the DB and executing some j-job to collect the 
logs. This logs could be uploaded to artifact repo for further analysis as we 
do for the Functest logs.

Regards,
Jose




On 13 Oct 2017, at 09:27, Tim Irnich 
<tim.irn...@ericsson.com<mailto:tim.irn...@ericsson.com>> wrote:

Jose,

I see the point of avoiding log collection in the code for test execution to 
avoid false negatives in case something with the log collection itself goes 
wrong (although I’m wondering a bit if that cannot also be handled by carefully 
implementing the test pass/fail criteria).

There are however cases where we’ll want to collect logs only if a test fails, 
in order to enable a first tier of offline troubleshooting. You’ll remember the 
gather_logs function Niko implemented for the bgpvpn tests – that is one 
example. Can that effectively be handled if we separate things as you suggest?

Regards, Tim


From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Friday, October 13, 2017 09:16
To: xudan (N) <xuda...@huawei.com<mailto:xuda...@huawei.com>>; Morgan, Jack 
<jack.mor...@intel.com<mailto:jack.mor...@intel.com>>
Cc: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi,

When it comes to log collection, I strongly believe it should be done post-job 
in our CI pipeline, not as part of the test case. Users can always collect logs 
manually regardless of the installer tools they use…

And regarding making it automated in OPNFV after Functest/Yardstick execution, 
would it make sense to re-use the PDF (Pod Descriptor File)? Otherwise, we are 
duplicating config files like pod.yaml or new files with information about the 
servers…
@Jack: is there a possibility to include login information in the PDF (user, 
password, path to private key, …) of the nodes already deployed?

Regards,
Jose




On 12 Oct 2017, at 03:50, xudan (N) 
<xuda...@huawei.com<mailto:xuda...@huawei.com>> wrote:

Hi Srikanth,

I have one question. Are the paths of all these log files constant for 
different environment (Apex, Fuel and commercial deployments)?
If all paths for different deployments are the same, then using config file to 
login and getting files can work.
If not, there will be some errors even though it can login with the config file.

One more question, what logs do SDNVPN get from all nodes? Are they useful for 
users? If not, can we have an option to disable it?

Thanks
Dan Xu

From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of limingjiang
Sent: Thursday, October 12, 2017 9:24 AM
To: Srikanth Vavilapalli
Cc: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Subject: [opnfv-tech-discuss] 答复: [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi Srikanth,

Yardstick can use a global pod.yaml for test cases.
Since each test case default use the “pod.yaml” located in 
“/etc/yardstick/pod.yaml”. so if you put “pod.yaml” here, it can apply to each 
test case.
The picture you show below is how yardstick test suite customize the input 
parameters, so it also support each test case with different “pod.yaml”
if you give each test case different “pod.yaml”

BR,
Rex

+---+

+ Mingjiang Li (Rex) Mobile: +86 13761275017
+ Shanghai Institute, Huawei
+ No. , Xinjinqiao Road, Pudong, Shanghai, 201206, P.R.China
+---

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-16 Thread Srikanth Vavilapalli 
Thanks Rex for the clarification.

From: limingjiang [mailto:limingji...@huawei.com]
Sent: Wednesday, October 11, 2017 6:24 PM
To: Srikanth Vavilapalli <srikanth.vavilapa...@ericsson.com>
Cc: opnfv-tech-discuss@lists.opnfv.org; Jose Lausuch <jalaus...@suse.com>; 
Georg Kunz <georg.k...@ericsson.com>
Subject: 答复: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi Srikanth,

Yardstick can use a global pod.yaml for test cases.
Since each test case default use the “pod.yaml” located in 
“/etc/yardstick/pod.yaml”. so if you put “pod.yaml” here, it can apply to each 
test case.
The picture you show below is how yardstick test suite customize the input 
parameters, so it also support each test case with different “pod.yaml”
if you give each test case different “pod.yaml”

BR,
Rex

+---+
[cid:image001.png@01D0A50A.DD5A8F20]
+ Mingjiang Li (Rex) Mobile: +86 13761275017
+ Shanghai Institute, Huawei
+ No. , Xinjinqiao Road, Pudong, Shanghai, 201206, P.R.China
+---+

发件人: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Srikanth Vavilapalli
发送时间: 2017年10月12日 9:03
收件人: Jose Lausuch; Georg Kunz
抄送: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
主题: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Thanks everyone for your inputs.

So if Yardstick based approach is the preferred one, then I am thinking of 
extending the existing Deployment Factory class with a new generic INSTALLER 
type (something like “config-file” or so) which will provide the same interface 
as other adapters (ApexAdapter, FuelAdapter…etc), but instead reads from the a 
configured pod.yaml file to provide Node information. Plz let me know if you 
see any issues with this approach.

One quick question on Yardstick: Looks like Yardstick accepts the custom 
pod.yaml file on a per test case basis as shown in the below example. Can it 
also accept a global pod.yaml file that can be applied to all or a group of 
test cases.
-
file_name: opnfv_yardstick_tc043.yaml
   constraint:
  installer: xxx
  pod: xxx-pod1
   task_args:
  xxx-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
  "host": "node1.LF","target": "node2.LF"}'

Thanks
Srikanth

From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Wednesday, October 11, 2017 10:45 AM
To: Georg Kunz <georg.k...@ericsson.com<mailto:georg.k...@ericsson.com>>
Cc: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi,

I would vote for having something similar to Yardstick [1] but centralized in 
Releng with an easy python lib that enables functions like SCP things to/from 
the deployed nodes.

For your third point, log collection shouldn’t be done at test case level. It 
should be performed by CI after running the test tools, otherwise you can a 
false negative when running those test on non-OPNFV installers.

Regards,
Jose

[1] 
https://raw.githubusercontent.com/opnfv/yardstick/master/etc/yardstick/nodes/fuel_virtual/pod.yaml


On 11 Oct 2017, at 18:08, Georg Kunz 
<georg.k...@ericsson.com<mailto:georg.k...@ericsson.com>> wrote:

Hi,

Just to highlight this, from a Dovetail/CVP perspective, the important aspect 
is that there are no dependencies on OPNFV-specific resources/lib in order to 
be able to run test cases against commercial non-OPNFV deployments.

Having to write an adapter for a particular commercial deployment before you 
can run Dovetail is obviously not really an option. So, for tests which require 
SSH/SCP access, we need to think about...

  *   If the adapter can be parameterized, so that we can make it a 
configuration option, e.g., specifying login credentials, source and target 
directories, etc., similarly to Yardstick.
  *   Reuse what Yardstick is using?
  *   If the test case be parameterized such that it does not attempt to gather 
logs if used for certification? (limited use, of course)
  *   …

Cheers
Georg

From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Wednesday, October 11, 2017 4:40 PM
To: Beierl, Mark <mark.bei...@dell.com<mailto:mark.bei...

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-13 Thread Jose Lausuch 
Hi Tim,

My only concern is that by implementing this function inside the test case, we 
might run into exceptions raised if there is a problem with SSH keys or 
whatever other problem accessing the nodes. 

Collecting logs when the test fails makes sense, but it doesn’t harm if we 
collect it when it passes as well. Anyway, if we do it only when having 
failures, I imagine that logic could also be implemented post-job by checking 
the "criteria": “FAIL" value on the DB and executing some j-job to collect the 
logs. This logs could be uploaded to artifact repo for further analysis as we 
do for the Functest logs.

Regards,
Jose





> On 13 Oct 2017, at 09:27, Tim Irnich <tim.irn...@ericsson.com> wrote:
> 
> Jose, 
>  
> I see the point of avoiding log collection in the code for test execution to 
> avoid false negatives in case something with the log collection itself goes 
> wrong (although I’m wondering a bit if that cannot also be handled by 
> carefully implementing the test pass/fail criteria). 
>  
> There are however cases where we’ll want to collect logs only if a test 
> fails, in order to enable a first tier of offline troubleshooting. You’ll 
> remember the gather_logs function Niko implemented for the bgpvpn tests – 
> that is one example. Can that effectively be handled if we separate things as 
> you suggest?
>  
> Regards, Tim
>  
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
> Sent: Friday, October 13, 2017 09:16
> To: xudan (N) <xuda...@huawei.com>; Morgan, Jack <jack.mor...@intel.com>
> Cc: opnfv-tech-discuss@lists.opnfv.org
> Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Hi,
>  
> When it comes to log collection, I strongly believe it should be done 
> post-job in our CI pipeline, not as part of the test case. Users can always 
> collect logs manually regardless of the installer tools they use… 
>  
> And regarding making it automated in OPNFV after Functest/Yardstick 
> execution, would it make sense to re-use the PDF (Pod Descriptor File)? 
> Otherwise, we are duplicating config files like pod.yaml or new files with 
> information about the servers… 
> @Jack: is there a possibility to include login information in the PDF (user, 
> password, path to private key, …) of the nodes already deployed?
>  
> Regards,
> Jose
>  
>  
>  
> 
>  
> On 12 Oct 2017, at 03:50, xudan (N) <xuda...@huawei.com 
> <mailto:xuda...@huawei.com>> wrote:
>  
> Hi Srikanth,
>  
> I have one question. Are the paths of all these log files constant for 
> different environment (Apex, Fuel and commercial deployments)?
> If all paths for different deployments are the same, then using config file 
> to login and getting files can work.
> If not, there will be some errors even though it can login with the config 
> file.
>  
> One more question, what logs do SDNVPN get from all nodes? Are they useful 
> for users? If not, can we have an option to disable it?
>  
> Thanks
> Dan Xu
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] On Behalf Of limingjiang
> Sent: Thursday, October 12, 2017 9:24 AM
> To: Srikanth Vavilapalli
> Cc: opnfv-tech-discuss@lists.opnfv.org 
> <mailto:opnfv-tech-discuss@lists.opnfv.org>
> Subject: [opnfv-tech-discuss] 答复: [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Hi Srikanth,
>  
> Yardstick can use a global pod.yaml for test cases.
> Since each test case default use the “pod.yaml” located in 
> “/etc/yardstick/pod.yaml”. so if you put “pod.yaml” here, it can apply to 
> each test case.
> The picture you show below is how yardstick test suite customize the input 
> parameters, so it also support each test case with different “pod.yaml”
> if you give each test case different “pod.yaml”
>  
> BR,
> Rex
>  
> +---+
> 
> + Mingjiang Li (Rex) Mobile: +86 13761275017
> + Shanghai Institute, Huawei
> + No. , Xinjinqiao Road, Pudong, Shanghai, 201206, P.R.China
> +---+
>  
> 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-13 Thread Tim Irnich 
Jose,

I see the point of avoiding log collection in the code for test execution to 
avoid false negatives in case something with the log collection itself goes 
wrong (although I’m wondering a bit if that cannot also be handled by carefully 
implementing the test pass/fail criteria).

There are however cases where we’ll want to collect logs only if a test fails, 
in order to enable a first tier of offline troubleshooting. You’ll remember the 
gather_logs function Niko implemented for the bgpvpn tests – that is one 
example. Can that effectively be handled if we separate things as you suggest?

Regards, Tim


From: opnfv-tech-discuss-boun...@lists.opnfv.org 
[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Friday, October 13, 2017 09:16
To: xudan (N) <xuda...@huawei.com>; Morgan, Jack <jack.mor...@intel.com>
Cc: opnfv-tech-discuss@lists.opnfv.org
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi,

When it comes to log collection, I strongly believe it should be done post-job 
in our CI pipeline, not as part of the test case. Users can always collect logs 
manually regardless of the installer tools they use…

And regarding making it automated in OPNFV after Functest/Yardstick execution, 
would it make sense to re-use the PDF (Pod Descriptor File)? Otherwise, we are 
duplicating config files like pod.yaml or new files with information about the 
servers…
@Jack: is there a possibility to include login information in the PDF (user, 
password, path to private key, …) of the nodes already deployed?

Regards,
Jose




On 12 Oct 2017, at 03:50, xudan (N) 
<xuda...@huawei.com<mailto:xuda...@huawei.com>> wrote:

Hi Srikanth,

I have one question. Are the paths of all these log files constant for 
different environment (Apex, Fuel and commercial deployments)?
If all paths for different deployments are the same, then using config file to 
login and getting files can work.
If not, there will be some errors even though it can login with the config file.

One more question, what logs do SDNVPN get from all nodes? Are they useful for 
users? If not, can we have an option to disable it?

Thanks
Dan Xu

From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of limingjiang
Sent: Thursday, October 12, 2017 9:24 AM
To: Srikanth Vavilapalli
Cc: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Subject: [opnfv-tech-discuss] 答复: [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi Srikanth,

Yardstick can use a global pod.yaml for test cases.
Since each test case default use the “pod.yaml” located in 
“/etc/yardstick/pod.yaml”. so if you put “pod.yaml” here, it can apply to each 
test case.
The picture you show below is how yardstick test suite customize the input 
parameters, so it also support each test case with different “pod.yaml”
if you give each test case different “pod.yaml”

BR,
Rex

+---+

+ Mingjiang Li (Rex) Mobile: +86 13761275017
+ Shanghai Institute, Huawei
+ No. , Xinjinqiao Road, Pudong, Shanghai, 201206, P.R.China
+---+

发件人: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Srikanth Vavilapalli
发送时间: 2017年10月12日 9:03
收件人: Jose Lausuch; Georg Kunz
抄送: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
主题: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Thanks everyone for your inputs.

So if Yardstick based approach is the preferred one, then I am thinking of 
extending the existing Deployment Factory class with a new generic INSTALLER 
type (something like “config-file” or so) which will provide the same interface 
as other adapters (ApexAdapter, FuelAdapter…etc), but instead reads from the a 
configured pod.yaml file to provide Node information. Plz let me know if you 
see any issues with this approach.

One quick question on Yardstick: Looks like Yardstick accepts the custom 
pod.yaml file on a per test case basis as shown in the below example. Can it 
also accept a global pod.yaml file that can be applied to all or a group of 
test cases.
-
file_name: opnfv_yardstick_tc043.yaml
   constraint:
  installer: xxx
  pod: xxx-pod1
   task_args:
  xxx-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
  "host": "node1.LF","target": "node2.LF"}'

Thanks
Srikanth

From: 
opnfv-tech-discuss

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-13 Thread Jose Lausuch 
Hi,

When it comes to log collection, I strongly believe it should be done post-job 
in our CI pipeline, not as part of the test case. Users can always collect logs 
manually regardless of the installer tools they use… 

And regarding making it automated in OPNFV after Functest/Yardstick execution, 
would it make sense to re-use the PDF (Pod Descriptor File)? Otherwise, we are 
duplicating config files like pod.yaml or new files with information about the 
servers… 
@Jack: is there a possibility to include login information in the PDF (user, 
password, path to private key, …) of the nodes already deployed?

Regards,
Jose





> On 12 Oct 2017, at 03:50, xudan (N) <xuda...@huawei.com> wrote:
> 
> Hi Srikanth,
>  
> I have one question. Are the paths of all these log files constant for 
> different environment (Apex, Fuel and commercial deployments)?
> If all paths for different deployments are the same, then using config file 
> to login and getting files can work.
> If not, there will be some errors even though it can login with the config 
> file.
>  
> One more question, what logs do SDNVPN get from all nodes? Are they useful 
> for users? If not, can we have an option to disable it?
>  
> Thanks
> Dan Xu
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] On Behalf Of limingjiang
> Sent: Thursday, October 12, 2017 9:24 AM
> To: Srikanth Vavilapalli
> Cc: opnfv-tech-discuss@lists.opnfv.org 
> <mailto:opnfv-tech-discuss@lists.opnfv.org>
> Subject: [opnfv-tech-discuss] 答复: [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Hi Srikanth,
>  
> Yardstick can use a global pod.yaml for test cases.
> Since each test case default use the “pod.yaml” located in 
> “/etc/yardstick/pod.yaml”. so if you put “pod.yaml” here, it can apply to 
> each test case.
> The picture you show below is how yardstick test suite customize the input 
> parameters, so it also support each test case with different “pod.yaml”
> if you give each test case different “pod.yaml”
>  
> BR,
> Rex
>  
> +---+
> 
> + Mingjiang Li (Rex) Mobile: +86 13761275017
> + Shanghai Institute, Huawei
> + No. , Xinjinqiao Road, Pudong, Shanghai, 201206, P.R.China
> +---+
>  
> 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] 代表 Srikanth Vavilapalli
> 发送时间: 2017年10月12日 9:03
> 收件人: Jose Lausuch; Georg Kunz
> 抄送: opnfv-tech-discuss@lists.opnfv.org 
> <mailto:opnfv-tech-discuss@lists.opnfv.org>
> 主题: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Thanks everyone for your inputs. 
>  
> So if Yardstick based approach is the preferred one, then I am thinking of 
> extending the existing Deployment Factory class with a new generic INSTALLER 
> type (something like “config-file” or so) which will provide the same 
> interface as other adapters (ApexAdapter, FuelAdapter…etc), but instead reads 
> from the a configured pod.yaml file to provide Node information. Plz let me 
> know if you see any issues with this approach.
>  
> One quick question on Yardstick: Looks like Yardstick accepts the custom 
> pod.yaml file on a per test case basis as shown in the below example. Can it 
> also accept a global pod.yaml file that can be applied to all or a group of 
> test cases. 
> -
> 
> file_name: opnfv_yardstick_tc043.yaml
> 
>constraint:
> 
>   installer: xxx
> 
>   pod: xxx-pod1
> 
>task_args:
> 
>   xxx-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
> 
>   "host": "node1.LF","target": "node2.LF"}'
> 
>  
> Thanks
> Srikanth
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] On Behalf Of Jose Lausuch
> Sent: Wednesday, October 11, 2017 10:45 AM
> To: Georg Kunz <georg.k...@ericsson.com <mailto:georg.k...@ericsson.com>>
> Cc: opnfv-tech-discuss@lists.opnfv.org 
> <mailto:opnfv-tech-discuss@lists.opnf

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-11 Thread Srikanth Vavilapalli 
Thanks everyone for your inputs.

So if Yardstick based approach is the preferred one, then I am thinking of 
extending the existing Deployment Factory class with a new generic INSTALLER 
type (something like “config-file” or so) which will provide the same interface 
as other adapters (ApexAdapter, FuelAdapter…etc), but instead reads from the a 
configured pod.yaml file to provide Node information. Plz let me know if you 
see any issues with this approach.

One quick question on Yardstick: Looks like Yardstick accepts the custom 
pod.yaml file on a per test case basis as shown in the below example. Can it 
also accept a global pod.yaml file that can be applied to all or a group of 
test cases.
-
file_name: opnfv_yardstick_tc043.yaml
   constraint:
  installer: xxx
  pod: xxx-pod1
   task_args:
  xxx-pod1: '{"pod_info": "etc/yardstick/.../pod.yaml",
  "host": "node1.LF","target": "node2.LF"}'

Thanks
Srikanth

From: opnfv-tech-discuss-boun...@lists.opnfv.org 
[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Wednesday, October 11, 2017 10:45 AM
To: Georg Kunz <georg.k...@ericsson.com>
Cc: opnfv-tech-discuss@lists.opnfv.org
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi,

I would vote for having something similar to Yardstick [1] but centralized in 
Releng with an easy python lib that enables functions like SCP things to/from 
the deployed nodes.

For your third point, log collection shouldn’t be done at test case level. It 
should be performed by CI after running the test tools, otherwise you can a 
false negative when running those test on non-OPNFV installers.

Regards,
Jose

[1] 
https://raw.githubusercontent.com/opnfv/yardstick/master/etc/yardstick/nodes/fuel_virtual/pod.yaml


On 11 Oct 2017, at 18:08, Georg Kunz 
<georg.k...@ericsson.com<mailto:georg.k...@ericsson.com>> wrote:

Hi,

Just to highlight this, from a Dovetail/CVP perspective, the important aspect 
is that there are no dependencies on OPNFV-specific resources/lib in order to 
be able to run test cases against commercial non-OPNFV deployments.

Having to write an adapter for a particular commercial deployment before you 
can run Dovetail is obviously not really an option. So, for tests which require 
SSH/SCP access, we need to think about...

  *   If the adapter can be parameterized, so that we can make it a 
configuration option, e.g., specifying login credentials, source and target 
directories, etc., similarly to Yardstick.
  *   Reuse what Yardstick is using?
  *   If the test case be parameterized such that it does not attempt to gather 
logs if used for certification? (limited use, of course)
  *   …

Cheers
Georg

From: 
opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>
 [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Wednesday, October 11, 2017 4:40 PM
To: Beierl, Mark <mark.bei...@dell.com<mailto:mark.bei...@dell.com>>
Cc: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi,

With regards to Functest, you can run it on any OpenStack deployment as long as 
you provide a proper RC file and meet the requirements on the jumphost (docker, 
connectivity to the deployment, …).

However, in some cases, some test cases from feature projects require SSH 
access to the deployment and to make things centralized, the deployment handler 
was created [1]. This is a library that allows users to get the number of nodes 
from the deployment, functions to SCP things from the nodes and some other 
utils. The bad part of it is that it only supports Apex, Fuel and OSA for now…  
unless someones volunteers to write the other adapters for joid, mcp, compass 
osa..  This library might be used to extract the desired logs after 
Functest/Yardstick runs in CI to place them in artifact repo and post-analize.

Regards,
Jose

[1] https://git.opnfv.org/releng/tree/modules/opnfv/deployment




On 11 Oct 2017, at 16:23, Beierl, Mark 
<mark.bei...@dell.com<mailto:mark.bei...@dell.com>> wrote:

Hello,

StorPerf very much relies on knowledge of the installer to gather information 
about the block storage underlay.  For example, the number of Ceph nodes, or 
even Ceph vs. LVM, is very relevant to the final report.  I also wish there 
were an installer agnostic method of collecting this information as right now I 
keep that code in the ci/daily.sh and other scripts.

With the new releng repository being created, perhaps it is time to start 
moving some of the installer specific code there?  I also see that being of 
benefit when adding XCI support, as technica

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-11 Thread Jose Lausuch 
Hi,

I would vote for having something similar to Yardstick [1] but centralized in 
Releng with an easy python lib that enables functions like SCP things to/from 
the deployed nodes.

For your third point, log collection shouldn’t be done at test case level. It 
should be performed by CI after running the test tools, otherwise you can a 
false negative when running those test on non-OPNFV installers.

Regards,
Jose

[1] 
https://raw.githubusercontent.com/opnfv/yardstick/master/etc/yardstick/nodes/fuel_virtual/pod.yaml
 
<https://raw.githubusercontent.com/opnfv/yardstick/master/etc/yardstick/nodes/fuel_virtual/pod.yaml>



> On 11 Oct 2017, at 18:08, Georg Kunz <georg.k...@ericsson.com> wrote:
> 
> Hi,
>  
> Just to highlight this, from a Dovetail/CVP perspective, the important aspect 
> is that there are no dependencies on OPNFV-specific resources/lib in order to 
> be able to run test cases against commercial non-OPNFV deployments.
>  
> Having to write an adapter for a particular commercial deployment before you 
> can run Dovetail is obviously not really an option. So, for tests which 
> require SSH/SCP access, we need to think about...
> If the adapter can be parameterized, so that we can make it a configuration 
> option, e.g., specifying login credentials, source and target directories, 
> etc., similarly to Yardstick.
> Reuse what Yardstick is using?
> If the test case be parameterized such that it does not attempt to gather 
> logs if used for certification? (limited use, of course)
> …
>  
> Cheers
> Georg
>  
> From: opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> 
> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org 
> <mailto:opnfv-tech-discuss-boun...@lists.opnfv.org>] On Behalf Of Jose Lausuch
> Sent: Wednesday, October 11, 2017 4:40 PM
> To: Beierl, Mark <mark.bei...@dell.com <mailto:mark.bei...@dell.com>>
> Cc: opnfv-tech-discuss@lists.opnfv.org 
> <mailto:opnfv-tech-discuss@lists.opnfv.org>
> Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
> installer dependent information in the test tools
>  
> Hi,
>  
> With regards to Functest, you can run it on any OpenStack deployment as long 
> as you provide a proper RC file and meet the requirements on the jumphost 
> (docker, connectivity to the deployment, …).
>  
> However, in some cases, some test cases from feature projects require SSH 
> access to the deployment and to make things centralized, the deployment 
> handler was created [1]. This is a library that allows users to get the 
> number of nodes from the deployment, functions to SCP things from the nodes 
> and some other utils. The bad part of it is that it only supports Apex, Fuel 
> and OSA for now…  unless someones volunteers to write the other adapters for 
> joid, mcp, compass osa..  This library might be used to extract the desired 
> logs after Functest/Yardstick runs in CI to place them in artifact repo and 
> post-analize. 
>  
> Regards,
> Jose
>  
> [1] https://git.opnfv.org/releng/tree/modules/opnfv/deployment 
> <https://git.opnfv.org/releng/tree/modules/opnfv/deployment>
>  
>  
>  
>  
> On 11 Oct 2017, at 16:23, Beierl, Mark <mark.bei...@dell.com 
> <mailto:mark.bei...@dell.com>> wrote:
>  
> Hello,
>  
> StorPerf very much relies on knowledge of the installer to gather information 
> about the block storage underlay.  For example, the number of Ceph nodes, or 
> even Ceph vs. LVM, is very relevant to the final report.  I also wish there 
> were an installer agnostic method of collecting this information as right now 
> I keep that code in the ci/daily.sh and other scripts.
>  
> With the new releng repository being created, perhaps it is time to start 
> moving some of the installer specific code there?  I also see that being of 
> benefit when adding XCI support, as technically that would be yet another 
> type of installer.
>  
> Regards,
> Mark
>  
> Mark Beierl
> SW System Sr Principal Engineer
> Dell EMC | Office of the CTO
> mobile +1 613 314 8106 
> mark.bei...@dell.com <mailto:mark.bei...@dell.com>
>  
> On Oct 11, 2017, at 02:25, xudan (N) <xuda...@huawei.com 
> <mailto:xuda...@huawei.com>> wrote:
>  
> Hi Srikanth,
>  
> As I know, some Yardstick test cases also need to login nodes. Yardstick uses 
> a file providing all the login information.
> You can refer to 
> https://github.com/opnfv/yardstick/tree/master/etc/yardstick/nodes 
> <https://github.com/opnfv/yardstick/tree/master/etc/yardstick/nodes> which 
> gives some examples.
> Hope this will help you.
>  
> BR
> Dan Xu
>  
> From: Srikanth Vavi

Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-11 Thread Georg Kunz 
Hi,

Just to highlight this, from a Dovetail/CVP perspective, the important aspect 
is that there are no dependencies on OPNFV-specific resources/lib in order to 
be able to run test cases against commercial non-OPNFV deployments.

Having to write an adapter for a particular commercial deployment before you 
can run Dovetail is obviously not really an option. So, for tests which require 
SSH/SCP access, we need to think about...

  *   If the adapter can be parameterized, so that we can make it a 
configuration option, e.g., specifying login credentials, source and target 
directories, etc., similarly to Yardstick.
  *   Reuse what Yardstick is using?
  *   If the test case be parameterized such that it does not attempt to gather 
logs if used for certification? (limited use, of course)
  *   …

Cheers
Georg

From: opnfv-tech-discuss-boun...@lists.opnfv.org 
[mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of Jose Lausuch
Sent: Wednesday, October 11, 2017 4:40 PM
To: Beierl, Mark <mark.bei...@dell.com>
Cc: opnfv-tech-discuss@lists.opnfv.org
Subject: Re: [opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing 
installer dependent information in the test tools

Hi,

With regards to Functest, you can run it on any OpenStack deployment as long as 
you provide a proper RC file and meet the requirements on the jumphost (docker, 
connectivity to the deployment, …).

However, in some cases, some test cases from feature projects require SSH 
access to the deployment and to make things centralized, the deployment handler 
was created [1]. This is a library that allows users to get the number of nodes 
from the deployment, functions to SCP things from the nodes and some other 
utils. The bad part of it is that it only supports Apex, Fuel and OSA for now…  
unless someones volunteers to write the other adapters for joid, mcp, compass 
osa..  This library might be used to extract the desired logs after 
Functest/Yardstick runs in CI to place them in artifact repo and post-analize.

Regards,
Jose

[1] https://git.opnfv.org/releng/tree/modules/opnfv/deployment




On 11 Oct 2017, at 16:23, Beierl, Mark 
<mark.bei...@dell.com<mailto:mark.bei...@dell.com>> wrote:

Hello,

StorPerf very much relies on knowledge of the installer to gather information 
about the block storage underlay.  For example, the number of Ceph nodes, or 
even Ceph vs. LVM, is very relevant to the final report.  I also wish there 
were an installer agnostic method of collecting this information as right now I 
keep that code in the ci/daily.sh and other scripts.

With the new releng repository being created, perhaps it is time to start 
moving some of the installer specific code there?  I also see that being of 
benefit when adding XCI support, as technically that would be yet another type 
of installer.

Regards,
Mark

Mark Beierl
SW System Sr Principal Engineer
Dell EMC | Office of the CTO
mobile +1 613 314 8106
mark.bei...@dell.com<mailto:mark.bei...@dell.com>

On Oct 11, 2017, at 02:25, xudan (N) 
<xuda...@huawei.com<mailto:xuda...@huawei.com>> wrote:

Hi Srikanth,

As I know, some Yardstick test cases also need to login nodes. Yardstick uses a 
file providing all the login information.
You can refer to 
https://github.com/opnfv/yardstick/tree/master/etc/yardstick/nodes which gives 
some examples.
Hope this will help you.

BR
Dan Xu

From: Srikanth Vavilapalli [mailto:srikanth.vavilapa...@ericsson.com]
Sent: Wednesday, October 11, 2017 12:28 PM
To: 
opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>
Cc: Tim Irnich; xudan (N)
Subject: [functest] [sdnvpn] Proposal for removing installer dependent 
information in the test tools

Hi

I am looking into Jira ticket 
“SDNVPN-181<https://jira.opnfv.org/browse/SDNVPN-181>: Function "gather_logs" 
restricts to Apex and Fuel”, which raises concerns on having installer 
dependent logic in the sdnvpn repo. The issue is, at the end of the sdnvpn test 
execution, we are invoking gather_logs() utility which internally tries to 
gather the information about all the OpenStack nodes based on the configured 
INSTALLER_TYPE in order to run the fetch_logs.sh script on the target OpenStack 
nodes 
(https://gerrit.opnfv.org/gerrit/gitweb?p=sdnvpn.git;a=blob;f=sdnvpn/lib/utils.py;h=ad0714ea9dd40ee8305cd17e42695f0176e88328;hb=HEAD#l215)

So, the jira ticket proposes to accept all the needed information about the 
OpenStack controllers, compute nodes and the associated username, keys…etc. in 
a file format such that these tests can also be run on OPNFV based commercial 
products deployed with their custom deployment tools.

So in general, in the test tools, is there any need to have awareness of what 
installers being used when we all care about the target OpenStack node IPs, 
associated attributes and jumphost IP (in some cases)?

I would like to get the community opinion here. Appreciate your inputs.

Thanks
Srikanth

[opnfv-tech-discuss] [functest] [sdnvpn] Proposal for removing installer dependent information in the test tools

2017-10-10 Thread Srikanth Vavilapalli 
Hi

I am looking into Jira ticket 
"SDNVPN-181: Function "gather_logs" 
restricts to Apex and Fuel", which raises concerns on having installer 
dependent logic in the sdnvpn repo. The issue is, at the end of the sdnvpn test 
execution, we are invoking gather_logs() utility which internally tries to 
gather the information about all the OpenStack nodes based on the configured 
INSTALLER_TYPE in order to run the fetch_logs.sh script on the target OpenStack 
nodes 
(https://gerrit.opnfv.org/gerrit/gitweb?p=sdnvpn.git;a=blob;f=sdnvpn/lib/utils.py;h=ad0714ea9dd40ee8305cd17e42695f0176e88328;hb=HEAD#l215)

So, the jira ticket proposes to accept all the needed information about the 
OpenStack controllers, compute nodes and the associated username, keys...etc. 
in a file format such that these tests can also be run on OPNFV based 
commercial products deployed with their custom deployment tools.

So in general, in the test tools, is there any need to have awareness of what 
installers being used when we all care about the target OpenStack node IPs, 
associated attributes and jumphost IP (in some cases)?

I would like to get the community opinion here. Appreciate your inputs.

Thanks
Srikanth

___
opnfv-tech-discuss mailing list
opnfv-tech-discuss@lists.opnfv.org
https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss