Re: Call for participation: Issue triaging and PR review/testing

[Rohit Yadav]

Hi Ivan,


I took some time to reflect and get back to you:


I agree the freeze date may be a bit aggressive, based on past experiences that 
we've all seen the final release may take some time (weeks even). After the 
freeze, we can all work towards the stability, fix bugs for a stable and proper 
release.


A non-LTS branch such as 4.10 may not receive much usage and attention, 
therefore less reporting of bugs and fixes. For production users, an LTS 
release is recommended who are keen on stability than (new) features, the 
current one being the 4.9.x releases. Starting with 4.11 it's something we'll 
fix on the website i.e. to recommend most users to stay with LTS releases, give 
information about support dates for such releases etc. Like any software 
release, there will be bugs and it's not an issue as long as they get reported 
and fixed.


I'm glad to see your participation, PRs and will work with you and others for a 
stable release. Keep sending PRs, review other's PRs and help with testing and 
reporting of bugs!


- Rohit


From: Ivan Kudryavtsev 
Sent: Wednesday, December 13, 2017 9:34:43 AM
To: d...@cloudstack.apache.org
Cc: Rohit Yadav; users@cloudstack.apache.org
Subject: Re: Call for participation: Issue triaging and PR review/testing

Hello, devs, users, Rohit. Have a good day.

Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I see
risks here. A major risk is that 4.10 is too buggy and it seems nobody uses
it actually right now in production because it's unusable, unfortunately,
so we are planning to freeze 4.11 which stands on untested 4.10 with a lot
of lacks still undiscovered and not reported. I believe it's a very
dangerous way to release one more release with bad quality. Actually,
marvin and units don't cover regressions I meet in 4.10. Ok, let's take a
look at new one our engineers found today in 4.10:

It happens rarely for Domain Admin, probably other roles are affected. We
meet it once a week. It's for resource accounting. Domain admin resource
quota is 200GB of primary storage. During continouos creation of VMs and
removing them it leads to "Maximum number of resources of type
'primary_storage' for domain id=2 has been exceeded", however only 26 GB is
used actually.

I mean smoke tests run well, unit tests run well, but *nobody reported very
obvious bug* which should be met number of times if community use 4.10. I
suppose, there are a lot of undiscovered and unreported regressions in 4.10
and this is a huge risk to the 4.11 release quality and more we move that
way quality decreases. Unfortunately, I'm not able to propose a silver
bullet, but I suppose feature development speed should be decreased until
master is tested very thoroughly and might be 8 january is too early for
4.11 freeze.



2017-12-09 2:00 GMT+07:00 Wido den Hollander :

>
>
> On 12/08/2017 03:24 PM, Rohit Yadav wrote:
>
>> All,
>>
>> Given we've about a month left until 4.11/LTS freeze date of 8th Jan 2018
>> [1], I would like to call our community for active participation.
>>
>> Given a huge pile of open issues, please share on this thread a 'brief'
>> list of top bugs/issues that you would want to see fixed and are
>> applicable
>> to master branch, especially critical/blocker issues and regressions. I
>> would especially like to engage with the users of the community towards
>> this effort. I'll start weekly reporting of open issues by end of next
>> week.
>>
>> Developers - feel free to comment tagging Daan (@DaanHoogland), Paul
>> (@PaulAngus) and me (@rhtyd) in your pull requests if you're seeking
>> review
>> and testing (and merging) of your PR. I hope to work with you all with my
>> committer/contributor hat on.
>>
>> Finally, I look forward to all of your help and support towards the next
>> (LTS) release.
>>
>>
> Let's make it work!
>
> Thoughts, comments?
>>
>>
> With the holiday season coming up we might see things slow down a bit. But
> I think we already have enough PRs open for 4.11
>
> We should be able to get out a proper release again.
>
> Wido
>
>
> [1] http://markmail.org/message/mszlluye35acvn2j
>>
>> Regards,
>> Rohit Yadav
>> http://rohityadav.cloud | @rhtyd
>>
>>__?.o/   Apache CloudStack
>>   ()# The best of CloudStack is yet to come!
>> (___(_)   https://cloudstack.apache.org
>>
>>


--
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>

rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

RE: Call for participation: Issue triaging and PR review/testing

[Paul Angus]

That really depends on what the tests that you are think of are.

Marvin covers a range of tests relating to logical but mostly physical 
functions, like network creation and the functions that CloudStack can 
orchestrate within those networks, VM lifecycle as well as storage functions.
The only thing that Marvin can’t really check is the UI.

It’s downside is that the tests are written in Python, so non-developers will 
struggle to create them.

Rene’s use of ansible will provide an alternate framework to setup a scenario 
and ensure that everything behaves as expected, which non-developers could work 
with (personnally I’m pretty excited about that).


Kind regards,

Paul Angus


paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

From: Luis [mailto:lmartinez...@yahoo.com]
Sent: 14 December 2017 19:15
To: Paul Angus ; users@cloudstack.apache.org; Ivan 
Kudryavtsev 
Cc: dev 
Subject: RE: Call for participation: Issue triaging and PR review/testing

Manual testing, not sure if automated test could do an entire test
Sent from Yahoo Mail on 
Android<https://overview.mail.yahoo.com/mobile/?.src=Android>

On Thu, Dec 14, 2017 at 11:25 AM, Paul Angus
mailto:paul.an...@shapeblue.com>> wrote:
Hi Luis,

Can you explain what you mean please?  Do you mean people writing automated 
tests or manual testing of discrete features?



Kind regards,

Paul Angus

paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



-Original Message-
From: Luis 
[mailto:lmartinez...@yahoo.com.INVALID<mailto:lmartinez...@yahoo.com.INVALID>]
Sent: 14 December 2017 02:04
To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>; Ivan 
Kudryavtsev mailto:kudryavtsev...@bw-sw.com>>; 
users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
Cc: dev mailto:d...@cloudstack.apache.org>>
Subject: Re: Call for participation: Issue triaging and PR review/testing

Hi
What about creating a team for testing and create a check list of what to test 
and how. Besides the people that uses CS. This may increase the quality.
Just an idea.

Sent from Yahoo Mail on Android

  On Wed, Dec 13, 2017 at 10:21 AM, Ivan 
Kudryavtsevmailto:kudryavtsev...@bw-sw.com>> wrote:  
Hi, Paul. Thank you for your response. I just still feel that it's a very
risky approach to deliver a new release if community haven't adopted and tried 
a previous one because future unidentified regressions are multiplied to 
currently unidentified regressions. But, I see it's a trade and controversity 
here.

2017-12-13 21:46 GMT+07:00 Paul Angus 
mailto:paul.an...@shapeblue.com>>:

> Thanks Rene.
>
> @Ivan, I understand your concerns. But if 4.10 is unusable, then it
> will never get much production testing.
> The longer between releases, the harder testing and triage becomes.
>
> By putting a line in the sand for 4.11 and 4.12, and with the desire
>to  keep making every release better than the last we can keep moving forward.
>  I think we're all largely in agreement that the process around 4.10
>was  sub-optimal, which is why we've set out clear guidelines that we'd
>like to  work to.
>
> You are correct that there is more to quality than just Marvin tests
> (or at least the current ones), and long term, if community members
> like yourselves and Rene, come up with tests/test structures that push
> the boundaries of CloudStack, then automated testing will only get better.
>
> For now though, I would suggest that the best way to galvanise the
> community around the manual testing of CloudStack is to have a release
> candidate that everyone can coalesce around.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>
>
>
>
> -Original Message-
> From: Rene Moser [mailto:m...@renemoser.net<mailto:m...@renemoser.net>]
> Sent: 13 December 2017 12:56
> To: dev mailto:d...@cloudstack.apache.org>>; 
> users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>
> Subject: Re: Call for participation: Issue triaging and PR review/testing
>
> Hi all
>
> On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> > Hello, devs, users, Rohit. Have a good day.
> >
> > Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I
> > see risks here. A major risk is that 4.10 is too buggy and it seems
> > nobody uses it actually right now in production because it's unusable,
> > unfortunately, so we are planning to freeze 4.11 which stands on
> > unt

Re: Call for participation: Issue triaging and PR review/testing

[Ron Wheeler]

Are there scripts for manual testing?

If there are scripts, these could certainly be done by 
non-developers/sysadmins as long as they have a test bed to use.


The scripts certainly would be a "good thing" to have for acceptance 
testing for anyone planning to put Cloudstack onto production for 
themselves or a client.


What is the minimal hardware configuration required to test the UI and 
user level functionality?
Do we have instructions to create a minimal test station? Is it more 
than two old desktops and a hub?
Need additional hardware to test specific routers and networks but 
shouldn't organizations wanting to put a new version into prosuction 
already have test/spare equipment.


What are the key UI/system features that are best tested by humans with 
a script
 - Clearly documentation and installation instructions are high on this 
list.


Ron

On 14/12/2017 2:15 PM, Luis wrote:

Manual testing, not sure if automated test could do an entire test

Sent from Yahoo Mail on Android
  
   On Thu, Dec 14, 2017 at 11:25 AM, Paul Angus wrote:   Hi Luis,


Can you explain what you mean please?  Do you mean people writing automated 
tests or manual testing of discrete features?



Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
   
  



-Original Message-
From: Luis [mailto:lmartinez...@yahoo.com.INVALID]
Sent: 14 December 2017 02:04
To: users@cloudstack.apache.org; Ivan Kudryavtsev ; 
users@cloudstack.apache.org
Cc: dev 
Subject: Re: Call for participation: Issue triaging and PR review/testing

Hi
What about creating a team for testing and create a check list of what to test 
and how. Besides the people that uses CS. This may increase the quality.
Just an idea.

Sent from Yahoo Mail on Android
  
   On Wed, Dec 13, 2017 at 10:21 AM, Ivan Kudryavtsev wrote:  Hi, Paul. Thank you for your response. I just still feel that it's a very

risky approach to deliver a new release if community haven't adopted and tried 
a previous one because future unidentified regressions are multiplied to 
currently unidentified regressions. But, I see it's a trade and controversity 
here.

2017-12-13 21:46 GMT+07:00 Paul Angus :


Thanks Rene.

@Ivan, I understand your concerns. But if 4.10 is unusable, then it
will never get much production testing.
The longer between releases, the harder testing and triage becomes.

By putting a line in the sand for 4.11 and 4.12, and with the desire
to  keep making every release better than the last we can keep moving forward.
   I think we're all largely in agreement that the process around 4.10
was  sub-optimal, which is why we've set out clear guidelines that we'd
like to  work to.

You are correct that there is more to quality than just Marvin tests
(or at least the current ones), and long term, if community members
like yourselves and Rene, come up with tests/test structures that push
the boundaries of CloudStack, then automated testing will only get better.

For now though, I would suggest that the best way to galvanise the
community around the manual testing of CloudStack is to have a release
candidate that everyone can coalesce around.



Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue




-Original Message-
From: Rene Moser [mailto:m...@renemoser.net]
Sent: 13 December 2017 12:56
To: dev ; users@cloudstack.apache.org
Subject: Re: Call for participation: Issue triaging and PR review/testing

Hi all

On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:

Hello, devs, users, Rohit. Have a good day.

Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I
see risks here. A major risk is that 4.10 is too buggy and it seems
nobody uses it actually right now in production because it's unusable,
unfortunately, so we are planning to freeze 4.11 which stands on
untested 4.10 with a lot of lacks still undiscovered and not reported.
I believe it's a very dangerous way to release one more release with
bad quality. Actually, marvin and units don't cover regressions I meet
in 4.10. Ok, let's take a look at new one our engineers found today in

4.10:

So, the point is, how do we (users, devs, all) improve quality?

Marvin is great for smoke testing but CloudStack is dealing with many
infra vendor components, which are not covered by the tests. How can we
detect flows not covered by marvin?

For me, I decided (independent of this discussion) to write integration
tests in a way one would not expect, not following the "happy path":

Try to break CloudStack, to make a better CloudStack.

Put a chaos monkey in your test infra: Shut down storage, kill a host, put
latency on storage, disable network on hosts, make load on a host.
read only fs on a cluster wide primary fs. shut down a VR, remove a VR.

Things that can happen!

Not sur

RE: Call for participation: Issue triaging and PR review/testing

[Luis]

Manual testing, not sure if automated test could do an entire test

Sent from Yahoo Mail on Android 
 
  On Thu, Dec 14, 2017 at 11:25 AM, Paul Angus wrote: 
  Hi Luis,

Can you explain what you mean please?  Do you mean people writing automated 
tests or manual testing of discrete features?



Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Luis [mailto:lmartinez...@yahoo.com.INVALID] 
Sent: 14 December 2017 02:04
To: users@cloudstack.apache.org; Ivan Kudryavtsev ; 
users@cloudstack.apache.org
Cc: dev 
Subject: Re: Call for participation: Issue triaging and PR review/testing

Hi
What about creating a team for testing and create a check list of what to test 
and how. Besides the people that uses CS. This may increase the quality.
Just an idea.

Sent from Yahoo Mail on Android 
 
  On Wed, Dec 13, 2017 at 10:21 AM, Ivan Kudryavtsev 
wrote:  Hi, Paul. Thank you for your response. I just still feel that it's a 
very
risky approach to deliver a new release if community haven't adopted and tried 
a previous one because future unidentified regressions are multiplied to 
currently unidentified regressions. But, I see it's a trade and controversity 
here.

2017-12-13 21:46 GMT+07:00 Paul Angus :

> Thanks Rene.
>
> @Ivan, I understand your concerns. But if 4.10 is unusable, then it 
> will never get much production testing.
> The longer between releases, the harder testing and triage becomes.
>
> By putting a line in the sand for 4.11 and 4.12, and with the desire 
>to  keep making every release better than the last we can keep moving forward.
>  I think we're all largely in agreement that the process around 4.10 
>was  sub-optimal, which is why we've set out clear guidelines that we'd 
>like to  work to.
>
> You are correct that there is more to quality than just Marvin tests 
> (or at least the current ones), and long term, if community members 
> like yourselves and Rene, come up with tests/test structures that push 
> the boundaries of CloudStack, then automated testing will only get better.
>
> For now though, I would suggest that the best way to galvanise the 
> community around the manual testing of CloudStack is to have a release 
> candidate that everyone can coalesce around.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>
>
>
>
> -Original Message-
> From: Rene Moser [mailto:m...@renemoser.net]
> Sent: 13 December 2017 12:56
> To: dev ; users@cloudstack.apache.org
> Subject: Re: Call for participation: Issue triaging and PR review/testing
>
> Hi all
>
> On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> > Hello, devs, users, Rohit. Have a good day.
> >
> > Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I
> > see risks here. A major risk is that 4.10 is too buggy and it seems
> > nobody uses it actually right now in production because it's unusable,
> > unfortunately, so we are planning to freeze 4.11 which stands on
> > untested 4.10 with a lot of lacks still undiscovered and not reported.
> > I believe it's a very dangerous way to release one more release with
> > bad quality. Actually, marvin and units don't cover regressions I meet
> > in 4.10. Ok, let's take a look at new one our engineers found today in
> 4.10:
>
> So, the point is, how do we (users, devs, all) improve quality?
>
> Marvin is great for smoke testing but CloudStack is dealing with many
> infra vendor components, which are not covered by the tests. How can we
> detect flows not covered by marvin?
>
> For me, I decided (independent of this discussion) to write integration
> tests in a way one would not expect, not following the "happy path":
>
> Try to break CloudStack, to make a better CloudStack.
>
> Put a chaos monkey in your test infra: Shut down storage, kill a host, put
> latency on storage, disable network on hosts, make load on a host.
> read only fs on a cluster wide primary fs. shut down a VR, remove a VR.
>
> Things that can happen!
>
> Not surprisingly I use Ansible. It has an extensive amount of modules
> which can be used to battle prove anything of your infra. Ansible playbooks
> are fairly easy to write, even when you are not used to write code.
>
> I will share my works when ready.
>
> René
>
>
>
>
>
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>  
  

RE: Call for participation: Issue triaging and PR review/testing

[Paul Angus]

Hi Luis,

Can you explain what you mean please?   Do you mean people writing automated 
tests or manual testing of discrete features?



Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Luis [mailto:lmartinez...@yahoo.com.INVALID] 
Sent: 14 December 2017 02:04
To: users@cloudstack.apache.org; Ivan Kudryavtsev ; 
users@cloudstack.apache.org
Cc: dev 
Subject: Re: Call for participation: Issue triaging and PR review/testing

Hi
What about creating a team for testing and create a check list of what to test 
and how. Besides the people that uses CS. This may increase the quality.
Just an idea.

Sent from Yahoo Mail on Android 
 
  On Wed, Dec 13, 2017 at 10:21 AM, Ivan Kudryavtsev 
wrote:   Hi, Paul. Thank you for your response. I just still feel that it's a 
very
risky approach to deliver a new release if community haven't adopted and tried 
a previous one because future unidentified regressions are multiplied to 
currently unidentified regressions. But, I see it's a trade and controversity 
here.

2017-12-13 21:46 GMT+07:00 Paul Angus :

> Thanks Rene.
>
> @Ivan, I understand your concerns. But if 4.10 is unusable, then it 
> will never get much production testing.
> The longer between releases, the harder testing and triage becomes.
>
> By putting a line in the sand for 4.11 and 4.12, and with the desire 
>to  keep making every release better than the last we can keep moving forward.
>  I think we're all largely in agreement that the process around 4.10 
>was  sub-optimal, which is why we've set out clear guidelines that we'd 
>like to  work to.
>
> You are correct that there is more to quality than just Marvin tests 
> (or at least the current ones), and long term, if community members 
> like yourselves and Rene, come up with tests/test structures that push 
> the boundaries of CloudStack, then automated testing will only get better.
>
> For now though, I would suggest that the best way to galvanise the 
> community around the manual testing of CloudStack is to have a release 
> candidate that everyone can coalesce around.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>
>
>
>
> -Original Message-
> From: Rene Moser [mailto:m...@renemoser.net]
> Sent: 13 December 2017 12:56
> To: dev ; users@cloudstack.apache.org
> Subject: Re: Call for participation: Issue triaging and PR review/testing
>
> Hi all
>
> On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> > Hello, devs, users, Rohit. Have a good day.
> >
> > Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I
> > see risks here. A major risk is that 4.10 is too buggy and it seems
> > nobody uses it actually right now in production because it's unusable,
> > unfortunately, so we are planning to freeze 4.11 which stands on
> > untested 4.10 with a lot of lacks still undiscovered and not reported.
> > I believe it's a very dangerous way to release one more release with
> > bad quality. Actually, marvin and units don't cover regressions I meet
> > in 4.10. Ok, let's take a look at new one our engineers found today in
> 4.10:
>
> So, the point is, how do we (users, devs, all) improve quality?
>
> Marvin is great for smoke testing but CloudStack is dealing with many
> infra vendor components, which are not covered by the tests. How can we
> detect flows not covered by marvin?
>
> For me, I decided (independent of this discussion) to write integration
> tests in a way one would not expect, not following the "happy path":
>
> Try to break CloudStack, to make a better CloudStack.
>
> Put a chaos monkey in your test infra: Shut down storage, kill a host, put
> latency on storage, disable network on hosts, make load on a host.
> read only fs on a cluster wide primary fs. shut down a VR, remove a VR.
>
> Things that can happen!
>
> Not surprisingly I use Ansible. It has an extensive amount of modules
> which can be used to battle prove anything of your infra. Ansible playbooks
> are fairly easy to write, even when you are not used to write code.
>
> I will share my works when ready.
>
> René
>
>
>
>
>
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>  

Re: Call for participation: Issue triaging and PR review/testing

[Luis]

Hi
What about creating a team for testing and create a check list of what to test 
and how. Besides the people that uses CS. This may increase the quality.
Just an idea.

Sent from Yahoo Mail on Android 
 
  On Wed, Dec 13, 2017 at 10:21 AM, Ivan Kudryavtsev 
wrote:   Hi, Paul. Thank you for your response. I just still feel that it's a 
very
risky approach to deliver a new release if community haven't adopted and
tried a previous one because future unidentified regressions are multiplied
to currently unidentified regressions. But, I see it's a trade and
controversity here.

2017-12-13 21:46 GMT+07:00 Paul Angus :

> Thanks Rene.
>
> @Ivan, I understand your concerns. But if 4.10 is unusable, then it will
> never get much production testing.
> The longer between releases, the harder testing and triage becomes.
>
> By putting a line in the sand for 4.11 and 4.12, and with the desire to
> keep making every release better than the last we can keep moving forward.
>  I think we're all largely in agreement that the process around 4.10 was
> sub-optimal, which is why we've set out clear guidelines that we'd like to
> work to.
>
> You are correct that there is more to quality than just Marvin tests (or
> at least the current ones), and long term, if community members like
> yourselves and Rene, come up with tests/test structures that push the
> boundaries of CloudStack, then automated testing will only get better.
>
> For now though, I would suggest that the best way to galvanise the
> community around the manual testing of CloudStack is to have a release
> candidate that everyone can coalesce around.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Rene Moser [mailto:m...@renemoser.net]
> Sent: 13 December 2017 12:56
> To: dev ; users@cloudstack.apache.org
> Subject: Re: Call for participation: Issue triaging and PR review/testing
>
> Hi all
>
> On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> > Hello, devs, users, Rohit. Have a good day.
> >
> > Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I
> > see risks here. A major risk is that 4.10 is too buggy and it seems
> > nobody uses it actually right now in production because it's unusable,
> > unfortunately, so we are planning to freeze 4.11 which stands on
> > untested 4.10 with a lot of lacks still undiscovered and not reported.
> > I believe it's a very dangerous way to release one more release with
> > bad quality. Actually, marvin and units don't cover regressions I meet
> > in 4.10. Ok, let's take a look at new one our engineers found today in
> 4.10:
>
> So, the point is, how do we (users, devs, all) improve quality?
>
> Marvin is great for smoke testing but CloudStack is dealing with many
> infra vendor components, which are not covered by the tests. How can we
> detect flows not covered by marvin?
>
> For me, I decided (independent of this discussion) to write integration
> tests in a way one would not expect, not following the "happy path":
>
> Try to break CloudStack, to make a better CloudStack.
>
> Put a chaos monkey in your test infra: Shut down storage, kill a host, put
> latency on storage, disable network on hosts, make load on a host.
> read only fs on a cluster wide primary fs. shut down a VR, remove a VR.
>
> Things that can happen!
>
> Not surprisingly I use Ansible. It has an extensive amount of modules
> which can be used to battle prove anything of your infra. Ansible playbooks
> are fairly easy to write, even when you are not used to write code.
>
> I will share my works when ready.
>
> René
>
>
>
>
>
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>  

Re: Call for participation: Issue triaging and PR review/testing

[Ivan Kudryavtsev]

Hi, Paul. Thank you for your response. I just still feel that it's a very
risky approach to deliver a new release if community haven't adopted and
tried a previous one because future unidentified regressions are multiplied
to currently unidentified regressions. But, I see it's a trade and
controversity here.

2017-12-13 21:46 GMT+07:00 Paul Angus :

> Thanks Rene.
>
> @Ivan, I understand your concerns. But if 4.10 is unusable, then it will
> never get much production testing.
> The longer between releases, the harder testing and triage becomes.
>
> By putting a line in the sand for 4.11 and 4.12, and with the desire to
> keep making every release better than the last we can keep moving forward.
>  I think we're all largely in agreement that the process around 4.10 was
> sub-optimal, which is why we've set out clear guidelines that we'd like to
> work to.
>
> You are correct that there is more to quality than just Marvin tests (or
> at least the current ones), and long term, if community members like
> yourselves and Rene, come up with tests/test structures that push the
> boundaries of CloudStack, then automated testing will only get better.
>
> For now though, I would suggest that the best way to galvanise the
> community around the manual testing of CloudStack is to have a release
> candidate that everyone can coalesce around.
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Rene Moser [mailto:m...@renemoser.net]
> Sent: 13 December 2017 12:56
> To: dev ; users@cloudstack.apache.org
> Subject: Re: Call for participation: Issue triaging and PR review/testing
>
> Hi all
>
> On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> > Hello, devs, users, Rohit. Have a good day.
> >
> > Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I
> > see risks here. A major risk is that 4.10 is too buggy and it seems
> > nobody uses it actually right now in production because it's unusable,
> > unfortunately, so we are planning to freeze 4.11 which stands on
> > untested 4.10 with a lot of lacks still undiscovered and not reported.
> > I believe it's a very dangerous way to release one more release with
> > bad quality. Actually, marvin and units don't cover regressions I meet
> > in 4.10. Ok, let's take a look at new one our engineers found today in
> 4.10:
>
> So, the point is, how do we (users, devs, all) improve quality?
>
> Marvin is great for smoke testing but CloudStack is dealing with many
> infra vendor components, which are not covered by the tests. How can we
> detect flows not covered by marvin?
>
> For me, I decided (independent of this discussion) to write integration
> tests in a way one would not expect, not following the "happy path":
>
> Try to break CloudStack, to make a better CloudStack.
>
> Put a chaos monkey in your test infra: Shut down storage, kill a host, put
> latency on storage, disable network on hosts, make load on a host.
> read only fs on a cluster wide primary fs. shut down a VR, remove a VR.
>
> Things that can happen!
>
> Not surprisingly I use Ansible. It has an extensive amount of modules
> which can be used to battle prove anything of your infra. Ansible playbooks
> are fairly easy to write, even when you are not used to write code.
>
> I will share my works when ready.
>
> René
>
>
>
>
>
>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>

Re: Call for participation: Issue triaging and PR review/testing

[Ivan Kudryavtsev]

Hi, Rene.

That sounds great, chaos engineering and controlled experiments are great
tools, smoke tests improvements are also subject of discussion and I think
that UAT (user acceptance tests) suite should be designed finally. It's
important to a business to have one. It's important to the community to
have one, otherwise quality can not be expected. I think the policy should
be developed which forces and motivates movement toward the direction
observed.

13 дек. 2017 г. 19:59 пользователь "Rene Moser" 
написал:

> Hi all
>
> On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> > Hello, devs, users, Rohit. Have a good day.
> >
> > Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I
> see
> > risks here. A major risk is that 4.10 is too buggy and it seems nobody
> uses
> > it actually right now in production because it's unusable, unfortunately,
> > so we are planning to freeze 4.11 which stands on untested 4.10 with a
> lot
> > of lacks still undiscovered and not reported. I believe it's a very
> > dangerous way to release one more release with bad quality. Actually,
> > marvin and units don't cover regressions I meet in 4.10. Ok, let's take a
> > look at new one our engineers found today in 4.10:
>
> So, the point is, how do we (users, devs, all) improve quality?
>
> Marvin is great for smoke testing but CloudStack is dealing with many
> infra vendor components, which are not covered by the tests. How can we
> detect flows not covered by marvin?
>
> For me, I decided (independent of this discussion) to write integration
> tests in a way one would not expect, not following the "happy path":
>
> Try to break CloudStack, to make a better CloudStack.
>
> Put a chaos monkey in your test infra: Shut down storage, kill a host,
> put latency on storage, disable network on hosts, make load on a host.
> read only fs on a cluster wide primary fs. shut down a VR, remove a VR.
>
> Things that can happen!
>
> Not surprisingly I use Ansible. It has an extensive amount of modules
> which can be used to battle prove anything of your infra. Ansible
> playbooks are fairly easy to write, even when you are not used to write
> code.
>
> I will share my works when ready.
>
> René
>
>
>
>
>
>

RE: Call for participation: Issue triaging and PR review/testing

[Paul Angus]

Thanks Rene.

@Ivan, I understand your concerns. But if 4.10 is unusable, then it will never 
get much production testing.
The longer between releases, the harder testing and triage becomes.

By putting a line in the sand for 4.11 and 4.12, and with the desire to keep 
making every release better than the last we can keep moving forward.   I think 
we're all largely in agreement that the process around 4.10 was sub-optimal, 
which is why we've set out clear guidelines that we'd like to work to.

You are correct that there is more to quality than just Marvin tests (or at 
least the current ones), and long term, if community members like yourselves 
and Rene, come up with tests/test structures that push the boundaries of 
CloudStack, then automated testing will only get better.

For now though, I would suggest that the best way to galvanise the community 
around the manual testing of CloudStack is to have a release candidate that 
everyone can coalesce around.



Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Rene Moser [mailto:m...@renemoser.net] 
Sent: 13 December 2017 12:56
To: dev ; users@cloudstack.apache.org
Subject: Re: Call for participation: Issue triaging and PR review/testing

Hi all

On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> Hello, devs, users, Rohit. Have a good day.
> 
> Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I 
> see risks here. A major risk is that 4.10 is too buggy and it seems 
> nobody uses it actually right now in production because it's unusable, 
> unfortunately, so we are planning to freeze 4.11 which stands on 
> untested 4.10 with a lot of lacks still undiscovered and not reported. 
> I believe it's a very dangerous way to release one more release with 
> bad quality. Actually, marvin and units don't cover regressions I meet 
> in 4.10. Ok, let's take a look at new one our engineers found today in 4.10:

So, the point is, how do we (users, devs, all) improve quality?

Marvin is great for smoke testing but CloudStack is dealing with many infra 
vendor components, which are not covered by the tests. How can we detect flows 
not covered by marvin?

For me, I decided (independent of this discussion) to write integration tests 
in a way one would not expect, not following the "happy path":

Try to break CloudStack, to make a better CloudStack.

Put a chaos monkey in your test infra: Shut down storage, kill a host, put 
latency on storage, disable network on hosts, make load on a host.
read only fs on a cluster wide primary fs. shut down a VR, remove a VR.

Things that can happen!

Not surprisingly I use Ansible. It has an extensive amount of modules which can 
be used to battle prove anything of your infra. Ansible playbooks are fairly 
easy to write, even when you are not used to write code.

I will share my works when ready.

René

Re: Call for participation: Issue triaging and PR review/testing

[Rene Moser]

Hi all

On 12/13/2017 05:04 AM, Ivan Kudryavtsev wrote:
> Hello, devs, users, Rohit. Have a good day.
> 
> Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I see
> risks here. A major risk is that 4.10 is too buggy and it seems nobody uses
> it actually right now in production because it's unusable, unfortunately,
> so we are planning to freeze 4.11 which stands on untested 4.10 with a lot
> of lacks still undiscovered and not reported. I believe it's a very
> dangerous way to release one more release with bad quality. Actually,
> marvin and units don't cover regressions I meet in 4.10. Ok, let's take a
> look at new one our engineers found today in 4.10:

So, the point is, how do we (users, devs, all) improve quality?

Marvin is great for smoke testing but CloudStack is dealing with many
infra vendor components, which are not covered by the tests. How can we
detect flows not covered by marvin?

For me, I decided (independent of this discussion) to write integration
tests in a way one would not expect, not following the "happy path":

Try to break CloudStack, to make a better CloudStack.

Put a chaos monkey in your test infra: Shut down storage, kill a host,
put latency on storage, disable network on hosts, make load on a host.
read only fs on a cluster wide primary fs. shut down a VR, remove a VR.

Things that can happen!

Not surprisingly I use Ansible. It has an extensive amount of modules
which can be used to battle prove anything of your infra. Ansible
playbooks are fairly easy to write, even when you are not used to write
code.

I will share my works when ready.

René

Re: Call for participation: Issue triaging and PR review/testing

[Ivan Kudryavtsev]

Hello, devs, users, Rohit. Have a good day.

Rohit, you intend to freeze 4.11 on 8 january and, frankly speaking, I see
risks here. A major risk is that 4.10 is too buggy and it seems nobody uses
it actually right now in production because it's unusable, unfortunately,
so we are planning to freeze 4.11 which stands on untested 4.10 with a lot
of lacks still undiscovered and not reported. I believe it's a very
dangerous way to release one more release with bad quality. Actually,
marvin and units don't cover regressions I meet in 4.10. Ok, let's take a
look at new one our engineers found today in 4.10:

It happens rarely for Domain Admin, probably other roles are affected. We
meet it once a week. It's for resource accounting. Domain admin resource
quota is 200GB of primary storage. During continouos creation of VMs and
removing them it leads to "Maximum number of resources of type
'primary_storage' for domain id=2 has been exceeded", however only 26 GB is
used actually.

I mean smoke tests run well, unit tests run well, but *nobody reported very
obvious bug* which should be met number of times if community use 4.10. I
suppose, there are a lot of undiscovered and unreported regressions in 4.10
and this is a huge risk to the 4.11 release quality and more we move that
way quality decreases. Unfortunately, I'm not able to propose a silver
bullet, but I suppose feature development speed should be decreased until
master is tested very thoroughly and might be 8 january is too early for
4.11 freeze.



2017-12-09 2:00 GMT+07:00 Wido den Hollander :

>
>
> On 12/08/2017 03:24 PM, Rohit Yadav wrote:
>
>> All,
>>
>> Given we've about a month left until 4.11/LTS freeze date of 8th Jan 2018
>> [1], I would like to call our community for active participation.
>>
>> Given a huge pile of open issues, please share on this thread a 'brief'
>> list of top bugs/issues that you would want to see fixed and are
>> applicable
>> to master branch, especially critical/blocker issues and regressions. I
>> would especially like to engage with the users of the community towards
>> this effort. I'll start weekly reporting of open issues by end of next
>> week.
>>
>> Developers - feel free to comment tagging Daan (@DaanHoogland), Paul
>> (@PaulAngus) and me (@rhtyd) in your pull requests if you're seeking
>> review
>> and testing (and merging) of your PR. I hope to work with you all with my
>> committer/contributor hat on.
>>
>> Finally, I look forward to all of your help and support towards the next
>> (LTS) release.
>>
>>
> Let's make it work!
>
> Thoughts, comments?
>>
>>
> With the holiday season coming up we might see things slow down a bit. But
> I think we already have enough PRs open for 4.11
>
> We should be able to get out a proper release again.
>
> Wido
>
>
> [1] http://markmail.org/message/mszlluye35acvn2j
>>
>> Regards,
>> Rohit Yadav
>> http://rohityadav.cloud | @rhtyd
>>
>>__?.o/   Apache CloudStack
>>   ()# The best of CloudStack is yet to come!
>> (___(_)   https://cloudstack.apache.org
>>
>>


-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ 

Re: Call for participation: Issue triaging and PR review/testing

[Wido den Hollander]

On 12/08/2017 03:24 PM, Rohit Yadav wrote:

All,

Given we've about a month left until 4.11/LTS freeze date of 8th Jan 2018
[1], I would like to call our community for active participation.

Given a huge pile of open issues, please share on this thread a 'brief'
list of top bugs/issues that you would want to see fixed and are applicable
to master branch, especially critical/blocker issues and regressions. I
would especially like to engage with the users of the community towards
this effort. I'll start weekly reporting of open issues by end of next week.

Developers - feel free to comment tagging Daan (@DaanHoogland), Paul
(@PaulAngus) and me (@rhtyd) in your pull requests if you're seeking review
and testing (and merging) of your PR. I hope to work with you all with my
committer/contributor hat on.

Finally, I look forward to all of your help and support towards the next
(LTS) release.



Let's make it work!


Thoughts, comments?



With the holiday season coming up we might see things slow down a bit. 
But I think we already have enough PRs open for 4.11


We should be able to get out a proper release again.

Wido


[1] http://markmail.org/message/mszlluye35acvn2j

Regards,
Rohit Yadav
http://rohityadav.cloud | @rhtyd

   __?.o/   Apache CloudStack
  ()# The best of CloudStack is yet to come!
(___(_)   https://cloudstack.apache.org

Call for participation: Issue triaging and PR review/testing

[Rohit Yadav]

All,

Given we've about a month left until 4.11/LTS freeze date of 8th Jan 2018
[1], I would like to call our community for active participation.

Given a huge pile of open issues, please share on this thread a 'brief'
list of top bugs/issues that you would want to see fixed and are applicable
to master branch, especially critical/blocker issues and regressions. I
would especially like to engage with the users of the community towards
this effort. I'll start weekly reporting of open issues by end of next week.

Developers - feel free to comment tagging Daan (@DaanHoogland), Paul
(@PaulAngus) and me (@rhtyd) in your pull requests if you're seeking review
and testing (and merging) of your PR. I hope to work with you all with my
committer/contributor hat on.

Finally, I look forward to all of your help and support towards the next
(LTS) release.

Thoughts, comments?

[1] http://markmail.org/message/mszlluye35acvn2j

Regards,
Rohit Yadav
http://rohityadav.cloud | @rhtyd

  __?.o/   Apache CloudStack
 ()# The best of CloudStack is yet to come!
(___(_)   https://cloudstack.apache.org