Re: [asterisk-dev] Branching in the Testsuite

[Kevin Harwell]

On Mon, Jan 22, 2018 at 12:41 PM, Kevin Harwell  wrote:

> I'm getting ready to start the process of branching the testsuite. I'm
> going to give a basic overview here of what I'm going to do, so as things
> get posted on gerrit people will have some idea as to what is coming, and
> why.
>
> To start I'm not going to touch the current master branch. This will be
> renamed later to "legacy" once the new branches have all been created and
> readied.
>
> The first branch I'll create is the 13 branch. Once created I'm going to
> remove all tests that have their min version's marked as < 13.19.0 (current
> release of Asterisk). I'll then push this up for review. Once it's accepted
> I'll move onto the next step.
>
> Next I will create the 14 branch from the 13 branch. I'll then remove all
> tests from the 14 branch that have their min version's marked as < 14.7.5.
> I'll also remove all tests from the 13 branch that have a max version > 13.
> I'll then push both of these for review.
>
> Once accepted I'll then do similar for the 15 branch. It will be created
> from 14, and all tests will be removed that have a min version < 15.2.0.
> All tests with a max version > 14 will be removed from the 14 branch. Once
> these have been reviewed and accepted I will do the same for what will be
> the new master branch (probably temporarily call it that, "new_master",
> until we rename the old one to "legacy").
>
> Once all this part is done, or mostly done for a branch, I'll go through
> all the tests for each branch and remove the versioning stuff from the yaml
> files.
>
> At this point the branches should be in their final ready states. I'll
> update the Jenkins scripts to point to the new branches (13, 14, 15).  If
> everything seems good to go I'll rename the current master to "legacy", and
> rename "new_master" to "master". I'll also I'll create a tag for each
> branch based on the latest Asterisk version that's been released. At some
> point in this process I'll also update the mkrelease script to auto-create
> a new tag in the testsuite based on the version of Asterisk being released.
>
> I'm hoping by doing it this way it will minimize the size of the each
> review and help separate out the different types of changes.
>
> Thanks!
>
>
Actually, as the head of the testsuite branch is always moving toward the
next release I'm going to remove tests for each branch less than or equal
to the current released version of Asterisk for a branch. That'll make the
new branches starting point align with the following Asterisk versions:

13.20.0
14.7.6
15.3.0

To run any test against Asterisk prior to these versions one will need to
run them against the "legacy" branch.



Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev

Re: [asterisk-dev] Branching in the Testsuite

[Kevin Harwell]

I'm getting ready to start the process of branching the testsuite. I'm
going to give a basic overview here of what I'm going to do, so as things
get posted on gerrit people will have some idea as to what is coming, and
why.

To start I'm not going to touch the current master branch. This will be
renamed later to "legacy" once the new branches have all been created and
readied.

The first branch I'll create is the 13 branch. Once created I'm going to
remove all tests that have their min version's marked as < 13.19.0 (current
release of Asterisk). I'll then push this up for review. Once it's accepted
I'll move onto the next step.

Next I will create the 14 branch from the 13 branch. I'll then remove all
tests from the 14 branch that have their min version's marked as < 14.7.5.
I'll also remove all tests from the 13 branch that have a max version > 13.
I'll then push both of these for review.

Once accepted I'll then do similar for the 15 branch. It will be created
from 14, and all tests will be removed that have a min version < 15.2.0.
All tests with a max version > 14 will be removed from the 14 branch. Once
these have been reviewed and accepted I will do the same for what will be
the new master branch (probably temporarily call it that, "new_master",
until we rename the old one to "legacy").

Once all this part is done, or mostly done for a branch, I'll go through
all the tests for each branch and remove the versioning stuff from the yaml
files.

At this point the branches should be in their final ready states. I'll
update the Jenkins scripts to point to the new branches (13, 14, 15).  If
everything seems good to go I'll rename the current master to "legacy", and
rename "new_master" to "master". I'll also I'll create a tag for each
branch based on the latest Asterisk version that's been released. At some
point in this process I'll also update the mkrelease script to auto-create
a new tag in the testsuite based on the version of Asterisk being released.

I'm hoping by doing it this way it will minimize the size of the each
review and help separate out the different types of changes.

Thanks!
-- 

Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev

Re: [asterisk-dev] Branching in the Testsuite

[Kevin Harwell]

I went ahead and created an issue to track this:

https://issues.asterisk.org/jira/browse/ASTERISK-27492

Feel free to add comments or suggestions on the issue as well.

On Fri, Dec 15, 2017 at 10:59 AM, Kevin Harwell  wrote:

> Greetings,
>
> We're thinking about adding a branching system to the Asterisk Testsuite.
> Each branch would be named the same as, and correspond to, an Asterisk
> branch. So for instance the following branches would probably be created:
>
> 13, 14, 15
>
> For each release of Asterisk we will also create a tag in the Testsuite
> that corresponds to that release's tag. That way someone could checkout
> both tags for easy testing
>
> Other advantages? Most all, if not all, the current versioning stuff found
> in the Testsuite could go away, or be safely ignored moving forward. The
> versioning has become a bit cumbersome especially when you have to make a
> backward incompatible change to a test. Moving the version control out of
> the Testsuite and into a version control system should alleviate the need
> for this moving forward.
>
> Please let us know your thoughts and considerations on moving forward with
> this model. Especially any potential pitfalls or problems you might see
> with it.
>
> Thanks!
>
> --
>
> Kevin Harwell
> Digium, Inc. | Software Developer
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> Check us out at: http://digium.com & http://asterisk.org
>
>


-- 

Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at: http://digium.com & http://asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev

Re: [asterisk-dev] Branching in the Testsuite

[George Joseph]

On Mon, Dec 18, 2017 at 9:01 AM, Corey Farrell  wrote:

> Sounds good.  So after this is done am I correct that when my review
> against asterisk 13 is approved jenkins will 'gate' using the 13 branch of
> testsuite?
>
Correct.


> Will the testsuite also get minor branches (like 13.19)?
>
Not quite sure yet.  I'd say "no" until there's a good reason to create the
minor branches.  As it is, nothing would go into an asterisk minor branch
without going through the mainline branch first.  So for 13.19 for
instance, we'd just gate against 13.  Now when 13.19.0 is released, we'd
tag the testsuite 13 branch with "13.19.0" then when and if we need to
release 13.19.1, we'd test it with the "13.19.0" tag from the testsuite.

I think. :)





>
> On 12/15/2017 03:24 PM, George Joseph wrote:
>
>
>
> On Fri, Dec 15, 2017 at 10:56 AM, Corey Farrell  wrote:
>
>> It would be nice to strip out some/most of the per version conditionals
>> in tests.  The need to always cherry-pick changes is the only pitfall I see
>> with your proposal.  If a test never had per version differences then the
>> cherry-pick is trivial amount of extra work, but for tests with differences
>> per branch it would mean dealing with conflicts.  In the testsuite I think
>> I'd rather deal with per branch conditionals over merge with conflicts.
>>
> Given the low commit volume, even lower commit volume against existing
> tests, high compartmentalization and small nature of the files in the
> testsuite,  I don't see conflicts being a big thing.  If there are any,
> then they're probably justified.
>
>> I just want to propose an alternative way of (mostly) accomplishing the
>> goal.  We could create a tag '12' from current master.  This would
>> represent the last revision of the testsuite known to work with Asterisk 12
>> and below.  We would then be free to remove compatibility with EOL Asterisk
>> branches from testsuite master.  We would tag testsuite '14' soon after
>> September 26th, 2018 (EOL for Asterisk 14).  This would avoid multiplying
>> the number of gerrit reviews for testsuite changes, but it would require
>> continuing to maintain version declarations for supported versions of
>> Asterisk.  I think this would be a good trade-off so we aren't stuck with
>> all the 1.8/11 baggage.
>>
> We were thinking of taking the current master, copying it to 13, 14, 15,
> and "legacy" then starting with the next release our release scripts would
> automatically create tags in the testsuite just as they do in asterisk.
>
> This also makes Jenkins much happier BTW.
>
>
>
>
>>
>> On 12/15/2017 11:59 AM, Kevin Harwell wrote:
>>
>> Greetings,
>>
>> We're thinking about adding a branching system to the Asterisk Testsuite.
>> Each branch would be named the same as, and correspond to, an Asterisk
>> branch. So for instance the following branches would probably be created:
>>
>> 13, 14, 15
>>
>> For each release of Asterisk we will also create a tag in the Testsuite
>> that corresponds to that release's tag. That way someone could checkout
>> both tags for easy testing
>>
>> Other advantages? Most all, if not all, the current versioning stuff
>> found in the Testsuite could go away, or be safely ignored moving forward.
>> The versioning has become a bit cumbersome especially when you have to make
>> a backward incompatible change to a test. Moving the version control out of
>> the Testsuite and into a version control system should alleviate the need
>> for this moving forward.
>>
>> Please let us know your thoughts and considerations on moving forward
>> with this model. Especially any potential pitfalls or problems you might
>> see with it.
>>
>> Thanks!
>>
>> --
>>
>> Kevin Harwell
>> Digium, Inc. | Software Developer
>> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
>> Check us out at: http://digium.com & http://asterisk.org
>>
>>
>>
>>
>>
>> --
>> _
>> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>>
>> asterisk-dev mailing list
>> To UNSUBSCRIBE or update options visit:
>>http://lists.digium.com/mailman/listinfo/asterisk-dev
>>
>
>
>
> --
> George Joseph
> Digium, Inc. | Software Developer
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - US
> Check us out at: www.digium.com & www.asterisk.org
>
>
>
>
>
> --
> _
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>http://lists.digium.com/mailman/listinfo/asterisk-dev
>



-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or 

Re: [asterisk-dev] Branching in the Testsuite

[Corey Farrell]

Sounds good.  So after this is done am I correct that when my review 
against asterisk 13 is approved jenkins will 'gate' using the 13 branch 
of testsuite?  Will the testsuite also get minor branches (like 13.19)?



On 12/15/2017 03:24 PM, George Joseph wrote:



On Fri, Dec 15, 2017 at 10:56 AM, Corey Farrell > wrote:


It would be nice to strip out some/most of the per version
conditionals in tests.  The need to always cherry-pick changes is
the only pitfall I see with your proposal.  If a test never had
per version differences then the cherry-pick is trivial amount of
extra work, but for tests with differences per branch it would
mean dealing with conflicts.  In the testsuite I think I'd rather
deal with per branch conditionals over merge with conflicts.

Given the low commit volume, even lower commit volume against existing 
tests, high compartmentalization and small nature of the files in the 
testsuite,  I don't see conflicts being a big thing.  If there are 
any, then they're probably justified.


I just want to propose an alternative way of (mostly)
accomplishing the goal.  We could create a tag '12' from current
master.  This would represent the last revision of the testsuite
known to work with Asterisk 12 and below.  We would then be free
to remove compatibility with EOL Asterisk branches from testsuite
master.  We would tag testsuite '14' soon after September 26th,
2018 (EOL for Asterisk 14). This would avoid multiplying the
number of gerrit reviews for testsuite changes, but it would
require continuing to maintain version declarations for supported
versions of Asterisk.  I think this would be a good trade-off so
we aren't stuck with all the 1.8/11 baggage.

We were thinking of taking the current master, copying it to 13, 14, 
15, and "legacy" then starting with the next release our release 
scripts would automatically create tags in the testsuite just as they 
do in asterisk.


This also makes Jenkins much happier BTW.



On 12/15/2017 11:59 AM, Kevin Harwell wrote:

Greetings,

We're thinking about adding a branching system to the Asterisk
Testsuite. Each branch would be named the same as, and correspond
to, an Asterisk branch. So for instance the following branches
would probably be created:

13, 14, 15

For each release of Asterisk we will also create a tag in the
Testsuite that corresponds to that release's tag. That way
someone could checkout both tags for easy testing

Other advantages? Most all, if not all, the current versioning
stuff found in the Testsuite could go away, or be safely ignored
moving forward. The versioning has become a bit cumbersome
especially when you have to make a backward incompatible change
to a test. Moving the version control out of the Testsuite and
into a version control system should alleviate the need for this
moving forward.

Please let us know your thoughts and considerations on moving
forward with this model. Especially any potential pitfalls or
problems you might see with it.

Thanks!

-- 
Kevin Harwell

Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at:http://digium.com  &http://asterisk.org





--
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-dev





--
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com  & 
www.asterisk.org 






-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev

Re: [asterisk-dev] Branching in the Testsuite

[George Joseph]

On Fri, Dec 15, 2017 at 10:56 AM, Corey Farrell  wrote:

> It would be nice to strip out some/most of the per version conditionals in
> tests.  The need to always cherry-pick changes is the only pitfall I see
> with your proposal.  If a test never had per version differences then the
> cherry-pick is trivial amount of extra work, but for tests with differences
> per branch it would mean dealing with conflicts.  In the testsuite I think
> I'd rather deal with per branch conditionals over merge with conflicts.
>
Given the low commit volume, even lower commit volume against existing
tests, high compartmentalization and small nature of the files in the
testsuite,  I don't see conflicts being a big thing.  If there are any,
then they're probably justified.

> I just want to propose an alternative way of (mostly) accomplishing the
> goal.  We could create a tag '12' from current master.  This would
> represent the last revision of the testsuite known to work with Asterisk 12
> and below.  We would then be free to remove compatibility with EOL Asterisk
> branches from testsuite master.  We would tag testsuite '14' soon after
> September 26th, 2018 (EOL for Asterisk 14).  This would avoid multiplying
> the number of gerrit reviews for testsuite changes, but it would require
> continuing to maintain version declarations for supported versions of
> Asterisk.  I think this would be a good trade-off so we aren't stuck with
> all the 1.8/11 baggage.
>
We were thinking of taking the current master, copying it to 13, 14, 15,
and "legacy" then starting with the next release our release scripts would
automatically create tags in the testsuite just as they do in asterisk.

This also makes Jenkins much happier BTW.




>
> On 12/15/2017 11:59 AM, Kevin Harwell wrote:
>
> Greetings,
>
> We're thinking about adding a branching system to the Asterisk Testsuite.
> Each branch would be named the same as, and correspond to, an Asterisk
> branch. So for instance the following branches would probably be created:
>
> 13, 14, 15
>
> For each release of Asterisk we will also create a tag in the Testsuite
> that corresponds to that release's tag. That way someone could checkout
> both tags for easy testing
>
> Other advantages? Most all, if not all, the current versioning stuff found
> in the Testsuite could go away, or be safely ignored moving forward. The
> versioning has become a bit cumbersome especially when you have to make a
> backward incompatible change to a test. Moving the version control out of
> the Testsuite and into a version control system should alleviate the need
> for this moving forward.
>
> Please let us know your thoughts and considerations on moving forward with
> this model. Especially any potential pitfalls or problems you might see
> with it.
>
> Thanks!
>
> --
>
> Kevin Harwell
> Digium, Inc. | Software Developer
> 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
> Check us out at: http://digium.com & http://asterisk.org
>
>
>
>
>
> --
> _
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
>
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>http://lists.digium.com/mailman/listinfo/asterisk-dev
>



-- 
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev

Re: [asterisk-dev] Branching in the Testsuite

[Corey Farrell]

It would be nice to strip out some/most of the per version conditionals 
in tests.  The need to always cherry-pick changes is the only pitfall I 
see with your proposal.  If a test never had per version differences 
then the cherry-pick is trivial amount of extra work, but for tests with 
differences per branch it would mean dealing with conflicts.  In the 
testsuite I think I'd rather deal with per branch conditionals over 
merge with conflicts.


I just want to propose an alternative way of (mostly) accomplishing the 
goal.  We could create a tag '12' from current master.  This would 
represent the last revision of the testsuite known to work with Asterisk 
12 and below.  We would then be free to remove compatibility with EOL 
Asterisk branches from testsuite master.  We would tag testsuite '14' 
soon after September 26th, 2018 (EOL for Asterisk 14).  This would avoid 
multiplying the number of gerrit reviews for testsuite changes, but it 
would require continuing to maintain version declarations for supported 
versions of Asterisk.  I think this would be a good trade-off so we 
aren't stuck with all the 1.8/11 baggage.



On 12/15/2017 11:59 AM, Kevin Harwell wrote:

Greetings,

We're thinking about adding a branching system to the Asterisk 
Testsuite. Each branch would be named the same as, and correspond to, 
an Asterisk branch. So for instance the following branches would 
probably be created:


13, 14, 15

For each release of Asterisk we will also create a tag in the 
Testsuite that corresponds to that release's tag. That way someone 
could checkout both tags for easy testing


Other advantages? Most all, if not all, the current versioning stuff 
found in the Testsuite could go away, or be safely ignored moving 
forward. The versioning has become a bit cumbersome especially when 
you have to make a backward incompatible change to a test. Moving the 
version control out of the Testsuite and into a version control system 
should alleviate the need for this moving forward.


Please let us know your thoughts and considerations on moving forward 
with this model. Especially any potential pitfalls or problems you 
might see with it.


Thanks!

--
Kevin Harwell
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - USA
Check us out at:http://digium.com  &http://asterisk.org




-- 
_
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-dev