Re: [DISCUSSION] Moving NimBLE to separate project

[Andrzej Kaczmarek]

Hi Szymon,

On Thu, Mar 15, 2018 at 12:57 PM, Szymon Janc  wrote:
> Hi Andrzej,
>
> On Monday, 12 March 2018 15:30:52 CET Andrzej Kaczmarek wrote:
>> Hi,
>>
>> I created PR for apache-mynewt-core which removes NimBLE code and adds
>> dependency to apache-mynewy-nimble repository:
>> https://github.com/apache/mynewt-core/pull/907
>>
>> With latest changes in newt (thanks, Chris!) transition to new
>> structure is automatic - newt update will download new repository and
>> newt build will pick up all dependencies.
>>
>> apache-mynewt-nimble repository is basically rewritten history of
>> apache-mynewt-core with only commits touching NimBLE code included +
>> some directory structure reorg. Each new commit has id of original
>> commit added in commit message to make lookups easier.
>
> Since it looks like community agree with this proposal lets get this PR merged
> so that it can get some testing before next release.

All right, I'll update PR and apache-mynewt-nimble repository to
latest NimBLE code today evening (let's say not before 8pm CET / 12pm
PST) and so we can merge it.
If anyone disagrees, you still have few hours to let us know :-)

> --
> pozdrawiam
> Szymon Janc

Best regards,
Andrzej

Re: [DISCUSSION] Moving NimBLE to separate project

[Christopher Collins]

On Tue, Feb 27, 2018 at 08:39:04AM -0800, Christopher Collins wrote:
> I agree with others that the best option is 2a
> (@apache-mynewt-core/net/nimble* become empty packages that pull in the
> external nimble packages).  However, newt doesn't currently support repo
> dependencies; if a repo is not specified in the user's `project.yml`
> file, then newt won't download it.
[...]

I have submitted a PR which fixes repo dependencies:
https://github.com/apache/mynewt-newt/pull/140.  

This PR changes the `repository.deps` format to the one I used in my
previous email.  I thought this was a good idea because it eliminates
the need to repeat a depended-on repo's git information for each
version.

> 
> I've been looking at repo dependencies in newt, and it appears to be a
> pretty complex feature.  I don't think we can expect to have this
> working for 1.4.0, but maybe there are some compromises we can make to
> implement a simplified model that solves the nimble case.  I'll explain
> why I think this is complicated to solve below.  If you aren't
> interested in newt particulars, feel free to stop reading now :).
> 
> Here is how apache-mynewt-core's `repository.yml` file might look like
> this after the nimble and nffs dependencies are added (I've made some
> syntax changes for readability):
> 
> repository.deps:
> mynewt-nimble:
> type: git
> url: 'g...@github.com:apache/mynewt-nimble.git'
> vers:
> master: 0-dev
> mynewt_1_4_0_tag:   >=1.4.0
> 
> mynewt-nffs:
> type: git
> url: 'g...@github.com:apache/mynewt-nffs.git'
> vers:
> master: 0-dev
> mynewt_1_4_0_tag:   1.4.0
> 
> What this says is:
> * The master branch of apache-mynewt-core depends on:
> o nimble 0-dev
> o nffs 0-dev
> * The mynewt_1_4_0_tag tag of apache-mynewt-core depends on:
> o nimble 1.4.0 or later
> o nffs 1.4.0 only
> 
> The key points are: 
> * A repo's dependency tree may differ with version (branch).  
> * A dependency can specify a range of dependant repos (e.g., `>=`).
> 
> I think this is a good design, and I am not suggesting changing it.
> Some aspects of this are just quite complicated.  In particular, if the
> user wants to upgrade some repos, calculating the versions to upgrade to
> is a bit tricky.  Newt solves the similar problem of determining which
> packages to include during a build operation, but there is a fundamental
> difference: for packages, there is only a single dependency graph.  For
> repos, on the other hand, there are potentially many separate graphs,
> each of which needs to be evaluated to detect conflicts.  Furthermore,
> each graph may differ quite substantially, as some secondary dependency
> may add a tertiary dependency for certain versions (and so on and so
> on).  The number of potential graphs is quite large.  Consider a project
> with 10 repos, each of which specifies 10 versions in its
> `repository.yml` file.  Ignoring some details, there are potentially
> 10^10 (10 billion) graphs that would need to be evaluated to find the
> one combination that doesn't have any conflicts.  Furthermore, producing
> a helpful error message when there is no suitable combination of repo
> versions is quite a challenge in itself.
> 
> Linux distributions that use a rolling release model solve this problem,
> so it certainly is possible to implement.  However, it is going to
> require a lot of tricky code.
> 
> Chris
> 
> On Tue, Feb 27, 2018 at 09:44:37AM +0100, Szymon Janc wrote:
> > Hi,
> > 
> > 1. I'm fine with doing it either before or after 1.4. We just need to
> > make sure update works correctly for 1.4 if we do it before.
> > 2. I'd go with 2a as Will,  no need to keep duplicated code around.
> > 3. Agree with Will,  lets start with 1.0 NimBLE release. Only the
> > first release needs to be coordinated with -core anyway.
> > 
> > 
> > On 24 February 2018 at 01:07, will sanfilippo  wrote:
> > > My thoughts:
> > >
> > > 1. Since the 1.4 release is coming up very quickly I would do this for 
> > > 1.5 (personally).
> > > 2. I would choose 2a.
> > > 3. Seems a bit confusing to me to label nimBLE releases with same number 
> > > as Mynewt releases. Why not just make the first stable release of nimBLE 
> > > 1.0? Not a big deal either way but since they are going to diverge 
> > > eventually.
> > >
> > > Will
> > >> On Feb 22, 2018, at 1:01 AM, Andrzej Kaczmarek 
> > >>  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> As some of you may already noticed, there is apache/mynewt-nimble
> > >> repository created where NimBLE code was pushed along with some extra
> > >> changes, most notably initial attempt to create port of NimBLE for
> > >> FreeRTOS, but other platforms will be supported as well (native Linux 
> > >> port
> > >> is also prepared).
> > 

Re: [DISCUSSION] Moving NimBLE to separate project

[Christopher Collins]

I agree with others that the best option is 2a
(@apache-mynewt-core/net/nimble* become empty packages that pull in the
external nimble packages).  However, newt doesn't currently support repo
dependencies; if a repo is not specified in the user's `project.yml`
file, then newt won't download it.

I've been looking at repo dependencies in newt, and it appears to be a
pretty complex feature.  I don't think we can expect to have this
working for 1.4.0, but maybe there are some compromises we can make to
implement a simplified model that solves the nimble case.  I'll explain
why I think this is complicated to solve below.  If you aren't
interested in newt particulars, feel free to stop reading now :).

Here is how apache-mynewt-core's `repository.yml` file might look like
this after the nimble and nffs dependencies are added (I've made some
syntax changes for readability):

repository.deps:
mynewt-nimble:
type: git
url: 'g...@github.com:apache/mynewt-nimble.git'
vers:
master: 0-dev
mynewt_1_4_0_tag:   >=1.4.0

mynewt-nffs:
type: git
url: 'g...@github.com:apache/mynewt-nffs.git'
vers:
master: 0-dev
mynewt_1_4_0_tag:   1.4.0

What this says is:
* The master branch of apache-mynewt-core depends on:
o nimble 0-dev
o nffs 0-dev
* The mynewt_1_4_0_tag tag of apache-mynewt-core depends on:
o nimble 1.4.0 or later
o nffs 1.4.0 only

The key points are: 
* A repo's dependency tree may differ with version (branch).  
* A dependency can specify a range of dependant repos (e.g., `>=`).

I think this is a good design, and I am not suggesting changing it.
Some aspects of this are just quite complicated.  In particular, if the
user wants to upgrade some repos, calculating the versions to upgrade to
is a bit tricky.  Newt solves the similar problem of determining which
packages to include during a build operation, but there is a fundamental
difference: for packages, there is only a single dependency graph.  For
repos, on the other hand, there are potentially many separate graphs,
each of which needs to be evaluated to detect conflicts.  Furthermore,
each graph may differ quite substantially, as some secondary dependency
may add a tertiary dependency for certain versions (and so on and so
on).  The number of potential graphs is quite large.  Consider a project
with 10 repos, each of which specifies 10 versions in its
`repository.yml` file.  Ignoring some details, there are potentially
10^10 (10 billion) graphs that would need to be evaluated to find the
one combination that doesn't have any conflicts.  Furthermore, producing
a helpful error message when there is no suitable combination of repo
versions is quite a challenge in itself.

Linux distributions that use a rolling release model solve this problem,
so it certainly is possible to implement.  However, it is going to
require a lot of tricky code.

Chris

On Tue, Feb 27, 2018 at 09:44:37AM +0100, Szymon Janc wrote:
> Hi,
> 
> 1. I'm fine with doing it either before or after 1.4. We just need to
> make sure update works correctly for 1.4 if we do it before.
> 2. I'd go with 2a as Will,  no need to keep duplicated code around.
> 3. Agree with Will,  lets start with 1.0 NimBLE release. Only the
> first release needs to be coordinated with -core anyway.
> 
> 
> On 24 February 2018 at 01:07, will sanfilippo  wrote:
> > My thoughts:
> >
> > 1. Since the 1.4 release is coming up very quickly I would do this for 1.5 
> > (personally).
> > 2. I would choose 2a.
> > 3. Seems a bit confusing to me to label nimBLE releases with same number as 
> > Mynewt releases. Why not just make the first stable release of nimBLE 1.0? 
> > Not a big deal either way but since they are going to diverge eventually.
> >
> > Will
> >> On Feb 22, 2018, at 1:01 AM, Andrzej Kaczmarek 
> >>  wrote:
> >>
> >> Hi all,
> >>
> >> As some of you may already noticed, there is apache/mynewt-nimble
> >> repository created where NimBLE code was pushed along with some extra
> >> changes, most notably initial attempt to create port of NimBLE for
> >> FreeRTOS, but other platforms will be supported as well (native Linux port
> >> is also prepared).
> >>
> >> The problem is that this repo is now not synced with apache/mynewt-core and
> >> having two repositories with the same code is troublesome so we'd like to
> >> end development of NimBLE code in core repository and move it entirely to
> >> nimble repository. There are three open points on how this should be done:
> >> 1. When to do this switch? Before 1.4 release or after it?
> >> 2. How to deal with NimBLE in core repository?
> >> 3. How to manage NimBLE releases?
> >>
> >> My proposals are as follows:
> >>
> >> 2a. Remove NimBLE code from mynewt-core repository leaving only packages
> >> with 

Re: [DISCUSSION] Moving NimBLE to separate project

[Szymon Janc]

Hi,

1. I'm fine with doing it either before or after 1.4. We just need to
make sure update works correctly for 1.4 if we do it before.
2. I'd go with 2a as Will,  no need to keep duplicated code around.
3. Agree with Will,  lets start with 1.0 NimBLE release. Only the
first release needs to be coordinated with -core anyway.


On 24 February 2018 at 01:07, will sanfilippo  wrote:
> My thoughts:
>
> 1. Since the 1.4 release is coming up very quickly I would do this for 1.5 
> (personally).
> 2. I would choose 2a.
> 3. Seems a bit confusing to me to label nimBLE releases with same number as 
> Mynewt releases. Why not just make the first stable release of nimBLE 1.0? 
> Not a big deal either way but since they are going to diverge eventually.
>
> Will
>> On Feb 22, 2018, at 1:01 AM, Andrzej Kaczmarek 
>>  wrote:
>>
>> Hi all,
>>
>> As some of you may already noticed, there is apache/mynewt-nimble
>> repository created where NimBLE code was pushed along with some extra
>> changes, most notably initial attempt to create port of NimBLE for
>> FreeRTOS, but other platforms will be supported as well (native Linux port
>> is also prepared).
>>
>> The problem is that this repo is now not synced with apache/mynewt-core and
>> having two repositories with the same code is troublesome so we'd like to
>> end development of NimBLE code in core repository and move it entirely to
>> nimble repository. There are three open points on how this should be done:
>> 1. When to do this switch? Before 1.4 release or after it?
>> 2. How to deal with NimBLE in core repository?
>> 3. How to manage NimBLE releases?
>>
>> My proposals are as follows:
>>
>> 2a. Remove NimBLE code from mynewt-core repository leaving only packages
>> with dependencies to mynewt-nimble repository. The process of upgrading to
>> new version should be as easy as doing 'newt upgrade' to fetch newt
>> repository, assuming there are no local changes to NimBLE code. This is
>> preferred option.
>> 2b. Leave NimBLE code at its current state in mynewt-core and use it by
>> default for next release, with option to use mynewt-nimble instead. This is
>> safe option and can be also applied before 1.4 release.
>>
>> 3. NimBLE has its own releases, depending on needs, and Mynewt will use
>> latest stable release of NimBLE at all time. First release of NimBLE will
>> be synced with Mynewt release, I guess we can call it the same as Mynewt
>> release and then start independent releases with 2.0. For those who would
>> like to use latest NimBLE, of course it would be just a matter of switching
>> repository version manually.
>>
>> Any thoughts on this?
>>
>> Best regards,
>> Andrzej
>



-- 
pozdrawiam
Szymon K. Janc

Re: [DISCUSSION] Moving NimBLE to separate project

[will sanfilippo]

My thoughts:

1. Since the 1.4 release is coming up very quickly I would do this for 1.5 
(personally).
2. I would choose 2a.
3. Seems a bit confusing to me to label nimBLE releases with same number as 
Mynewt releases. Why not just make the first stable release of nimBLE 1.0? Not 
a big deal either way but since they are going to diverge eventually.

Will
> On Feb 22, 2018, at 1:01 AM, Andrzej Kaczmarek 
>  wrote:
> 
> Hi all,
> 
> As some of you may already noticed, there is apache/mynewt-nimble
> repository created where NimBLE code was pushed along with some extra
> changes, most notably initial attempt to create port of NimBLE for
> FreeRTOS, but other platforms will be supported as well (native Linux port
> is also prepared).
> 
> The problem is that this repo is now not synced with apache/mynewt-core and
> having two repositories with the same code is troublesome so we'd like to
> end development of NimBLE code in core repository and move it entirely to
> nimble repository. There are three open points on how this should be done:
> 1. When to do this switch? Before 1.4 release or after it?
> 2. How to deal with NimBLE in core repository?
> 3. How to manage NimBLE releases?
> 
> My proposals are as follows:
> 
> 2a. Remove NimBLE code from mynewt-core repository leaving only packages
> with dependencies to mynewt-nimble repository. The process of upgrading to
> new version should be as easy as doing 'newt upgrade' to fetch newt
> repository, assuming there are no local changes to NimBLE code. This is
> preferred option.
> 2b. Leave NimBLE code at its current state in mynewt-core and use it by
> default for next release, with option to use mynewt-nimble instead. This is
> safe option and can be also applied before 1.4 release.
> 
> 3. NimBLE has its own releases, depending on needs, and Mynewt will use
> latest stable release of NimBLE at all time. First release of NimBLE will
> be synced with Mynewt release, I guess we can call it the same as Mynewt
> release and then start independent releases with 2.0. For those who would
> like to use latest NimBLE, of course it would be just a matter of switching
> repository version manually.
> 
> Any thoughts on this?
> 
> Best regards,
> Andrzej

[DISCUSSION] Moving NimBLE to separate project

[Andrzej Kaczmarek]

Hi all,

As some of you may already noticed, there is apache/mynewt-nimble
repository created where NimBLE code was pushed along with some extra
changes, most notably initial attempt to create port of NimBLE for
FreeRTOS, but other platforms will be supported as well (native Linux port
is also prepared).

The problem is that this repo is now not synced with apache/mynewt-core and
having two repositories with the same code is troublesome so we'd like to
end development of NimBLE code in core repository and move it entirely to
nimble repository. There are three open points on how this should be done:
1. When to do this switch? Before 1.4 release or after it?
2. How to deal with NimBLE in core repository?
3. How to manage NimBLE releases?

My proposals are as follows:

2a. Remove NimBLE code from mynewt-core repository leaving only packages
with dependencies to mynewt-nimble repository. The process of upgrading to
new version should be as easy as doing 'newt upgrade' to fetch newt
repository, assuming there are no local changes to NimBLE code. This is
preferred option.
2b. Leave NimBLE code at its current state in mynewt-core and use it by
default for next release, with option to use mynewt-nimble instead. This is
safe option and can be also applied before 1.4 release.

3. NimBLE has its own releases, depending on needs, and Mynewt will use
latest stable release of NimBLE at all time. First release of NimBLE will
be synced with Mynewt release, I guess we can call it the same as Mynewt
release and then start independent releases with 2.0. For those who would
like to use latest NimBLE, of course it would be just a matter of switching
repository version manually.

Any thoughts on this?

Best regards,
Andrzej