Re: [GSoC] migration

[Umesh Singla]

Hi all,

Sorry, I got caught up in academics and college stuff. Continuing from
where I left:

1. I broke migrate action into two parts - `port migrate` followed by `port
restore --last` to achieve the functionality we wanted. Now, migrate
creates a snapshot, uninstalls the ports and upgrades the port command
using forced and nosync selfupdate.

2. Restore now has 3 variations - `port restore`, `port restore --last`,
`port restore --snapshot-id=$some_id`.
i. `port restore` asks the user to select a snapshot interactively to
restore. It deactivates the currently installed ports and restore the
selected snapshot.
ii. `port restore --last` restores the last snapshot.
iii. `port restore --snapshot-id=$some_id` restores the specified
snapshot. It also deactivates the current state.

3. I have tried adding error handling and documentation as much as possible
but always open to suggestions and improvements.

Keeping this aside, I still wonder what happens in case port command
doesn't work on an upgraded OS at all and user doesn't have any or recent
snapshot in the registry. We won't be able to create a snapshot in
migration step.

Regards,
Umesh Singla


On Sat, Sep 9, 2017 at 3:48 AM, Joshua Root  wrote:

> On 2017-9-9 08:10 , Rainer Müller wrote:
>
>> On 2017-09-08 23:33, Umesh Singla wrote:
>>
>>> When I run the `migrate` action with only one port (expat) installed, I
>>> get the following:
>>>
>>> $ sudo ./bin/port migrate
>>>
>>> Taking a snapshot of the current state...
>>>
>>> Done: Snapshot '8':'snapshot created for migration' created at
>>> 2017-09-08 21:09:21
>>>
>>> Migration will first uninstall all the installed ports, upgrade MacPorts
>>> and then reinstall them again. Would you like to continue? [Y/n]: Y
>>>
>>
>> I would like to repeat that upgrading MacPorts from the 'port migrate'
>> action is not possible. After a major upgrade of macOS we cannot
>> guarantee that the old version of MacPorts runs at all on this new OS
>> version. MacPorts has to be reinstalled for the new OS version in an
>> extra step before.
>>
>> The best you can do is printing instructions on where to download
>> MacPorts.
>>
>
> I think that doing a forced selfupdate is probably possible in most cases.
> You just can't do everything in one run.
>
> - Josh
>

Re: Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Mojca Miklavec]

On 4 September 2017 at 14:10, Ryan Schmidt wrote:
> On Sep 4, 2017, at 04:25, Mojca Miklavec wrote:
>>
>> Is there any chance that we could schedule one 30-(max 90) minute long
>> IRC meeting to finally settle this issue? At least from the
>> philosophical point of view; code itself should be much easier to
>> write once the decision is met.
>
> Perhaps, but why does this one decision have to be reached over IRC, when we 
> successfully use email for other discussions?

In my opinion it will take at least another few months (years?) to
reach any agreement (if at all) if we continue at the current pace. If
we agree on a short (but sufficiently long slot), the idea is to force
anyone to reach the agreement within the designated 90 minutes and
volunteer for / delegate the work that's needed to actually implement
the solution. Plus discuss which things will be implemented and which
ones not.

We could decide on any way to proceed. Some are more correct than the
others, but require maybe 100 times more work, others use compromises.
I would be glad if we could meet shortly in a not too distant future
and decide what would work best.

Mojca

Re: [GSoC] migration

[Joshua Root]

On 2017-9-9 08:10 , Rainer Müller wrote:

On 2017-09-08 23:33, Umesh Singla wrote:

When I run the `migrate` action with only one port (expat) installed, I
get the following:

$ sudo ./bin/port migrate

Taking a snapshot of the current state...

Done: Snapshot '8':'snapshot created for migration' created at
2017-09-08 21:09:21

Migration will first uninstall all the installed ports, upgrade MacPorts
and then reinstall them again. Would you like to continue? [Y/n]: Y


I would like to repeat that upgrading MacPorts from the 'port migrate'
action is not possible. After a major upgrade of macOS we cannot
guarantee that the old version of MacPorts runs at all on this new OS
version. MacPorts has to be reinstalled for the new OS version in an
extra step before.

The best you can do is printing instructions on where to download MacPorts.


I think that doing a forced selfupdate is probably possible in most 
cases. You just can't do everything in one run.


- Josh

Re: [GSoC] migration

[Rainer Müller]

On 2017-09-08 23:33, Umesh Singla wrote:
> When I run the `migrate` action with only one port (expat) installed, I
> get the following:
> 
> $ sudo ./bin/port migrate
> 
> Taking a snapshot of the current state...
> 
> Done: Snapshot '8':'snapshot created for migration' created at
> 2017-09-08 21:09:21
> 
> Migration will first uninstall all the installed ports, upgrade MacPorts
> and then reinstall them again. Would you like to continue? [Y/n]: Y

I would like to repeat that upgrading MacPorts from the 'port migrate'
action is not possible. After a major upgrade of macOS we cannot
guarantee that the old version of MacPorts runs at all on this new OS
version. MacPorts has to be reinstalled for the new OS version in an
extra step before.

The best you can do is printing instructions on where to download MacPorts.

Rainer

Re: [GSoC] migration

[Joshua Root]

On 2017-9-9 07:33 , Umesh Singla wrote:
Does it mean that new port command is not being used for the next step 
of installation?


That's exactly what it means. Overwriting the files on disk doesn't do a 
"hot swap" of base versions; if you just keep running, you're still in 
the old version.


- Josh

Re: [GSoC] migration

[Mojca Miklavec]

Hi,

On 2 September 2017 at 15:18, Umesh Singla wrote:
> On Thu, Jun 15, 2017 at 12:05 AM, Mojca Miklavec wrote:
>> On 14 June 2017 at 05:47, Umesh Singla wrote:
>> >
>> > Okay, since there's already a OS comparison made, which I think can be
>> > directly used here. But to clarify, just the OS check? a check on
>> > x86_64/ppc
>> > changes is also needed.
>>
>> And potentially cxx_stdlib?
>
> How can such a check be made possible? All we have at present is here, [0].
> It doesn't compare stored values and current platform values. Are you
> suggesting a manual check while migration?
>
> - Umesh
>
> [0]:
> https://github.com/macports/macports-base/blob/master/src/macports1.0/macports.tcl#L1050-L1057

Well, if we don't store that value yet, we probably should.

What I have in mind is a scenario of a user stuck at a particular OS
version who upgrades MacPorts and the new version of MP makes sure
that that variable is stored. Then the user decides to switch to
libc++. It would be awesome if this process was fully automated and
the ports would all be reinstalled automatically. (But don't ask me
for links to line numbers, I'm not familiar with the core code well
enough.)

Mojca

Re: [GSoC] migration

[Umesh Singla]

Hi all,

When I run the `migrate` action with only one port (expat) installed, I get
the following:

$ sudo ./bin/port migrate

Taking a snapshot of the current state...

Done: Snapshot '8':'snapshot created for migration' created at 2017-09-08
21:09:21

Migration will first uninstall all the installed ports, upgrade MacPorts
and then reinstall them again. Would you like to continue? [Y/n]: Y

Uninstalling all ports...

Uninstalling: expat

--->  Deactivating expat @2.2.4_0

--->  Cleaning expat

--->  Uninstalling expat @2.2.4_0

--->  Cleaning expat

Upgrading MacPorts...

--->  Updating MacPorts base sources using rsync

MacPorts base version 2.4.99 installed,

MacPorts base version 2.4.1 downloaded.

--->  MacPorts base is outdated, installing new version 2.4.1

Installing new MacPorts release in /GSoC17/macports-test as root:admin;
permissions 0755


Fetching ports to install...

Restoring the original state...

MacPorts Version: 2.4.99

Installing (and activating): expat

--->  Fetching distfiles for expat

--->  Verifying checksums for expat

--->  Extracting expat

--->  Configuring expat

--->  Building expat

--->  Staging expat into destroot

--->  Installing expat @2.2.4_0

--->  Activating expat @2.2.4_0

--->  Cleaning expat

$


Since I have an updated version of registry db (v1.203) as opposed to its
current version (v1.202), I have a doubt if the above should have proceeded
in the way it did. I was expecting it to break after MacPorts base gets
upgraded complaining something of the sort, "Version number in metadata
table is higher than expected".

Does it mean that new port command is not being used for the next step of
installation? Or it's fine because mportinit didn't get initialized again?

- Umesh

Re: Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Ryan Schmidt]

On Sep 4, 2017, at 04:25, Mojca Miklavec wrote:

> On 4 September 2017 at 10:47, Ryan Schmidt wrote:
>> On Sep 2, 2017, at 11:25, Rainer Müller wrote:
>> 
>>> * archivefetch does not yet respect cxx_stdlib in any way
>> 
>> Yeah that would need adjustment, based on how we decide to differentiate 
>> libc++ archives from libstdc++ archives.
> 
> Is there any chance that we could schedule one 30-(max 90) minute long
> IRC meeting to finally settle this issue? At least from the
> philosophical point of view; code itself should be much easier to
> write once the decision is met.
> 
> (If we start at 15 UTC or later, we coud hopefully cover all timezones
> from west USA to India, including even China in the evening.)

Perhaps, but why does this one decision have to be reached over IRC, when we 
successfully use email for other discussions?


> We've been (not)discussing this for nearly a few years by now. The
> infrastructure (buildbot) is nearly in place, it's probably just about
> "a few lines of code" (and well ... lots of bytes on our mirrors) that
> set us apart from reaching the goal of supporting older machines.

Re: Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Mojca Miklavec]

On 4 September 2017 at 10:47, Ryan Schmidt wrote:
> On Sep 2, 2017, at 11:25, Rainer Müller wrote:
>
>> * archivefetch does not yet respect cxx_stdlib in any way
>
> Yeah that would need adjustment, based on how we decide to differentiate 
> libc++ archives from libstdc++ archives.

Is there any chance that we could schedule one 30-(max 90) minute long
IRC meeting to finally settle this issue? At least from the
philosophical point of view; code itself should be much easier to
write once the decision is met.

(If we start at 15 UTC or later, we coud hopefully cover all timezones
from west USA to India, including even China in the evening.)

We've been (not)discussing this for nearly a few years by now. The
infrastructure (buildbot) is nearly in place, it's probably just about
"a few lines of code" (and well ... lots of bytes on our mirrors) that
set us apart from reaching the goal of supporting older machines.

Mojca

Re: Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Ryan Schmidt]

On Sep 2, 2017, at 11:25, Rainer Müller wrote:

> The following goes beyond your GSoC project, but I think it fits into this
> discussion. Back in May, I published an experiment on my GitHub repo that I 
> had
> lying around. This adds functionality to stores the cxx_stdlib option in
> registry, such that 'port outdated' will consider ports as outdated if the
> registry entry does not match the cxx_stdlib option in macports.conf.

Thanks Rainer, I didn't remember this.


> This was originally written to support switching the cxx_stdlib from libstdc++
> to libc++ on old releases of OS X in order to support C++11. However, I
> personally lost interest to support legacy systems any longer and do not run 
> any
> of them any more.
> 
> https://github.com/raimue/macports-base/commit/c4386f8c5be01e3f8eeba9e351373df860d9d8ab
> 
> WARNING: Do not install this commit/branch over your regular prefix as
> it will upgrade the SQLite registry.db and make it incompatible with
> MacPorts 2.4.x or master!
> 
> I am not sure how useful this experiment is yet, because this approach
> still has a number of problems and shortcomings:
> 
> * show stopper: cxx_stdlib is stored for all ports, not only those
>  actually linking against the C++ stdlib

I don't consider this a show stopper; I think it's fine to store cxx_stdlib for 
all ports, even non-C++ ports. I consider cxx_stdlib basically part of the 
platform/version, and we store that for all ports, even those that don't change 
how they build on different platforms/versions.


> * cxx_stdlib cannot be changed separately for a single port
>  This might be necessary for some ports that want to link against
>  system frameworks and do not link against anything else. Such a port
>  is now always be considered outdated.

Yeah I would want to fix this. There are already several ports that do this and 
there will be more.


> * I did not know/think of libstdc++ vs. macports-libstdc++

I also don't know how to think about that.


> * archivefetch does not yet respect cxx_stdlib in any way

Yeah that would need adjustment, based on how we decide to differentiate libc++ 
archives from libstdc++ archives.



> For some of these problems, I don't know how they should be solved,
> especially the first show stopper. But maybe this helps to get the ball
> rolling a bit more.

Re: [GSoC] migration

[Umesh Singla]

@Brad, for `migrate`, we're only installing requested ports but for
`restore`, I imagine we will be installing all of them, after sorting?

- Umesh

On Sat, Sep 2, 2017 at 6:48 PM, Umesh Singla 
wrote:

> Hi Mojca,
>
> On Thu, Jun 15, 2017 at 12:05 AM, Mojca Miklavec 
> wrote:
>
>> On 14 June 2017 at 05:47, Umesh Singla wrote:
>> >
>> > Okay, since there's already a OS comparison made, which I think can be
>> > directly used here. But to clarify, just the OS check? a check on
>> x86_64/ppc
>> > changes is also needed.
>>
>> And potentially cxx_stdlib?
>
>
> How can such a check be made possible? All we have at present is here,
> [0]. It doesn't compare stored values and current platform values. Are you
> suggesting a manual check while migration?
>
> - Umesh
>
> [0]: https://github.com/macports/macports-base/blob/
> master/src/macports1.0/macports.tcl#L1050-L1057
>
>

Re: Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Ken Cunningham]

On 2017-09-02, at 11:04 AM, Ken Cunningham wrote:

>> 
>> As stupid as it seems but I don't see any cxx_stdlib option in my 
>> macports.conf (see attached).
> 
> base defaults it to stdlibc++ on <10.9, and libc++ on 10.9+ behind the 
> scenes. 

libstdc++ , not stdlibc++

haven't had coffee yet.

Re: Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Ken Cunningham]

> 
> As stupid as it seems but I don't see any cxx_stdlib option in my 
> macports.conf (see attached).

base defaults it to stdlibc++ on <10.9, and libc++ on 10.9+ behind the scenes. 

> 
> Sure, this is a helpful point to keep in mind. Though, any reasons to not 
> switch to libc++?

Although I personally would have preferred if macports switched wholesale to 
libc++ for all Intel systems for consistency, Marcus' additions to clang-3.9+ 
allowing clang to link nicely against libgcc does the trick quite well in the 
end, and actually offer some benefits in cases (gcc6 offers 
thread-local-storage on systems that libc++ does not).

There are some hassles with this due to idiosyncracies between the headers 
 that we would not have 
otherwise had to deal with, but these are minor enough I guess.


Ken

Re: Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Umesh Singla]

On Sat, Sep 2, 2017 at 9:55 PM, Rainer Müller  wrote:

> On 09/02/2017 03:18 PM, Umesh Singla wrote:
> > On Thu, Jun 15, 2017 at 12:05 AM, Mojca Miklavec  > > wrote:
> >
> > On 14 June 2017 at 05:47, Umesh Singla wrote:
> > >
> > > Okay, since there's already a OS comparison made, which I think
> can be
> > > directly used here. But to clarify, just the OS check? a check on
> x86_64/ppc
> > > changes is also needed.
> >
> > And potentially cxx_stdlib?
> >
> >
> > How can such a check be made possible? All we have at present is here,
> [0]. It
> > doesn't compare stored values and current platform values. Are you
> suggesting a
> > manual check while migration?
>
> I do not think we need any special check at migration, as a rebuild will
> always
> use the current setting in macports.conf. However, after editing the
> default
> cxx_stdlib in macports.conf all affected ports need to be rebuild and
> linked
> with the new default to avoid problems later when installing new ports.
>

As stupid as it seems but I don't see any cxx_stdlib option in my
macports.conf (see attached). Option $macports::cxx_stdlib seems to be
configured while mportinit (which means runtime, I guess) depending on
$os_platform and $os_major. So, it should suffice not to check for it.

The fact that surprises me is that even $mapcorts::build_arch is left to
configure on runtime depending on again, only $os_platform, $os_arch and
$os_major and they are NOT stored anywhere (macports.conf or
macports::autoconf namespace). All these options come from $tcl_platform
which will change as soon as the platform changes. So, this says that only
the check like this:

{($os_platform ne $macports::autoconf::os_platform) || ($os_major !=
$macports::autoconf::os_major)}

should be enough for suggesting to migrate. I'm not able to find any way to
include the check on, as quoted in migration guide, "architecture
migrations (e.g., from PowerPC to Intel)" which could be helpful.

Also, if it is, I can simply add migrate action to run `-f selfupdate`
because it seems that selfupdate.tcl handles downloading and installing a
appropriate macports-base release. Any help is appreciated.


> The following goes beyond your GSoC project, but I think it fits into this
> discussion. Back in May, I published an experiment on my GitHub repo that
> I had
> lying around. This adds functionality to stores the cxx_stdlib option in
> registry, such that 'port outdated' will consider ports as outdated if the
> registry entry does not match the cxx_stdlib option in macports.conf.
>
> This was originally written to support switching the cxx_stdlib from
> libstdc++
> to libc++ on old releases of OS X in order to support C++11. However, I
> personally lost interest to support legacy systems any longer and do not
> run any
> of them any more.
>
> https://github.com/raimue/macports-base/commit/c4386f8c5be01
> e3f8eeba9e351373df860d9d8ab
>
> WARNING: Do not install this commit/branch over your regular prefix as
> it will upgrade the SQLite registry.db and make it incompatible with
> MacPorts 2.4.x or master!


Sure, this is a helpful point to keep in mind. Though, any reasons to not
switch to libc++?

I am not sure how useful this experiment is yet, because this approach
> still has a number of problems and shortcomings:
>
> * show stopper: cxx_stdlib is stored for all ports, not only those
>   actually linking against the C++ stdlib
> * cxx_stdlib cannot be changed separately for a single port
>   This might be necessary for some ports that want to link against
>   system frameworks and do not link against anything else. Such a port
>   is now always be considered outdated.
> * I did not know/think of libstdc++ vs. macports-libstdc++
> * archivefetch does not yet respect cxx_stdlib in any way
>
> For some of these problems, I don't know how they should be solved,
> especially the first show stopper. But maybe this helps to get the ball
> rolling a bit more.


I obviously can't suggest anything right now but once I wrap up, I'll try
looking more.

- Umesh


macports.conf
Description: Binary data

Switching cxx_stdlib for C++11 on legacy macOS (was: Re: [GSoC] migration)

[Rainer Müller]

On 09/02/2017 03:18 PM, Umesh Singla wrote:
> On Thu, Jun 15, 2017 at 12:05 AM, Mojca Miklavec  > wrote:
> 
> On 14 June 2017 at 05:47, Umesh Singla wrote:
> >
> > Okay, since there's already a OS comparison made, which I think can be
> > directly used here. But to clarify, just the OS check? a check on 
> x86_64/ppc
> > changes is also needed.
> 
> And potentially cxx_stdlib?
> 
>  
> How can such a check be made possible? All we have at present is here, [0]. It
> doesn't compare stored values and current platform values. Are you suggesting 
> a
> manual check while migration?

I do not think we need any special check at migration, as a rebuild will always
use the current setting in macports.conf. However, after editing the default
cxx_stdlib in macports.conf all affected ports need to be rebuild and linked
with the new default to avoid problems later when installing new ports.

The following goes beyond your GSoC project, but I think it fits into this
discussion. Back in May, I published an experiment on my GitHub repo that I had
lying around. This adds functionality to stores the cxx_stdlib option in
registry, such that 'port outdated' will consider ports as outdated if the
registry entry does not match the cxx_stdlib option in macports.conf.

This was originally written to support switching the cxx_stdlib from libstdc++
to libc++ on old releases of OS X in order to support C++11. However, I
personally lost interest to support legacy systems any longer and do not run any
of them any more.

https://github.com/raimue/macports-base/commit/c4386f8c5be01e3f8eeba9e351373df860d9d8ab

WARNING: Do not install this commit/branch over your regular prefix as
it will upgrade the SQLite registry.db and make it incompatible with
MacPorts 2.4.x or master!

I am not sure how useful this experiment is yet, because this approach
still has a number of problems and shortcomings:

* show stopper: cxx_stdlib is stored for all ports, not only those
  actually linking against the C++ stdlib
* cxx_stdlib cannot be changed separately for a single port
  This might be necessary for some ports that want to link against
  system frameworks and do not link against anything else. Such a port
  is now always be considered outdated.
* I did not know/think of libstdc++ vs. macports-libstdc++
* archivefetch does not yet respect cxx_stdlib in any way

For some of these problems, I don't know how they should be solved,
especially the first show stopper. But maybe this helps to get the ball
rolling a bit more.

Rainer

Re: [GSoC] migration

[Umesh Singla]

Hi Mojca,

On Thu, Jun 15, 2017 at 12:05 AM, Mojca Miklavec  wrote:

> On 14 June 2017 at 05:47, Umesh Singla wrote:
> >
> > Okay, since there's already a OS comparison made, which I think can be
> > directly used here. But to clarify, just the OS check? a check on
> x86_64/ppc
> > changes is also needed.
>
> And potentially cxx_stdlib?


How can such a check be made possible? All we have at present is here, [0].
It doesn't compare stored values and current platform values. Are you
suggesting a manual check while migration?

- Umesh

[0]:
https://github.com/macports/macports-base/blob/master/src/macports1.0/macports.tcl#L1050-L1057

Re: [GSoC] migration

[Umesh Singla]

Hi Brad,

It occurred to me that I have been committing but did not update you for a
while. For the last few weeks, I have the following notes. I have tried to
write them in a to-the-point manner.

*GSoC Week #10 (4 Aug - 10 Aug)*

Points:


   1. if no variants, then avoid searching for them, so another field in
   snapshot_ports?
   2. to bring the whole snapshot to use Tcl, need to create the entire
   snapshot-stack
   3. limit the number of snapshots a user can store?


*GSoC Week #11 (11 Aug - 18 Aug)*

Points:


   1. We're not doing requested variants yet.


*GSoC Week #12 (19 Aug - 25 Aug)*

I am taking up the only safe approach to restoring all the ports that were
installed and let the user trim leaves afterward.

Points:

   1. added documentation for all the registry-snapshot functions
   2. finally fixed the snapshot format for passing an object from the
   registry to tcl
   3. added copyright notices to all the files with “2017 The MacPorts
   Project”
   4. removed printf/puts statements from all my files
   5. made installation and sort function use the new format of [snapshot
   ports]
   6. removed TODOs which were over and described the left ones at proper
   places

Doubts:

   1. How many snapshots to list for restore? I'm doing 10, I just don't
   feel like listing everything for now.

TODO:

   1. no hash for snapshot->proc, yet.
   2. working on installing the port command after uninstalling.

All comments are appreciated.

Regards,

Umesh Singla

Re: [GSoC] migration

[Umesh Singla]

> > That was a suggested design; if you're already doing it differently then
> I guess you don't need a design. I disagree with the last sentence though,
> a snapshot can be viewed as precisely the state of the original tables at a
> previous time.
>
> If that is the case, we can simply (insert into snapshot_ports select *
> from ports) to create a snapshot. Obviously the snapshot_ports table needs
> an extra snapshot_id column.
>

Yes, that's what we are indirectly doing at present.


> >> Also, we are not using version and revision. Even going by the literal
> meaning of a snapshot, it should not have a key or id linked to something
> that can change over time. It's simply the present state.
> >
> > The row in the ports table would not change over time, it would simply
> persist until no longer needed. If we ever get the ability to install old
> versions then that information would come in handy.
>
> Sounds right.


Sure.

- Umesh

Re: [GSoC] migration

[Umesh Singla]

> > Later, I am planning to keep information on the manual portgroups in the
> snapshot, if there are any.
> >
> > What would this information be used for?
> >
> > I am under the impression that a user can categorize and classify the
> ports into portgroups, so it should be better if we migrate them too.
> Though, it seems highly
>
> Checkout "man portgroup”, I think your understanding may not be 100%
> correct. Portgroups simply provide a way for port authors to include common
> code in there ports.
>
> Perhaps you are thinking that we would need to snapshot portgroup files if
> we where to ever support installing older versions of software. In that
> case you would be correct, the portgroup files would need to match the
> version of the portfile.
>

As I said, I myself am not very clear at present but just listing the
points to be used later. But the above is what I meant more or less, though
I was clear on the relation between portgroups and the versions.

- Umesh

Re: [GSoC] migration

[Bradley Giesbrecht]

> On Aug 11, 2017, at 12:46 PM, Umesh Singla  wrote:
> 
> Hi Josh,
> 
> And then again, I am sensing a confusion with the idea of
> snapshot with Josh, like when he says "remove ports when they
> are no longer referenced by any snapshot".
> 
> 
> What confusion exactly? A snapshot is simply a set of ports (by
> which I mean rows in the 'ports' table, with a unique combination of
> name,version,revision,variants). When nothing references a row any
> more, it needs to be deleted.
> 
> 
> By 'ports', do you mean 'registry.ports' table? If yes, then I disagree. It's 
> actually 'registry.snapshot_ports' table. A snapshot has nothing to do with 
> the original registry "tables".
> 
> That was a suggested design; if you're already doing it differently then I 
> guess you don't need a design. I disagree with the last sentence though, a 
> snapshot can be viewed as precisely the state of the original tables at a 
> previous time.
> 
> This is on the lines what I had in mind regarding a snapshot.
> 
> Also, we are not using version and revision. Even going by the literal 
> meaning of a snapshot, it should not have a key or id linked to something 
> that can change over time. It's simply the present state.
> 
> The row in the ports table would not change over time, it would simply 
> persist until no longer needed. If we ever get the ability to install old 
> versions then that information would come in handy.
> 
> I meant when the port gets updates, the row in the 'ports' table will change 
> but not in 'snapshot_ports' table because for a particular snapshot, the 
> state has changed. But I think it's clear now.
> 
> And yes, version and revision can help in restoring older versions of ports 
> present in older snapshots.
>  
> Later, I am planning to keep information on the manual portgroups in the 
> snapshot, if there are any.
> 
> What would this information be used for?
>  
> I am under the impression that a user can categorize and classify the ports 
> into portgroups, so it should be better if we migrate them too. Though, it 
> seems highly 

Checkout "man portgroup”, I think your understanding may not be 100% correct. 
Portgroups simply provide a way for port authors to include common code in 
there ports.

Perhaps you are thinking that we would need to snapshot portgroup files if we 
where to ever support installing older versions of software. In that case you 
would be correct, the portgroup files would need to match the version of the 
portfile.

—
Brad

Re: [GSoC] migration

[Bradley Giesbrecht]

> On Aug 11, 2017, at 11:16 AM, Joshua Root  wrote:
> 
> On 2017-8-11 21:14 , Umesh Singla wrote:
>> Hi
>>And then again, I am sensing a confusion with the idea of
>>snapshot with Josh, like when he says "remove ports when they
>>are no longer referenced by any snapshot".
>>What confusion exactly? A snapshot is simply a set of ports (by
>>which I mean rows in the 'ports' table, with a unique combination of
>>name,version,revision,variants). When nothing references a row any
>>more, it needs to be deleted.
>> By 'ports', do you mean 'registry.ports' table? If yes, then I disagree. 
>> It's actually 'registry.snapshot_ports' table. A snapshot has nothing to do 
>> with the original registry "tables".
> 
> That was a suggested design; if you're already doing it differently then I 
> guess you don't need a design. I disagree with the last sentence though, a 
> snapshot can be viewed as precisely the state of the original tables at a 
> previous time.

If that is the case, we can simply (insert into snapshot_ports select * from 
ports) to create a snapshot. Obviously the snapshot_ports table needs an extra 
snapshot_id column.

>> Also, we are not using version and revision. Even going by the literal 
>> meaning of a snapshot, it should not have a key or id linked to something 
>> that can change over time. It's simply the present state.
> 
> The row in the ports table would not change over time, it would simply 
> persist until no longer needed. If we ever get the ability to install old 
> versions then that information would come in handy.

Sounds right.

>> Later, I am planning to keep information on the manual portgroups in the 
>> snapshot, if there are any.
> 
> What would this information be used for?

Sorry, I don’t understand what “manual ports groups” refers to. Maybe I missed 
an earlier conversation.

—
Brad

Re: [GSoC] migration

[Joshua Root]

On 2017-8-11 21:14 , Umesh Singla wrote:

Hi

And then again, I am sensing a confusion with the idea of
snapshot with Josh, like when he says "remove ports when they
are no longer referenced by any snapshot".


What confusion exactly? A snapshot is simply a set of ports (by
which I mean rows in the 'ports' table, with a unique combination of
name,version,revision,variants). When nothing references a row any
more, it needs to be deleted.


By 'ports', do you mean 'registry.ports' table? If yes, then I disagree. 
It's actually 'registry.snapshot_ports' table. A snapshot has nothing to 
do with the original registry "tables".


That was a suggested design; if you're already doing it differently then 
I guess you don't need a design. I disagree with the last sentence 
though, a snapshot can be viewed as precisely the state of the original 
tables at a previous time.


Also, we are not using version and revision. Even going by the literal 
meaning of a snapshot, it should not have a key or id linked to 
something that can change over time. It's simply the present state.


The row in the ports table would not change over time, it would simply 
persist until no longer needed. If we ever get the ability to install 
old versions then that information would come in handy.


Later, I am planning to keep information on the manual portgroups in the 
snapshot, if there are any.


What would this information be used for?

- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi

On Fri, Aug 11, 2017 at 4:19 AM, Joshua Root  wrote:

> On 2017-8-11 03:41 , Bradley Giesbrecht wrote:
>
>> I think Josh is referring to 3NF normalization (third normal form). I
>> don’t think this use case warrants this complexity. I think it is fine for
>> two snapshot id’s to reference the same port+variant combination. When a
>> snapshot id is deleted, cascade delete.
>>
>
> Again I'm not sure how this differs from what I wrote. A snapshot contains
> any number of ports, and a port can be in any number of snapshots.
>

I tried to clear this in my other mail.


> Don't worry about having a completely textbook normalised DB, but do avoid
> replicating large amounts of the data.


Sure.


> And TBH a join query would be one of the simplest parts of all this...
>

I started with a join query actually. Something like this,

char* query = "SELECT port_name, requested, variant_name, variant_sign "
"FROM registry.snapshots "
"INNER JOIN "
"registry.snapshot_ports ON "
"snapshots.id=snapshot_ports.snapshots_id "
"LEFT JOIN "
"registry.snapshot_port_variants ON "
"snapshot_ports.id=snapshot_port_variants.snapshot_ports_id"
"WHERE snapshots.id=?";

But then I was having a hard time passing the result to Tcl without a
snapshot struct. And if there is a need for reg_snapshot, then need for
methods as well to operate on it. So, I separated the SQL functions to
fetch ports first and then variants.

But, having said this all, now I think I could have solved this with JOIN
query as well. All these ideas keep coming to me, the longer I spend my
time.

I can really use an idea on workflow here.

- Umesh

Re: [GSoC] migration

[Umesh Singla]

Hi


> And then again, I am sensing a confusion with the idea of snapshot with
>> Josh, like when he says "remove ports when they are no longer referenced by
>> any snapshot".
>>
>
> What confusion exactly? A snapshot is simply a set of ports (by which I
> mean rows in the 'ports' table, with a unique combination of
> name,version,revision,variants). When nothing references a row any more,
> it needs to be deleted.
>

By 'ports', do you mean 'registry.ports' table? If yes, then I disagree.
It's actually 'registry.snapshot_ports' table. A snapshot has nothing to do
with the original registry "tables".

Also, we are not using version and revision. Even going by the literal
meaning of a snapshot, it should not have a key or id linked to something
that can change over time. It's simply the present state.

Later, I am planning to keep information on the manual portgroups in the
snapshot, if there are any.

- Umesh

Re: [GSoC] migration

[Umesh Singla]

> As usual, Josh makes good points.
>
> A text-based database is less convenient to query. I really don’t see why
> storing the information in a database should be challenging. But I’m not
> writing the TCL or C.
>
> I would feel more comfortable running sql queries to verify the snapshots
> are being created accurately.
>
> If text-based storage moves the project forward more quickly do that for
> now. That would be my advice.
>
> It is your call Umesh.
>

At this stage, it's equal time and effort going either side. So I suggest
to continue with the original approach UNLESS there's an additional
advantage with the other.

> And yes, you would need procedures to manipulate this stuff from Tcl as
> you suggested above. And the existing code would need to be updated to only
> remove ports when they are no longer referenced by any snapshot.
> >
> > And then again, I am sensing a confusion with the idea of snapshot with
> Josh, like when he says "remove ports when they are no longer referenced by
> any snapshot”.
>
> I think Josh is referring to 3NF normalization (third normal form). I
> don’t think this use case warrants this complexity. I think it is fine for
> two snapshot id’s to reference the same port+variant combination. When a
> snapshot id is deleted, cascade delete.
>

I think we can pick up the deletion of a snapshot later.

- Umesh

Re: [GSoC] migration

[Joshua Root]

On 2017-8-11 03:41 , Bradley Giesbrecht wrote:

I think Josh is referring to 3NF normalization (third normal form). I don’t 
think this use case warrants this complexity. I think it is fine for two 
snapshot id’s to reference the same port+variant combination. When a snapshot 
id is deleted, cascade delete.


Again I'm not sure how this differs from what I wrote. A snapshot 
contains any number of ports, and a port can be in any number of snapshots.


Don't worry about having a completely textbook normalised DB, but do 
avoid replicating large amounts of the data. And TBH a join query would 
be one of the simplest parts of all this...


- Josh

Re: [GSoC] migration

[Umesh Singla]

On Thu, Aug 10, 2017 at 2:54 AM, Joshua Root  wrote:

> On 2017-8-10 04:59 , Umesh Singla wrote:
>
>> Hi
>>
>> I was trying to streamline the whole process and I felt the need to have
>> the snapshot as a separate entity just like a reg_entry or a reg_portgroup
>> is, that is, "registry::snapshot" with a bunch of functions like create,
>> get, list_all etc. I think this might help in writing the whole thing from
>> a higher Tcl level.
>>
>> But then again, a snapshot is not something coming directly from the
>> database, so I'm not sure how to keep it. Can I get your view on this?
>>
>
> Well, you already know that my view is that it would be far simpler to
> store the snapshots as a text-based format, rather than write and modify a
> large amount of non-trivial C.
>

I think what Josh means here, is a complete text-based dump of the current
state in a file and then using restore, something like what we currently
expect our users to do themselves.

It could be useful in a case when a user wants to completely re-install the
OS from scratch and then, a text-formatted file can easily be backed up for
use by restore later. We can argue that keeping in registry.db is also
similar to keeping in a file, but the interface we are providing a user
help migrate are bit different. Because it is not intuitive to take a dump
of registry.db manually but running a take_snapshot command is.


> But if you are set on doing this in the sqlite database, the relational
> way of doing it would be to add:
>
> 1. A table of snapshots, consisting minimally of names and ids
> 2. A table associating snapshot ids with port ids on a many-to-many basis
>
> Under this model, the set of currently installed ports would be just
> another snapshot, albeit one with a special meaning. The activation state
> of each port would have to move from the ports table and become
> per-snapshot (maybe stored in the second table?).
>
> And yes, you would need procedures to manipulate this stuff from Tcl as
> you suggested above. And the existing code would need to be updated to only
> remove ports when they are no longer referenced by any snapshot.


And then again, I am sensing a confusion with the idea of snapshot with
Josh, like when he says "remove ports when they are no longer referenced by
any snapshot".

- Umesh

Re: [GSoC] migration

[Umesh Singla]

> >> I was trying to streamline the whole process and I felt the need to
> have the snapshot as a separate entity just like a reg_entry or a
> reg_portgroup is, that is, "registry::snapshot" with a bunch of functions
> like create, get, list_all etc. I think this might help in writing the
> whole thing from a higher Tcl level.
> >> But then again, a snapshot is not something coming directly from the
> database, so I'm not sure how to keep it. Can I get your view on this?
> >
> > Well, you already know that my view is that it would be far simpler to
> store the snapshots as a text-based format, rather than write and modify a
> large amount of non-trivial C.
> >
> > But if you are set on doing this in the sqlite database, the relational
> way of doing it would be to add:
> >
> > 1. A table of snapshots, consisting minimally of names and ids
> > 2. A table associating snapshot ids with port ids on a many-to-many basis
>
> The snapshot tables would not be updated. I see no reason for a many to
> many. Each port/variant in a snapshot would have a snapshot id. When a
> snapshot is deleted delete it’s id everywhere.
>

It is simply a one-to-many foreign key, that is, linking one snapshot to
many port ids and further, one port to many variants.

Speaking of deleting a snapshot, do we want to limit the number of
snapshots a user can have? or something like keeping an expiry date for a
snapshot.

- Umesh

Re: [GSoC] migration

[Umesh Singla]

Hi

On Sun, Aug 6, 2017 at 6:43 PM, Jackson Isaac  wrote:

> Hi,
>
> On Sun, Aug 6, 2017 at 5:41 PM, Umesh Singla 
> wrote:
> > On Sat, Jul 22, 2017 at 7:26 AM, Joshua Root  wrote:
> >>
> >>
> >>> For now, I'd like to ask in what order does "registry::entry imaged"
> >>> returns the port list? Because I'm running the sorting function which
> the
> >>> restore_ports.tcl uses but it's giving me the ports in the same order
> as
> >>> result.
> >>
> >>
> >> Probably just ordered by rowid, i.e. might as well be random. Note that
> >> the sort_ports proc from restore_ports.tcl does not take a list of
> registry
> >> references like registry::entry returns, but a list of strings
> representing
> >> port names, versions and variants in the format generated by 'port
> >> installed'.
> >
> >
> > `port installed` returns the result of `registry::entry imaged` sorted
> in an
> > alphabetical order, first by name, then version etc.
> >
> > `registry::entry imaged` might be returning in a random order but
> sorting it
> > (with ports coming before their dependencies) doesn't change the result
> at
> > all.
> >
>
> [1] and [2] might be useful. The list of ports are passed to
> _mportexec after topologically sorting them for installing. This might
> also answer some of your previous queries regarding using [mportexec
> $workername $install_target] and sorting the dependencies at once.
>

Thanks for this.


> Variants is still pending. You could try to integrate sorting and
> installation of ports with libsolv and extend variant handling from
> your script. You might have to tweak the libsolv code if you pass the
> variants like foo+bar-baz. One idea could be to add 'bar' as
> dependency and 'baz' as conflict explicitly since it will only read
> the portindex for 'foo'.
>

When you say variants is still pending, do you mean we can't specify
particular set of variants along with a port to be installed? So, it always
assumes the default variants for a port at present?

If the variants are still not supported by libsolv, what I think is best to
go with calculating dependencies recursively as I'm doing now and then
integrate it with libsolv.


> Libsolv proved to be faster than our traditional recursive dependency
> calculation engine even after reading through the complete portindex
> every time it was called.
>

I had doubts about "reading the portindex contents first and writing into
libsolv formatted solvables"  every time when I went through the code but
for now, I can take your word.


> Let me know if you have any doubts regarding the libsolv code. The
> comments should be explanatory enough by itself but feel free if you
> want to understand what is actually happening behind the scenes by
> libsolv.
>

The comments are really descriptive there. macports-base can hopefully
learn from it.

- Umesh

Re: [GSoC] migration

[Umesh Singla]

Hi

On Sun, Aug 6, 2017 at 6:56 PM, Joshua Root  wrote:

> On 2017-8-6 22:11 , Umesh Singla wrote:
>
>> On Sat, Jul 22, 2017 at 7:26 AM, Joshua Root  j...@macports.org>> wrote:
>>
>>
>> For now, I'd like to ask in what order does "registry::entry
>> imaged" returns the port list? Because I'm running the sorting
>> function which the restore_ports.tcl uses but it's giving me the
>> ports in the same order as result.
>>
>>
>> Probably just ordered by rowid, i.e. might as well be random. Note
>> that the sort_ports proc from restore_ports.tcl does not take a list
>> of registry references like registry::entry returns, but a list of
>> strings representing port names, versions and variants in the format
>> generated by 'port installed'.
>>
>>
>> `port installed` returns the result of `registry::entry imaged` sorted in
>> an alphabetical order, first by name, then version etc.
>>
>
> Yes. The order of the input lines doesn't matter for restore_ports.tcl of
> course because it's sorting them. My point was just that that particular
> sort procedure takes something like "zlib @1.2.11_0 (active)" as input,
> rather than "::registry::entry150".


Yes, I was aware of that anyway.


>
> `registry::entry imaged` might be returning in a random order but sorting
>> it (with ports coming before their dependencies) doesn't change the result
>> at all.
>>
>
> This could easily be true in some cases but definitely not in the general
> case. Are you testing on real-world registry contents or just on an
> installation you created recently for testing? In the latter case, each
> port will happen to come before its dependents simply because dependencies
> are installed first. After some upgrading of ports over time, in
> practically random order, this will often no longer be the case.
>

I was only trying on installation I created which had about 60 ports in
total but only 5 requested. So, you are right because the order was always
stored sorted and was never modified. Thanks.


>
> Attached is a small script that I used to demonstrate that this property
> does not hold in my own registry. There, entry0 is fftw-3, and comes before
> its dependent entry243 (py27-numpy). On the other hand, entry267 is
> py27-setuptools, which comes after its dependent entry76 (py27-nose).
>

- Umesh

Re: [GSoC] migration

[Joshua Root]

On 2017-8-6 22:11 , Umesh Singla wrote:
On Sat, Jul 22, 2017 at 7:26 AM, Joshua Root > wrote:



For now, I'd like to ask in what order does "registry::entry
imaged" returns the port list? Because I'm running the sorting
function which the restore_ports.tcl uses but it's giving me the
ports in the same order as result.


Probably just ordered by rowid, i.e. might as well be random. Note
that the sort_ports proc from restore_ports.tcl does not take a list
of registry references like registry::entry returns, but a list of
strings representing port names, versions and variants in the format
generated by 'port installed'.


`port installed` returns the result of `registry::entry imaged` sorted 
in an alphabetical order, first by name, then version etc.


Yes. The order of the input lines doesn't matter for restore_ports.tcl 
of course because it's sorting them. My point was just that that 
particular sort procedure takes something like "zlib @1.2.11_0 (active)" 
as input, rather than "::registry::entry150".


`registry::entry imaged` might be returning in a random order but 
sorting it (with ports coming before their dependencies) doesn't change 
the result at all.


This could easily be true in some cases but definitely not in the 
general case. Are you testing on real-world registry contents or just on 
an installation you created recently for testing? In the latter case, 
each port will happen to come before its dependents simply because 
dependencies are installed first. After some upgrading of ports over 
time, in practically random order, this will often no longer be the case.


Attached is a small script that I used to demonstrate that this property 
does not hold in my own registry. There, entry0 is fftw-3, and comes 
before its dependent entry243 (py27-numpy). On the other hand, entry267 
is py27-setuptools, which comes after its dependent entry76 (py27-nose).


- Josh


reg_deps.tcl
Description: Tcl script

Re: [GSoC] migration

[Umesh Singla]

On Sat, Jul 22, 2017 at 7:26 AM, Joshua Root  wrote:

>
> For now, I'd like to ask in what order does "registry::entry imaged"
>> returns the port list? Because I'm running the sorting function which the
>> restore_ports.tcl uses but it's giving me the ports in the same order as
>> result.
>>
>
> Probably just ordered by rowid, i.e. might as well be random. Note that
> the sort_ports proc from restore_ports.tcl does not take a list of registry
> references like registry::entry returns, but a list of strings representing
> port names, versions and variants in the format generated by 'port
> installed'.
>

`port installed` returns the result of `registry::entry imaged` sorted in
an alphabetical order, first by name, then version etc.

`registry::entry imaged` might be returning in a random order but sorting
it (with ports coming before their dependencies) doesn't change the result
at all.

- Umesh

Re: [GSoC] migration

[Umesh Singla]

Hi Brad

A quick update.

Points -

   1. Is it okay to fetch dependents using mportlookup while installing?
   2. For migrate action, we agreed upon installing all and only the
   (active requested + inactive requested) ports, in short, all the requested
   in the sorted order, right? Dependencies can figure out themselves, they
   might be different for different platforms.
   3. Are the ports which are part of the snapshot related to the actual
   ports at implementation level? If no, then snapshot needs to move to a
   different data structure, and in that case:


   - move all to C file snapshot.c from entry.c in cregistry. Till now, I
  was just using a way around to it with "registry::entry create_snapshot"
  which doesn't comply with the design at all. A snapshot shouldn't use the
  data structure 'reg_entry', so I'm moving all to a separate module where
  the C registry functions dealing with snapshot could be kept.
  - OR if 'reg_entry' is possible, then why do we have a
  reg_portgroup struct and others.

Todo -

   1. Later: fetching dependencies only once while migrating. Right now,
   doing it twice for installation and while uninstalling.
   2. know the internals of `port selfupdate` and check if a part of it can
   be used for the only step left in migrate, having a fresh installation of
   macports for the updated platform.

Please find the work log here

.

Regards,
Umesh Singla

On Tue, Aug 1, 2017 at 6:17 AM, Bradley Giesbrecht 
wrote:

> > On Jul 27, 2017, at 2:10 PM, db  wrote:
> >
> > On 27 Jul 2017, at 21:21, Bradley Giesbrecht 
> wrote:
> >> Migration will uninstall all installed ports and then install all the
> ports from the snapshot
> >
> > Yes, but I'm talking about using snapshot and restore at different times
> with ports being updated between them, likely resulting in different
> outcomes.
>
> We are not keeping track of port versions so restore would deactivate all
> ports and then install the snap shot ports with variants. At least that is
> what I imagine.
>
> —
> Brad

Re: [GSoC] migration

[Bradley Giesbrecht]

> On Jul 27, 2017, at 2:10 PM, db  wrote:
> 
> On 27 Jul 2017, at 21:21, Bradley Giesbrecht  wrote:
>> Migration will uninstall all installed ports and then install all the ports 
>> from the snapshot
> 
> Yes, but I'm talking about using snapshot and restore at different times with 
> ports being updated between them, likely resulting in different outcomes.

We are not keeping track of port versions so restore would deactivate all ports 
and then install the snap shot ports with variants. At least that is what I 
imagine.

—
Brad

Re: [GSoC] migration

[db]

On 27 Jul 2017, at 21:21, Bradley Giesbrecht  wrote:
> Migration will uninstall all installed ports and then install all the ports 
> from the snapshot

Yes, but I'm talking about using snapshot and restore at different times with 
ports being updated between them, likely resulting in different outcomes.

Re: [GSoC] migration

[Bradley Giesbrecht]

> On Jul 27, 2017, at 11:39 AM, db  wrote:
> 
> On 27 Jul 2017, at 00:35, Umesh Singla  wrote:
>> Hi Josh
>>> We don't have version dependencies so no, this sort of check is not 
>>> possible a priori. Breakage will be caught by rev-upgrade after the fact in 
>>> many cases (and if it is set to rebuild automatically, it may well revert 
>>> you back to the newer version).
> 
>>> Also note that we can't actually revert to an older version of a port if it 
>>> has been uninstalled.
> 
> Does it mean that snapshot just outputs a list of ports, deps and variants 
> without their versions, so if I postpone the restore long enough, it'll 
> reinstall current versions of all of them? I'm thinking here about the 
> possibility of reinstalling a system or migrating between machines with same 
> operating system and ending up with some broken ports.

Migration will uninstall all installed ports and then install all the ports 
from the snapshot (active and inactive) finishing with the same active ports 
and requested flags matching the snapshot. I imagine the order of install 
matters so that dependent ports are built first so the correct variants are 
active when the ports with dependencies build.

—
Brad

Re: [GSoC] migration

[db]

On 27 Jul 2017, at 00:35, Umesh Singla  wrote:
> Hi Josh
>> We don't have version dependencies so no, this sort of check is not possible 
>> a priori. Breakage will be caught by rev-upgrade after the fact in many 
>> cases (and if it is set to rebuild automatically, it may well revert you 
>> back to the newer version).

>> Also note that we can't actually revert to an older version of a port if it 
>> has been uninstalled.

Does it mean that snapshot just outputs a list of ports, deps and variants 
without their versions, so if I postpone the restore long enough, it'll 
reinstall current versions of all of them? I'm thinking here about the 
possibility of reinstalling a system or migrating between machines with same 
operating system and ending up with some broken ports.

Re: [GSoC] migration

[Joshua Root]

On 2017-7-27 14:57 , Joshua Root wrote:
The order to follow when uninstalling is the reverse of the order used 
when installing, i.e. dependents-first rather than dependencies-first. 
Order does matter because we check whether a port has dependents when 
uninstalling and raise an error if so. You can set a flag to ignore this 
check, but warnings will still be printed, so it's better just to do it 
in the right order.


Technically the structure of the dependency relationships of a set of 
ports is a directed acyclic graph, not a tree. A DAG can be 
topologically sorted, which is a bit like flattening a tree. When you're 
dealing with a set of ports that don't all necessarily have dependencies 
in common, only a partial ordering can be established, and that's what 
you see in the sorting done by portlist_sortdependents in port.tcl (each 
port comes earlier in the list than all its dependencies) and sort_ports 
in restore_ports.tcl (each port is later in the list than all its 
dependencies).


Oh and if you want a visualisation of the structure, have a look at 
port-depgraph. 



- Josh

Re: [GSoC] migration

[Joshua Root]

On 2017-7-27 08:40 , Umesh Singla wrote:
On Sat, Jul 22, 2017 at 7:26 AM, Joshua Root > wrote:


On 2017-7-21 08:22 , Umesh Singla wrote:

For now, I'd like to ask in what order does "registry::entry
imaged" returns the port list? Because I'm running the sorting
function which the restore_ports.tcl uses but it's giving me the
ports in the same order as result.


Probably just ordered by rowid, i.e. might as well be random. Note
that the sort_ports proc from restore_ports.tcl does not take a list
of registry references like registry::entry returns, but a list of
strings representing port names, versions and variants in the format
generated by 'port installed'.


I have a question, might be stupid but seems to confuse me for more than 
the required time. Does the order of ports (in a portlist) followed 
while uninstalling all of them be same as while installing them? I'm 
trying to imagine a tree-based structure here but not quite getting the 
picture.


I mean, does the order even matter when we want to uninstall of them?


The order to follow when uninstalling is the reverse of the order used 
when installing, i.e. dependents-first rather than dependencies-first. 
Order does matter because we check whether a port has dependents when 
uninstalling and raise an error if so. You can set a flag to ignore this 
check, but warnings will still be printed, so it's better just to do it 
in the right order.


Technically the structure of the dependency relationships of a set of 
ports is a directed acyclic graph, not a tree. A DAG can be 
topologically sorted, which is a bit like flattening a tree. When you're 
dealing with a set of ports that don't all necessarily have dependencies 
in common, only a partial ordering can be established, and that's what 
you see in the sorting done by portlist_sortdependents in port.tcl (each 
port comes earlier in the list than all its dependencies) and sort_ports 
in restore_ports.tcl (each port is later in the list than all its 
dependencies).


- Josh

Re: [GSoC] migration

[Joshua Root]

On 2017-7-27 08:35 , Umesh Singla wrote:

Also note that we can't actually revert to an older version of a
port if it has been uninstalled.


I don't understand this. Does this mean that macports only allows me to 
install the most recent version of a port? You said "if it has been 
uninstalled", so if it's still installed, can we then revert to an older 
version?


That's right. If the older version is still installed but inactive, you 
can simply activate it. If it's no longer installed, manual intervention 
is required to check out the right commit in the git history and use 
that version of the Portfile.


The instructions for that are at 
.


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi

On Sun, Jul 23, 2017 at 5:03 AM, db  wrote:

> On 22 Jul 2017, at 03:01, Umesh Singla  wrote:
> > I don't know, in the above example, what do you mean when you say "..you
> realize that from its deps..", like, realize how? I am just asking this if
> there's some other way to get info on the latest modifications that I might
> not be aware of.
> > 'sync' or 'upgrade outdated' provide the information what all ports got
> updated immediately on console but I'm not sure if it can be accessed later
> since the user may not realize that another port has broken immediately,
> like hstr here.
>
> Since MP doesn't keep logs (!) I keep them myself and investigate when
> something doesn't work properly.
>
> > Another thing that comes to my mind now is if, suppose, updated version
> of ncurses was actually required for some another port and reverting it to
> the older state could possibly result in breaking of that port.
>
> No, because, as I said, it would revert to a previous tree state. The
> thing is doing it by reinstalling only the upgraded ports and not the whole
> tree. It shouldn't be difficult to implement.


@Bradley, when we came up with re-installing all the ports, I had this
doubt too. It sure shouldn't be difficult, getting only the upper and lower
dependencies to check for, if a user is able to figure out the faulty port
himself.

I understand the part of migration but I'd like to know more about the
restore here. This conversation actually helped me focus on the
"install/activate the snapshot ports" part of your statement
"Restoring a snapshot would turn off all parts and install/activate the
snapshot ports with variants."
in one of your previous replies.

Can we help with such a use case here? To me, "it would revert to a
previous tree state" sounds like restore but with less complexity. Though
issues will come up even if the number of such ports is greater than 2 and
will make it hard for the user.

Regards,
Umesh

Re: [GSoC] migration

[Umesh Singla]

Hi Josh

Another thing that comes to my mind now is if, suppose, updated version of
>> ncurses was actually required for some another port and reverting it to the
>> older state could possibly result in breaking of that port. May be, we
>> could get all the ports which depend on it and check if this specific port
>> requirement could be satisfied by its older versions as well and then just
>> ask the user if the user would like to restore or not?
>>
>> Again, I'm not really aware of the things, so I'd like the inputs of
>> community here.
>>
>
> We don't have version dependencies so no, this sort of check is not
> possible a priori. Breakage will be caught by rev-upgrade after the fact in
> many cases (and if it is set to rebuild automatically, it may well revert
> you back to the newer version).
>

Thanks for clearing this out.


> Also note that we can't actually revert to an older version of a port if
> it has been uninstalled.
>

I don't understand this. Does this mean that macports only allows me to
install the most recent version of a port? You said "if it has been
uninstalled", so if it's still installed, can we then revert to an older
version?

- Umesh

Re: [GSoC] migration

[db]

On 22 Jul 2017, at 03:01, Umesh Singla  wrote:
> I don't know, in the above example, what do you mean when you say "..you 
> realize that from its deps..", like, realize how? I am just asking this if 
> there's some other way to get info on the latest modifications that I might 
> not be aware of.
> 'sync' or 'upgrade outdated' provide the information what all ports got 
> updated immediately on console but I'm not sure if it can be accessed later 
> since the user may not realize that another port has broken immediately, like 
> hstr here.

Since MP doesn't keep logs (!) I keep them myself and investigate when 
something doesn't work properly.

> Another thing that comes to my mind now is if, suppose, updated version of 
> ncurses was actually required for some another port and reverting it to the 
> older state could possibly result in breaking of that port.

No, because, as I said, it would revert to a previous tree state. The thing is 
doing it by reinstalling only the upgraded ports and not the whole tree. It 
shouldn't be difficult to implement.

Re: [GSoC] migration

[Joshua Root]

On 2017-7-22 11:01 , Umesh Singla wrote:
Another thing that comes to my mind now is if, suppose, updated version 
of ncurses was actually required for some another port and reverting it 
to the older state could possibly result in breaking of that port. May 
be, we could get all the ports which depend on it and check if this 
specific port requirement could be satisfied by its older versions as 
well and then just ask the user if the user would like to restore or not?


Again, I'm not really aware of the things, so I'd like the inputs of 
community here.


We don't have version dependencies so no, this sort of check is not 
possible a priori. Breakage will be caught by rev-upgrade after the fact 
in many cases (and if it is set to rebuild automatically, it may well 
revert you back to the newer version).


Also note that we can't actually revert to an older version of a port if 
it has been uninstalled.


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi

On Fri, Jul 21, 2017 at 5:39 PM, db  wrote:

> On 21 Jul 2017, at 13:02, Umesh Singla  wrote:
> > Unless we have a snapshot of the previous state, that is, before it got
> hampered.
> > But then again, we reinstall all the ports presently. At this time, it
> could be hard for me to detect what went wrong while sync or upgrade.
>
> If I understood you correctly it will presently reinstall all ports in a
> given snapshot.
>
> I'm not saying that these commands should check for what went wrong during
> upgrade, but to reinstall only those whose upgrade caused another port to
> stop working as expected. The actual cause is for the user to find. For
> example, let's say you did upgrade outdated and hstr doesn't work anymore,
> but you realise that from its rdeps only ncurses was updated. Then you
> could use restore with the snapshot preceding the upgrade to just rollback
> the whole tree to that state by reinstalling in this case only ncurses and
> not hstr, readline, etc.
>
> hstr
>   pkgconfig
> libiconv
>   gperf
>   ncurses
>   readline
>
>
Let's say we have a snapshot of the state before sync or upgrade.

I don't know, in the above example, what do you mean when you say "..you
realize that from its deps..", like, realize how? I am just asking this if
there's some other way to get info on the latest modifications that I might
not be aware of.

'sync' or 'upgrade outdated' provide the information what all ports got
updated immediately on console but I'm not sure if it can be accessed later
since the user may not realize that another port has broken immediately,
like hstr here.

Another thing that comes to my mind now is if, suppose, updated version
of ncurses was actually required for some another port and reverting it to
the older state could possibly result in breaking of that port. May be, we
could get all the ports which depend on it and check if this specific port
requirement could be satisfied by its older versions as well and then just
ask the user if the user would like to restore or not?

Again, I'm not really aware of the things, so I'd like the inputs of
community here.

Regards,
Umesh

Re: [GSoC] migration

[db]

On 21 Jul 2017, at 13:02, Umesh Singla  wrote:
> Unless we have a snapshot of the previous state, that is, before it got 
> hampered.
> But then again, we reinstall all the ports presently. At this time, it could 
> be hard for me to detect what went wrong while sync or upgrade.

If I understood you correctly it will presently reinstall all ports in a given 
snapshot.

I'm not saying that these commands should check for what went wrong during 
upgrade, but to reinstall only those whose upgrade caused another port to stop 
working as expected. The actual cause is for the user to find. For example, 
let's say you did upgrade outdated and hstr doesn't work anymore, but you 
realise that from its rdeps only ncurses was updated. Then you could use 
restore with the snapshot preceding the upgrade to just rollback the whole tree 
to that state by reinstalling in this case only ncurses and not hstr, readline, 
etc.

hstr
  pkgconfig
libiconv
  gperf
  ncurses
  readline

Re: [GSoC] migration

[Umesh Singla]

Hi Brad, Clemens, Josh

Please see inline.

> GSoC Week #7 (13 Jul - 19 Jul)
> >
> > I moved the things to migrate action instead of 'restore' since
> 'restore' looks a subset of 'migrate', so can be dealt easily once migrate
> is over.
>
> I’m not sure I understand and if I do understand I think I disagree.
>

Ah, sorry. I agree. I shouldn't have used the word "subset". I was aware
that only a small section overlaps.

Snapshot is a state of installed and active ports with their variants.
> Restoring a snapshot would turn off all parts and install/activate the
> snapshot ports with variants.
> Migration would be taking a new snapshot, uninstalling all ports,
> upgrading the port command for the new arch and finally restoring the last
> snapshot which would entail installing all ports from snapshot since we
> installed earlier.
>
> Snapshot and restore should be able to be executed without migrate.
>

Yes, that was in mind. But then again, it's good having more clarity as I
go.

Reading below I think I misunderstand so please clarify.
>
> > Agenda for last week:
> >
> >
> > 3. Re-installing the ports from the portlist. I took the help of
> restore_ports.tcl here, which works when called migrate. It uses sort_ports
> [1] function.
> >
> >
> >
> > When we install, is it fine to use [mportexec $workername
> $install_target] from migrate.tcl?
>
> I’m sorry, I don’t know base well enough to answer this. Can you please
> ask Clemens or Josh?
>

@Clemens @Josh, can you clarify this?

> What took time is that these two functions at present take the port list
> in different formats. So, I tried debugging by simply printing on console
> method to know the formats of the two.
> >
> > I want to use the sorting function for dependencies only once instead of
> calling it twice while uninstalling and then re-installing. I'm still
> working on it and actually, close to solve this but thought, better send
> the weekly update first.
>


> I’d check with Clemens on this. He mentored a previous GSoC project which
> used libsolve for dependency calculations. Perhaps there is something to
> borrow or learn from that project?


Sure, I will look into it. For now, I'd like to ask in what order does
"registry::entry imaged" returns the port list? Because I'm running the
sorting function which the restore_ports.tcl uses but it's giving me the
ports in the same order as result.



> >
> > Since migrate action creates a snapshot within and then uses it to redo
> the whole state, I feel creating a snapshot here explicitly is redundant.
>
> Not sure I follow. I imaging migrate action would start by calling
> snapshot, register the snapshot id, uninstall all ports, reinstall the port
> command and finally call restore action with the snapshot id for the last
> snapshot.
>
> > Internally snapshot also uses the same registry C functions to get the
> information first and then save to database as [registry::entry imaged]
> does, so instead of first retrieving from [registry::entry imaged], saving
> it in sqlite and then again fetching all the info using the snapshot-id we
> just got, can't we directly use [registry::entry imaged] here? Am I missing
> something?
>
> Oh, maybe in this case we already have the list in memory? But when we
> upgrade the port command wouldn’t we then call the new port command and
> perform the restore with the upgrade port command?
>

Okay, yes. For a moment, I mixed restore and migrate and forgot about a
different port command now. Also, Brad, when we say "reinstall the port
command", what exactly does it mean? Like, what all steps it could have?
Does it mean including from the installation guide here [1]. I could use
some help here.


> Very good letter Umesh, thanks for the update. I hope my replies and
> questions are useful.
>

Welcome. They are indeed.

Regards,
Umesh Singla

[1]: https://www.macports.org/install.php

Re: [GSoC] migration

[db]

On 20 Jul 2017, at 06:04, Bradley Giesbrecht  wrote:
> Snapshot and restore should be able to be executed without migrate.

I have a question and I haven't been following in detail this GSoC, so bear 
with me.

Could snapshot and restore be used to rollback to a previous state in case a 
sync and upgrade outdated fails or installs a port that doesn't work properly 
and without reinstalling all non-relevant (unchanged) ports?

Re: [GSoC] migration

[Bradley Giesbrecht]

> On Jul 19, 2017, at 3:43 PM, Umesh Singla  wrote:
> 
> Hi Brad,
> 
> With another week over, I have the following updates to share. 
> 
> 
> GSoC Week #7 (13 Jul - 19 Jul)
> 
> I moved the things to migrate action instead of 'restore' since 'restore' 
> looks a subset of 'migrate', so can be dealt easily once migrate is over.

I’m not sure I understand and if I do understand I think I disagree.

Snapshot is a state of installed and active ports with their variants.
Restoring a snapshot would turn off all parts and install/activate the snapshot 
ports with variants.
Migration would be taking a new snapshot, uninstalling all ports, upgrading the 
port command for the new arch and finally restoring the last snapshot which 
would entail installing all ports from snapshot since we installed earlier.

Snapshot and restore should be able to be executed without migrate.

Reading below I think I misunderstand so please clarify.

> Agenda for last week:
> 
> The migrate action consisted of three parts, creating a new snapshot, 
> uninstall installed ports, re-installing them for the last snapshot.
> 
> 1. Creating a new snapshot was over before, though some refactoring is still 
> left.
> 
> 2. Regarding uninstalling installed ports, I took the help of reclaim.tcl 
> which works with the migrate action now. It uses 
> sort_portlist_by_dependendents procedure. So, all in all, it uses 
> "registry_uninstall::uninstall".
> 
> 3. Re-installing the ports from the portlist. I took the help of 
> restore_ports.tcl here, which works when called migrate. It uses sort_ports 
> [1] function. 
> 
> 
> 
> When we install, is it fine to use [mportexec $workername $install_target] 
> from migrate.tcl?

I’m sorry, I don’t know base well enough to answer this. Can you please ask 
Clemens or Josh?

> What took time is that these two functions at present take the port list in 
> different formats. So, I tried debugging by simply printing on console method 
> to know the formats of the two. 
> 
> I want to use the sorting function for dependencies only once instead of 
> calling it twice while uninstalling and then re-installing. I'm still working 
> on it and actually, close to solve this but thought, better send the weekly 
> update first.

It is indeed better to send updates more frequently. I’d check with Clemens on 
this. He mentored a previous GSoC project which used libsolve for dependency 
calculations. Perhaps there is something to borrow or learn from that project?

> For the next one week:
> 
>   • make appropriate changes to procedures according to one format.
>   • write for “SELECT PORTS FROM SNAPSHOT ..”, i.e. connect it with the 
> new tables we have.
>   • try finishing the migrate action asap?

Sounds good.

> Points:
> 
> Since migrate action creates a snapshot within and then uses it to redo the 
> whole state, I feel creating a snapshot here explicitly is redundant. 

Not sure I follow. I imaging migrate action would start by calling snapshot, 
register the snapshot id, uninstall all ports, reinstall the port command and 
finally call restore action with the snapshot id for the last snapshot.

> Internally snapshot also uses the same registry C functions to get the 
> information first and then save to database as [registry::entry imaged] does, 
> so instead of first retrieving from [registry::entry imaged], saving it in 
> sqlite and then again fetching all the info using the snapshot-id we just 
> got, can't we directly use [registry::entry imaged] here? Am I missing 
> something?

Oh, maybe in this case we already have the list in memory? But when we upgrade 
the port command wouldn’t we then call the new port command and perform the 
restore with the upgrade port command?

> Regarding "selected" snapshot for 'restore', it looks fine.
> 
> New points which I made during pondering to save for the end:
> 
>   • Add ui_debug statements
>   • Catch some error-prone areas.
>   • Remove the useless/structure comments and add actual doc-strings.
>   • Add copyright notices
>   • Using ui_msg instead of puts and else. Should probably add that.

Yes to all the above.

> Also, here's my view of restore_ports.tcl of what it does:
> 
>   • macports prefix configs
>   • umask 022
>   • set portList [read_portlist $filename] - reads the portlist from file 
> supplied or stdin
>   • set operationList [sort_ports $portList]
>   • sort_ports $portList
>   • extracts name, version, variants, active for a port
>   • sets it 1 in port_in_list if not there, else 
> increments the count for that port
>   • if port dependencies for the combination (port_name, 
> its variants) doesn’t exist already, fetches them using 
> dependenciesForPort(port_name, its variants), kind of recursively.
>   • dependenciesForPort $port_name $variants
>   • install_ports $operationList
> 

Re: [GSoC] migration

[Umesh Singla]

Hi Brad,

With another week over, I have the following updates to share.


*GSoC Week #7 (13 Jul - 19 Jul)*

I moved the things to migrate action instead of 'restore' since 'restore'
looks a subset of 'migrate', so can be dealt easily once migrate is over.


Agenda for last week:

The migrate action consisted of three parts, creating a new snapshot,
uninstall installed ports, re-installing them for the last snapshot.

1. Creating a new snapshot was over before, though some refactoring is
still left.

2. Regarding uninstalling installed ports, I took the help of reclaim.tcl
which works with the migrate action now. It uses
sort_portlist_by_dependendents procedure. So, all in all, it uses
"registry_uninstall::uninstall".

3. Re-installing the ports from the portlist. I took the help of
restore_ports.tcl here, which works when called migrate. It uses sort_ports
[1] function.


When we install, is it fine to use [mportexec $workername $install_target]
from migrate.tcl?

What took time is that these two functions at present take the port list in
different formats. So, I tried debugging by simply printing on console
method to know the formats of the two.

I want to use the sorting function for dependencies only once instead of
calling it twice while uninstalling and then re-installing. I'm still
working on it and actually, close to solve this but thought, better send
the weekly update first.


For the next one week:

   1. make appropriate changes to procedures according to one format.
   2. write for “SELECT PORTS FROM SNAPSHOT ..”, i.e. connect it with the
   new tables we have.
   3. try finishing the migrate action asap?


Points:

Since migrate action creates a snapshot within and then uses it to redo the
whole state, I feel creating a snapshot here explicitly is redundant.

Internally snapshot also uses the same registry C functions to get the
information first and then save to database as [registry::entry imaged]
does, so instead of first retrieving from [registry::entry imaged], saving
it in sqlite and then again fetching all the info using the snapshot-id we
just got, can't we directly use [registry::entry imaged] here? Am I missing
something?

Regarding "selected" snapshot for 'restore', it looks fine.


New points which I made during pondering to save for the end:

   1. Add ui_debug statements
   2. Catch some error-prone areas.
   3. Remove the useless/structure comments and add actual doc-strings.
   4. Add copyright notices
   5. Using ui_msg instead of puts and else. Should probably add that.


Also, here's my view of restore_ports.tcl of what it does:

   - macports prefix configs
   - umask 022
   - set portList [read_portlist $filename] - reads the portlist from file
   supplied or stdin
   - set operationList [sort_ports $portList]
  - sort_ports $portList
 - extracts name, version, variants, active for a port
 - sets it 1 in port_in_list if not there, else increments the
 count for that port
 - if port dependencies for the combination (port_name, its
 variants) doesn’t exist already, fetches them using
 dependenciesForPort(port_name, its variants), kind of recursively.
  - dependenciesForPort $port_name $variants
   - install_ports $operationList
  - checks $active for a port and use it to decide install_target
  - uses mportexec for installing


Please find the work log here [2]. It is largely in work-in-progress state.


[1]:
https://github.com/macports/macports-contrib/blob/master/restore_ports/restore_ports.tcl#L52

[2]: https://github.com/macports/macports-base/commits/gsoc17-migrate


Best Regards,

Umesh Singla

Re: [GSoC] migration

[Joshua Root]

On 2017-7-14 01:42 , Arno Hautala wrote:

On Thu, Jul 13, 2017 at 11:34 AM, Joshua Root  wrote:


Depending on a variant is currently impossible.

...

You do have to install the ports in dependency order to get it right, and
that is what restore_ports does. There may be cases where previously
inactive ports don't build due to requiring a specific variant in a
dependency that was also inactive. (Which is equivalent to "you can't depend
on a variant".)


So a port might "require" on a variant, but it can't express that
dependency without user action?


Correct, the best you can do is use the active_variants portgroup which 
will detect the problem and ask the user to fix it.


- Josh

Re: [GSoC] migration

[Arno Hautala]

On Thu, Jul 13, 2017 at 11:34 AM, Joshua Root  wrote:
>
> Depending on a variant is currently impossible.
>
> ...
>
> You do have to install the ports in dependency order to get it right, and
> that is what restore_ports does. There may be cases where previously
> inactive ports don't build due to requiring a specific variant in a
> dependency that was also inactive. (Which is equivalent to "you can't depend
> on a variant".)

So a port might "require" on a variant, but it can't express that
dependency without user action?

-- 
arno  s  hautala/-|   a...@alum.wpi.edu

pgp b2c9d448

Re: [GSoC] migration

[Joshua Root]

On 2017-7-13 22:49 , Arno Hautala wrote:

On Wed, Jul 12, 2017 at 8:07 PM, Joshua Root  wrote:

On 2017-7-13 08:10 , Umesh Singla wrote:


  3. ensuring that active versions are installed after the inactive
 versions


The install action of port(1) does execute the activate target, however your
code could run only the install target for ports that were not active. That
is what restore_ports.tcl does.


Wouldn't it still be possible to end up in the wrong state due to
dependencies? Imagine that port A+huge depends on port B+huge, while
A+tiny depends on B+tiny. +huge is inactive, +tiny is active.



This is also all predicated on the idea that MacPorts handles
dependent variants. I forget what the current state is there.


Depending on a variant is currently impossible.


Installing A+huge would install and activate A+huge and B+huge.
Installing, but not activating A+tiny, would first require installing
and activating B+tiny.

Certainly this could be resolved by installing and not activating the
port and everything in the dependency tree. I haven't examined
restore_ports to see what it does with dependencies.

Another method would be to install and deactivate all the inactive
ports first. Or even just default to not installing anything that's
inactive and providing an option to install those as well (or
presenting a list to pick from). That's perhaps getting a bit too
extensive at this point.


You do have to install the ports in dependency order to get it right, 
and that is what restore_ports does. There may be cases where previously 
inactive ports don't build due to requiring a specific variant in a 
dependency that was also inactive. (Which is equivalent to "you can't 
depend on a variant".)


- Josh

Re: [GSoC] migration

[Arno Hautala]

On Wed, Jul 12, 2017 at 8:07 PM, Joshua Root  wrote:
> On 2017-7-13 08:10 , Umesh Singla wrote:
>>
>>  3. ensuring that active versions are installed after the inactive
>> versions
>
> The install action of port(1) does execute the activate target, however your
> code could run only the install target for ports that were not active. That
> is what restore_ports.tcl does.

Wouldn't it still be possible to end up in the wrong state due to
dependencies? Imagine that port A+huge depends on port B+huge, while
A+tiny depends on B+tiny. +huge is inactive, +tiny is active.

This is also all predicated on the idea that MacPorts handles
dependent variants. I forget what the current state is there.

Installing A+huge would install and activate A+huge and B+huge.
Installing, but not activating A+tiny, would first require installing
and activating B+tiny.

Certainly this could be resolved by installing and not activating the
port and everything in the dependency tree. I haven't examined
restore_ports to see what it does with dependencies.

Another method would be to install and deactivate all the inactive
ports first. Or even just default to not installing anything that's
inactive and providing an option to install those as well (or
presenting a list to pick from). That's perhaps getting a bit too
extensive at this point.

-- 
arno  s  hautala/-|   a...@alum.wpi.edu

pgp b2c9d448

Re: [GSoC] migration

[Joshua Root]

On 2017-7-13 08:10 , Umesh Singla wrote:

 3. ensuring that active versions are installed after the inactive
versions since the output after running `port install vim -huge +tiny`:

——

Computing dependencies for vim

..

--->Staging vim into destroot

*--->Installing vim @8.0.0596_0+tiny*

*--->Deactivating vim @8.0.0596_0+tiny+x11*

--->Cleaning vim

*--->Activating vim @8.0.0596_0+tiny*

--->Cleaning vim

--->Updating database of binaries

..

shows that installation also activates and we don’t want to activate the 
ports which were originally inactivated.


The install action of port(1) does execute the activate target, however 
your code could run only the install target for ports that were not 
active. That is what restore_ports.tcl does.


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi

>   • A snapshot would be a list of install commands that created the
> current installed state.
> >   • Restoring a snapshot would deactivate the active ports and
> reproduce the install commands for the selected snapshot.
> >   • Migrate would amount to creating a new snapshot, uninstalling
> installed ports and reproducing the install commands for the last snapshot.
> > @Brad: Did you miss “uninstalling installed ports” in the second point
> above?
>
> No. I imagined being able to switch between snapshots with minimal
> building. Migration would uninstall because we are changing platforms.
>

I didn't look at the point with this perspective. But it sure is a great
idea.

Thanks,
Umesh

Re: [GSoC] migration

[Bradley Giesbrecht]

> On Jul 12, 2017, at 3:10 PM, Umesh Singla  wrote:
> 
> Hi Brad,
> 
> With another week over, I have the following updates to share. These are 
> actually the points from my Notes as I keep going on, so the email might not 
> be well crafted. Thought, would share with you without any delay.
> 
> Week #5-6 (28 Jun - 12 Jul)
> 
> Golden lines for the project:
> 
>   • A snapshot would be a list of install commands that created the 
> current installed state. 
>   • Restoring a snapshot would deactivate the active ports and reproduce 
> the install commands for the selected snapshot. 
>   • Migrate would amount to creating a new snapshot, uninstalling 
> installed ports and reproducing the install commands for the last snapshot.
> @Brad: Did you miss “uninstalling installed ports” in the second point above?

No. I imagined being able to switch between snapshots with minimal building. 
Migration would uninstall because we are changing platforms.

Regards,
Bradley Giesbrecht (pixilla)

Re: [GSoC] migration

[Umesh Singla]

Hi Brad,

With another week over, I have the following updates to share. These are
actually the points from my Notes as I keep going on, so the email might
not be well crafted. Thought, would share with you without any delay.

*Week #5-6 (28 Jun - 12 Jul)*

Golden lines for the project:

   1. A *snapshot* would be a list of install commands that created the
   current installed state.
   2. *Restoring* a snapshot would deactivate the active ports and
   reproduce the install commands for the selected snapshot.
   3. *Migrate* would amount to creating a new snapshot, uninstalling
   installed ports and reproducing the install commands for the last snapshot.

@Brad: Did you miss “uninstalling installed ports” in the second point
above?


Challenges from the first phase:

   1. “variants” - still not discussed by anyone after 2 emails, 1 post and
   1 comment on GitHub. -> resolved.
   2. I wrote at the end of my first phase that one variant of a port can
   be active at a time, turns out I was wrong. All thanks to the UI messages
   and the research.

Challenges still to tackle:

   1. snapshot needs to store which ports are active as well?
   2. refactoring snapshot to use registry::write wrappers and transaction
   logic.
   3. ensuring that active versions are installed after the inactive
   versions since the output after running `port install vim -huge +tiny`:

——

Computing dependencies for vim

..

--->  Staging vim into destroot

*--->  Installing vim @8.0.0596_0+tiny*

*--->  Deactivating vim @8.0.0596_0+tiny+x11*

--->  Cleaning vim

*--->  Activating vim @8.0.0596_0+tiny*

--->  Cleaning vim

--->  Updating database of binaries

..

shows that installation also activates and we don’t want to activate the
ports which were originally inactivated.

Agenda for last two weeks:

   1. Begin with restore action.
   2. Write for uninstalling ports - took the help of reclaim.tcl for this
   3. Work on reproducing the install commands for a snapshot
   4. Improve the snapshot action as needed for the restore action.

Still left from last two weeks:

   1. A part to include installation of ports with the help of
   restore_ports.tcl.

Agenda for next two:

   1. Finish the restore action by Jul 24.

Challenges that might come later:

   1. dealing with builds at MP_PREFIX and not at /opt/local. Don’t know
   now but keep in mind.
   2. conflicting ports as Josh said and migration guide says.

Important discoveries I made during this period:

   1. came to know that the port *vim +tiny+x11* is different from *vim
   +tiny*, says a lot how ports are variant-bundled.
   2. ‘imaged’ is installed but inactive, while ‘installed’ is installed
   and active.

Pondering:

   1. Reinstalling the MacPorts/Xcode CL tools for the new platform if it
   is not built for the new OS already is not something we can help with,
   right?

Work log: https://github.com/macports/macports-base/commits/gsoc17-migrate

All reviews and comments are welcome.

Regards,

Re: [GSoC] migration

[Bradley Giesbrecht]

Josh, thank you for responding.

Regards,
Bradley Giesbrecht (pixilla)


> On Jul 11, 2017, at 11:58 AM, Joshua Root  wrote:
> 
> On 2017-7-12 04:43 , Umesh Singla wrote:
>> Hi Brad,
>> While I was writing for restoring of ports, it seems that I need to include 
>> another field 'active' along with ports while taking snapshot. This might 
>> sound obvious now but it wasn't to me before I looked at what is going on, 
>> closely. While there is something like 'active' port but there's no such 
>> state as being 'inactive'.
> 
> You want the 'state' property, which is either 'imaged' for installed but 
> inactive or 'installed' for active.
> 
>> This led me to a further research and I came across this in the migration 
>> guide:
>> "Though it is now quite well-tested, the restore_ports script may fail in 
>> some cases. One known issue is that the script will fail if there are 
>> conflicting ports in the list. It's possible to have conflicting ports 
>> installed provided at most one of the conflicting set is active. If the 
>> script fails for this reason, you can delete one of the conflicting ports..."
>> Do you think it can be solved by merely adding active field? What possible 
>> conflicts is it talking about?
> 
> This is referring to a case such as having jpeg installed and inactive and 
> mozjpeg installed and active. The ports themselves conflict.
> 
> - Josh

Re: [GSoC] migration

[Joshua Root]

On 2017-7-12 04:43 , Umesh Singla wrote:

Hi Brad,

While I was writing for restoring of ports, it seems that I need to 
include another field 'active' along with ports while taking snapshot. 
This might sound obvious now but it wasn't to me before I looked at what 
is going on, closely. While there is something like 'active' port but 
there's no such state as being 'inactive'.


You want the 'state' property, which is either 'imaged' for installed 
but inactive or 'installed' for active.


This led me to a further research and I came across this in the 
migration guide:


"Though it is now quite well-tested, the restore_ports script may fail 
in some cases. One known issue is that the script will fail if there are 
conflicting ports in the list. It's possible to have conflicting ports 
installed provided at most one of the conflicting set is active. If the 
script fails for this reason, you can delete one of the conflicting 
ports..."


Do you think it can be solved by merely adding active field? What 
possible conflicts is it talking about?


This is referring to a case such as having jpeg installed and inactive 
and mozjpeg installed and active. The ports themselves conflict.


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi Brad,

While I was writing for restoring of ports, it seems that I need to include
another field 'active' along with ports while taking snapshot. This might
sound obvious now but it wasn't to me before I looked at what is going on,
closely. While there is something like 'active' port but there's no such
state as being 'inactive'.

This led me to a further research and I came across this in the migration
guide:

"Though it is now quite well-tested, the restore_ports script may fail in
some cases. One known issue is that the script will fail if there are
conflicting ports in the list. It's possible to have conflicting ports
installed provided at most one of the conflicting set is active. If the
script fails for this reason, you can delete one of the conflicting
ports..."

Do you think it can be solved by merely adding active field? What possible
conflicts is it talking about?

The example I went through is installing of vim. I ran:

*port install vim -huge +tiny +x11* -> this is active now

followed by,

*port install vim -huge +tiny* -> now this becomes active and previous one
deactivated (different ports, therefore). Of course, both are requested =
True.

So, in the snapshot database, it is:


ports:

id  snapshot_id  portname  req

2540104   vim 1 ...

2593104   vim 1

with variants stored as:

idport_id  variantsign

242 2540   tiny +

243 2540   x11 +

244 2540   huge   -   ...

250 2593   tiny +

251 2593   huge   -

Reproducing the install commands for these two, is that a conflict? I used
reg_entry_imaged which doesn't give info on a port being active. Probably,
I should use a combination of reg_entry_installed and reg_entry_imaged OR
registry::installed which is more straightforward and serves my purpose.
Can you help me with the next step here?

Regards,
Umesh Singla

Re: [GSoC] migration

[Joshua Root]

On 2017-6-25 06:29 , Bradley Giesbrecht wrote:

On Jun 22, 2017, at 8:15 PM, Joshua Root  wrote:

On 2017-6-23 11:29 , Bradley Giesbrecht wrote:

For the migration functionality wouldn’t we only install “requested” ports? 
Dependencies could be different with a platform change.
Also, if the installed variants for a given port are the default variants would 
we want to ignore variants?


These two points are related. A requested flag for variants is needed to be 
able to do the right thing.

We don't want to ignore variants exactly; we want to apply only the requested 
variants (positive and negative) when reinstalling ports.


Is a “default” variant requested? I believe some ports have different default 
variants depending on the platform, operation system or other environment 
factors. Shouldn’t we be replicating the actual user install commands as close 
as possible? Should we include variants the user did not supply to the port 
install command?


Default variants are unrequested (unless the user also asked for them 
explicitly). The idea is indeed to replicate what the user asked for, 
and allow default variants to be selected as normal after that.



A snapshot would be a list of install commands that created the current 
installed state. Restoring a snapshot would deactivate the active ports and 
reproduce the install commands for the selected snapshot. Migrate would amount 
to creating a new snapshot, uninstalling installed ports and reproducing the 
install commands for the last snapshot.


OK, well that's a bit more involved than I expected and relies on 
information that is not currently kept in the registry.


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi Bradley,

On Sun, Jun 25, 2017 at 1:59 AM, Bradley Giesbrecht 
wrote:
>
> >> For the migration functionality wouldn’t we only install “requested”
> ports? Dependencies could be different with a platform change.
> >> Also, if the installed variants for a given port are the default
> variants would we want to ignore variants?
> >
> > These two points are related. A requested flag for variants is needed to
> be able to do the right thing.
> >
> > We don't want to ignore variants exactly; we want to apply only the
> requested variants (positive and negative) when reinstalling ports.
>
> Is a “default” variant requested? I believe some ports have different
> default variants depending on the platform, operation system or other
> environment factors. Shouldn’t we be replicating the actual user install
> commands as close as possible?


I believe a default variant is requested unless the first command to
install the port has variants explicitly mentioned but it is not requested
in the context of a user because the user can probably be unaware of
available variants.

But again, as you said, there can be different default variants, storing
the original defaults can result in confusion between what originally was
default (they should not become requested) and what is the current default?
So, I think we should NOT treat the default as requested. Replicating the
actual user commands is the most ideal thing to do.

A confusion could have been here: `port install port_name` and `port
install port_name +default`. Both will install default but it's not
requested in first case while requested in second case. The second one
should get listed as requested in db and hence, not create any problem
further.


> Should we include variants the user did not supply to the port install
> command?
>

I didn't get it. Either the variants are default or they were requested,
both will be present in the stack. How can we get the variants which the
user never supplied? Are you trying to hint at the difference in the port
installed at the first run and the variants installed later on?

Also, I need your help in a technical detail. I'll try to push in some time
and get back to you if you're free.


Regards,
Umesh Singla

Re: [GSoC] migration

[Joshua Root]

On 2017-6-23 11:29 , Bradley Giesbrecht wrote:

On Jun 21, 2017, at 7:25 PM, Umesh Singla  wrote:

Hi

 Taking a step back for a moment, why is an SQL database the best way
 to store this data? What sorts of queries are you going to want to
 run on it? Would a text (Tcl array) representation similar to the
 PortIndex be a better fit?


We need to store all the information about the existing state of the ports 
first, then uninstall all the ports and re-install on the updated OS after 
self-updating using the info we have in the database. So, we can't have a 
temporary kind of Tcl array representation.

A Tcl array representation doesn't have to be temporary. It can easily be 
written to a file.

If the only operations are going to be storing the list of ports and then 
reinstalling all the ports in the list, that says to me that SQL is much more 
complexity than you need.

I'll check with my mentor. I may not have been clear in explaining the need for 
SQL or Tcl array representation could be an easy way go.


I assumed we would allow multiple snapshots and be able to chose from a list of 
snapshots by date-sequence.


Sure.


For the migration functionality wouldn’t we only install “requested” ports? 
Dependencies could be different with a platform change.

Also, if the installed variants for a given port are the default variants would 
we want to ignore variants?


These two points are related. A requested flag for variants is needed to 
be able to do the right thing.


We don't want to ignore variants exactly; we want to apply only the 
requested variants (positive and negative) when reinstalling ports.


There is a bit of a dilemma when a non-requested port has requested 
variants. In some cases (like quartz vs x11) the variants of 
dependencies must match, so you do want to restore that port as it was. 
And even when it's not strictly required in that way, the user may still 
have wanted that variant in the dependency for whatever reason. But in 
other cases, as you mentioned, the dependencies may have changed so the 
port is no longer needed.


I think the only safe approach is to (attempt to) restore all the ports 
that were installed and let the user trim leaves afterwards. (A message 
could even suggest that they do so.) Some may fail on the new platform, 
but that's always a possibility even with requested ports. The system 
needs to handle that as gracefully as possible.


I'll leave for now the question of whether you should pass requested 
variants down to new dependencies...


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi

Taking a step back for a moment, why is an SQL database the best way
>> to store this data? What sorts of queries are you going to want to
>> run on it? Would a text (Tcl array) representation similar to the
>> PortIndex be a better fit?
>>
>>
>> We need to store all the information about the existing state of the
>> ports first, then uninstall all the ports and re-install on the updated OS
>> after self-updating using the info we have in the database. So, we can't
>> have a temporary kind of Tcl array representation.
>>
>
> A Tcl array representation doesn't have to be temporary. It can easily be
> written to a file.
>
> If the only operations are going to be storing the list of ports and then
> reinstalling all the ports in the list, that says to me that SQL is much
> more complexity than you need.


I'll check with my mentor. I may not have been clear in explaining the need
for SQL or Tcl array representation could be an easy way go.

In registry2 parlance, "imaged" means installed and "installed" means
> active. Confusing I know.
>

Thanks for the clarification.

- Umesh

Re: [GSoC] migration

[Joshua Root]

On 2017-6-22 02:09 , Umesh Singla wrote:

Hi Josh,

Taking a step back for a moment, why is an SQL database the best way
to store this data? What sorts of queries are you going to want to
run on it? Would a text (Tcl array) representation similar to the
PortIndex be a better fit?



We need to store all the information about the existing state of the 
ports first, then uninstall all the ports and re-install on the updated 
OS after self-updating using the info we have in the database. So, we 
can't have a temporary kind of Tcl array representation.


A Tcl array representation doesn't have to be temporary. It can easily 
be written to a file.


If the only operations are going to be storing the list of ports and 
then reinstalling all the ports in the list, that says to me that SQL is 
much more complexity than you need.



The way to get the list of installed ports is with 'registry::entry
imaged'. Have a look at how reclaim.tcl does it for example.



Yes, thanks for this. @Brad, 'registry::entry imaged' or 
'registry::entry installed', may be?


I found 2 functions reg_entry_imaged() and reg_entry_installed() in 
registry, which look almost similar implementation wise. Can you tell me 
what is an imaged port? I mean, how will the two results be different?


In registry2 parlance, "imaged" means installed and "installed" means 
active. Confusing I know.


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi Josh,

Taking a step back for a moment, why is an SQL database the best way to
> store this data? What sorts of queries are you going to want to run on it?
> Would a text (Tcl array) representation similar to the PortIndex be a
> better fit?
>


We need to store all the information about the existing state of the ports
first, then uninstall all the ports and re-install on the updated OS after
self-updating using the info we have in the database. So, we can't have a
temporary kind of Tcl array representation.


I can get this info from `port -v installed` and `port -v installed
>> requested` command, that is, calling /action_installed/ [1] which in turn,
>> calls /registry::installed/ [2] for each port from /snapshot_main/. Is this
>> best way to go?
>>
>
> Using registry::installed will work, but it's not the most efficient since
> it provides the same interface as the old flat-file registry (See below).
>
> Further, can you point me to some port action where it deals with
>> retrieving and populating tables as a hint? The /action_target/ used for
>> most of the port commands like /install/, /clean/, /fetch/ doesn't really
>> hint on how to deal with it.
>>
>
> These don't deal with SQL directly, they use the registry API.
>
> I hunted down till I reached /macports::registry.format/  -->
>> /receipt_{flat/sqlite}.tcl /files//which have a bunch of functions using
>> /registry::entry/ which I think is the most "basic" operation and then,
>> also /receipt_sqlite:://create_entry_l/function.
>>
>
> The way to get the list of installed ports is with 'registry::entry
> imaged'. Have a look at how reclaim.tcl does it for example.
>


Yes, thanks for this. @Brad, 'registry::entry imaged' or 'registry::entry
installed', may be?

I found 2 functions reg_entry_imaged() and reg_entry_installed() in
registry, which look almost similar implementation wise. Can you tell me
what is an imaged port? I mean, how will the two results be different?


The create_entry_l procedure is there to help in converting the old
> flat-file format to sqlite. It's probably not very relevant for you.
>

Yeah, ignoring this.


> If you do end up adding new tables to the db, you will need to add support
> for them to the entire stack from cregistry up.
>


I think you're referring to the work in last two commits here

.


Thank you for the help,
Umesh

Re: [GSoC] migration

[Joshua Root]

On 2017-6-20 03:58 , Umesh Singla wrote:

Hi Bradley,

I'm having a bit difficulty in implementing the body of snapshot 
procedure. Basically what I need to do now is (as I have written in my 
notes, so you can tell me if I'm on right track in my thoughts as well):


"get the list of installed ports, their-installed-variants and 
if-requested,
then add one port at a time to the 2nd table, its variants to the 3rd 
table,

after all this, add an entry in the snapshot table for the entire snapshot,
that's it."


Taking a step back for a moment, why is an SQL database the best way to 
store this data? What sorts of queries are you going to want to run on 
it? Would a text (Tcl array) representation similar to the PortIndex be 
a better fit?


If there's a good reason for doing it in SQL, great. If not, maybe 
you're making life more difficult than it needs to be. :)


I can get this info from `port -v installed` and `port -v installed 
requested` command, that is, calling /action_installed/ [1] which in 
turn, calls /registry::installed/ [2] for each port from 
/snapshot_main/. Is this best way to go?


Using registry::installed will work, but it's not the most efficient 
since it provides the same interface as the old flat-file registry (See 
below).


Further, can you point me to some port action where it deals with 
retrieving and populating tables as a hint? The /action_target/ used for 
most of the port commands like /install/, /clean/, /fetch/ doesn't 
really hint on how to deal with it.


These don't deal with SQL directly, they use the registry API.

I hunted down till I reached /macports::registry.format/  --> 
/receipt_{flat/sqlite}.tcl /files//which have a bunch of functions using 
/registry::entry/ which I think is the most "basic" operation and then, 
also /receipt_sqlite:://create_entry_l/function.


The way to get the list of installed ports is with 'registry::entry 
imaged'. Have a look at how reclaim.tcl does it for example.


The create_entry_l procedure is there to help in converting the old 
flat-file format to sqlite. It's probably not very relevant for you.


The registry::entry proc is indeed where much of the magic happens at 
the Tcl level. It's implemented in src/registry2.0/entry.c and calls 
down into the code in the src/cregistry directory to do the actual 
sqlite calls.


Do I need to write for inserting into tables? If yes, write direct SQL 
queries or Tcl procedures ( a better way, I think)?


If you do end up adding new tables to the db, you will need to add 
support for them to the entire stack from cregistry up.


- Josh

Re: [GSoC] migration

[Umesh Singla]

Hi Bradley,

I'm having a bit difficulty in implementing the body of snapshot procedure.
Basically what I need to do now is (as I have written in my notes, so you
can tell me if I'm on right track in my thoughts as well):

"get the list of installed ports, their-installed-variants and
if-requested,
then add one port at a time to the 2nd table, its variants to the 3rd
table,
after all this, add an entry in the snapshot table for the entire snapshot,
that's it."

I can get this info from `port -v installed` and `port -v installed
requested` command, that is, calling *action_installed* [1] which in turn,
calls *registry::installed* [2] for each port from *snapshot_main*. Is this
best way to go?

Further, can you point me to some port action where it deals with
retrieving and populating tables as a hint? The *action_target* used for
most of the port commands like *install*, *clean*, *fetch* doesn't really
hint on how to deal with it.

I hunted down till I reached *macports::registry.format*  -->
*receipt_{flat/sqlite}.tcl
*files which have a bunch of functions using *registry::entry* which I
think is the most "basic" operation and then, also *receipt_sqlite::*
*create_entry_l* function.

Do I need to write for inserting into tables? If yes, write direct SQL
queries or Tcl procedures ( a better way, I think)?

Any help would be appreciated.

Regards,
Umesh


[1]:
https://github.com/macports/macports-base/blob/master/src/port/port.tcl#L3266
[2]:
https://github.com/macports/macports-base/blob/master/src/registry2.0/registry.tcl#L136

Re: [GSoC] migration

[Bradley Giesbrecht]

> On Jun 13, 2017, at 8:47 PM, Umesh Singla  wrote:
> 
> I suggest IRC #macports on Wednesdays at 2:00 PM UTC. If this time does not 
> work for anyone who would like to be included suggest an alternative or 
> additional day and time.
>  
> So, today!

I hung out by myself on IRC from 2:30-4:30 PM UTC :)

—
Brad

Re: [GSoC] migration

[Rainer Müller]

On 2017-06-14 05:48, Umesh Singla wrote:
> Thanks Rainer for this. This is really useful. I had doubts on how often
> cregistry/sql.c file gets updated or whether this is the one. The name
> sql.c seems to be so OS core file. So, I can just add the new tables here? 
> 
> Also, new tables means adding them in update_db() too, just like it is
> the case with existing port or portgroup tables?

Yes, add new tables to create_tables(). Also add them to update_db() so
older installations get them, too.

Rainer

Re: [GSoC] migration

[Rainer Müller]

On 2017-06-14 05:47, Umesh Singla wrote:
> > 4. Another thing that ran my mind while pondering that there are 2
> options for sqlite database as well: make the tables in the very
> beginning (while initial installation) or while running the snapshot
> for the first time. I suggest to go with the first one because it's
> simple.
> > The major target is to finish the snapshot action before Jun 24.
> 
> Does “port selfupdate” constitute an “initial installation”?
> 
> 
> The way I went through macports1.0 -> selfupdate.tcl file, I don't see
> any such thing.
> 
> Does port currently perform schema checks?
> 
> 
> If there's sql.c -> update_db() function, then it should be called
> somewhere. I only see it at one place inside registry attach function
> which is not really relevant here. 

It is called every time the registry is opened, therefore there is
nothing special in selfupdate. The registry will be upgraded when the
newly installed port command is called for the first time (which already
happens during selfupdate).

Rainer

Re: [GSoC] migration

[Umesh Singla]

Hi

For testing it should be enough to just modify the registry.db locally
> as you need it. Once you reach a stable schema, you will have to add
> modifications to the registry schema at two places.
>
> 1) The initial database schema for new installations is defined here:
>
> https://github.com/macports/macports-base/blob/
> e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L122-L131
>
> 2) The metadata table in registry.db has a row with key='version', where
> value holds the schema version. The code to update from one schema
> version to the next is here:
>
> https://github.com/macports/macports-base/blob/
> e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L231-L240


Thanks Rainer for this. This is really useful. I had doubts on how often
cregistry/sql.c file gets updated or whether this is the one. The name
sql.c seems to be so OS core file. So, I can just add the new tables here?

Also, new tables means adding them in update_db() too, just like it is the
case with existing port or portgroup tables?

Forgive me for asking almost everything.


--
Umesh Singla

Re: [GSoC] migration

[Umesh Singla]

Hi

> I have not followed Rainer's strategy for having `port snapshot --create`
> and `port snapshot --restore` as discussed in the previous thread, instead
> have 3 separate actions...
>
> If I am following this correctly I think isolating the functionality into
> actions makes sense. If we want to refactor the interface in the future it
> will be easier to combine pieces then split into parts.


Acknowledged.

> 4. Another thing that ran my mind while pondering that there are 2
> options for sqlite database as well: make the tables in the very beginning
> (while initial installation) or while running the snapshot for the first
> time. I suggest to go with the first one because it's simple.
> > The major target is to finish the snapshot action before Jun 24.
>
> Does “port selfupdate” constitute an “initial installation”?
>

The way I went through macports1.0 -> selfupdate.tcl file, I don't see any
such thing.

Does port currently perform schema checks?
>

If there's sql.c -> update_db() function, then it should be called
somewhere. I only see it at one place inside registry attach function which
is not really relevant here.


> If port can detect if the schema needs updating then perhaps we can hook
> in and “do the right thing".
>

But all this is for the first time, right? I'll ponder over it more.

Point to me if I missed something vital.


> I suggest IRC #macports on Wednesdays at 2:00 PM UTC. If this time does
> not work for anyone who would like to be included suggest an alternative or
> additional day and time.
>

So, today!


--
Umesh Singla

Re: [GSoC] migration

[Rainer Müller]

On 2017-06-11 19:07, Bradley Giesbrecht wrote:
>> 4. Another thing that ran my mind while pondering that there are 2 options 
>> for sqlite database as well: make the tables in the very beginning (while 
>> initial installation) or while running the snapshot for the first time. I 
>> suggest to go with the first one because it's simple.
>> The major target is to finish the snapshot action before Jun 24.
> 
> Does “port selfupdate” constitute an “initial installation”?
> 
> Does port currently perform schema checks?
> 
> If port can detect if the schema needs updating then perhaps we can hook in 
> and “do the right thing".

For testing it should be enough to just modify the registry.db locally
as you need it. Once you reach a stable schema, you will have to add
modifications to the registry schema at two places.

1) The initial database schema for new installations is defined here:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L122-L131

2) The metadata table in registry.db has a row with key='version', where
value holds the schema version. The code to update from one schema
version to the next is here:

https://github.com/macports/macports-base/blob/e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/cregistry/sql.c#L231-L240

Rainer

Re: [GSoC] migration

[Bradley Giesbrecht]

Hi Umesh,


Thanks to Rainer for his comments on 2 and 5. My comments and questions inline.


On Jun 8, 2017, at 3:12 PM, Umesh Singla  wrote:
> 
> 1. The agenda as per the proposal timeline [1] for the period from May 30 to 
> Jun 10 is to finalize the different cases to be included in the scope of the 
> project and their priorities, plan the design of the procedures to be written 
> in port.tcl (now in snapshot.tcl and migrate.tcl in macports1.0) and begin 
> with the implementation of the snapshot action. So, for now, we'll be 
> focusing only on the specific snapshot action.
> 
> I have not followed Rainer's strategy for having `port snapshot --create` and 
> `port snapshot --restore` as discussed in the previous thread, instead have 3 
> separate actions. Since we agreed on other use cases as well during the 
> proposal submission time like only keeping a snapshot for now, listing the 
> snapshots to choose from later etc. which will be cumbersome if we resort to 
> `port snapshot <--create, list, migrate, restore>` and could be confusing at 
> times. 

If I am following this correctly I think isolating the functionality into 
actions makes sense. If we want to refactor the interface in the future it will 
be easier to combine pieces then split into parts.

> 4. Another thing that ran my mind while pondering that there are 2 options 
> for sqlite database as well: make the tables in the very beginning (while 
> initial installation) or while running the snapshot for the first time. I 
> suggest to go with the first one because it's simple.
> The major target is to finish the snapshot action before Jun 24.

Does “port selfupdate” constitute an “initial installation”?

Does port currently perform schema checks?

If port can detect if the schema needs updating then perhaps we can hook in and 
“do the right thing".

> 6. One request, I asked this before too, can we please, please fix some time 
> for IRC/Hangouts meeting at least a week for the beginning if it's 
> comfortable with you or Clemens or other people involved? This will help me 
> keep on track. I am a person who can only work under strict deadlines. I have 
> failed under loose deadlines before.

I suggest IRC #macports on Wednesdays at 2:00 PM UTC. If this time does not 
work for anyone who would like to be included suggest an alternative or 
additional day and time.


Regards,
Bradley Giesbrecht (pixilla)

Re: [GSoC] migration

[Umesh Singla]

Hi Rainer,

I have already created a new branch gsoc17-migrate in macports-base after
discussing with Jackson on IRC and just now, pushed some initial code to
https://github.com/macports/macports-base/tree/gsoc17-migrate, to begin
with. For the copyright part, I'll just ignore it, for now, it doesn't seem
to be a big deal.

Thanks

On Fri, Jun 9, 2017 at 5:18 AM, Rainer Müller  wrote:

> Hello Umesh,
>
> good to hear from you! I was afraid we had already lost you...
>
> Where you will push your work to? Either create a new branch (e.g.
> gsoc17-migration) in macports/macports-base, or push to your own github
> account. It would be good to know which place to watch for updates.
>
> I will answer some of other questions inline below.
>
> On 2017-06-09 00:38, Umesh Singla wrote:
> > 2. In the proposal, I had written "what would be desirable is if `port`
> > was to /recommend/ a migration upon detecting a new environment", so we
> > can have it two ways - either check for the changes in the environment
> > before running every command or only check for the changes when a user
> > actually uses restore (or migrate) action.
> >
> > While the first one seems to be more realistic from user's perspective
> > but running it every time is also not a good idea since OS/hardware
> > changes are not frequent. I suggest running the detection for changes in
> > architecture before a set of some commands like install, sync,
> > selfupdate etc. The second option is actually on-top-of-head type
> > solution which assumes the user to be aware of any arch changes by
> > himself and thus, is actually not "recommending". Please suggest a way
> > to proceed.
> >
> > Also, is there any existing action which checks for changes in the
> > environment which I can use as a reference. I checked selfupdate but it
> > only checks how old port definitions are. Is it sufficed to check for
> > universal_arch/build_arch like options in `macports.conf` file,
> > comparing it with `uname -a` or `env` command outputs. How rigorous we
> > need the detection to be?
>
> Such a check already exists, which is executed an every initialization
> of the macports1.0 package. As this is just a simple compare of the OS
> version, it is no problem to run this every time. Currently it prints a
> link to the Migration wiki page:
>
> https://github.com/macports/macports-base/blob/
> e3a0dc2ebde62a9c5feac6a1edee1708a95bb02a/src/macports1.0/
> macports.tcl#L651-L656
>
> > 5. I saw the copyright license on top of most of the files. Do
> > developers have this, right from the beginning or after the initial work
> > on the module gets finished?
>
> The following goes with the usual IANAL disclaimer. In almost all
> jurisdictions, you own the copyright from the moment you create
> something. There is no need to explicitly claim copyright any more
> (mostly after USA reformed their copyright law in 1989). The headers in
> source code files have pure informational use.
>
> In my opinion, it only makes sense to list authors who contributed a
> significant amount of code to the file. Most of files in base should
> already list "The MacPorts Project". Although not being a legal entity,
> this is a kind of placeholder for all contributors holding the joint
> copyright ownership.
>
>
> While I could not answer all your questions right now (I have to leave
> some work for Bradley ;-)), feel free to ask questions as much as you
> like on the list or via IRC.
>
> Happy hacking on your project!
>
> Rainer
>