Re: Allowing PortGroups to bump port revision

[René J . V . Bertin]

On Saturday January 13 2018 16:19:10 Clemens Lang wrote:

Hi,

This went into my junk folder so I'm only seeing it now.


>This could be made much faster if Tcl had a way to 'clone' an
>interpreter that already had port1.0 loaded before running the rest of
>the Portfile. If somebody wants to implement that, that would probably
>improve portindex speeds significantly.

I suppose it should, esp. if Tcl doesn't compile expressions into some internal 
binary representation.

On top of that it would probably help if you could make an existing parsing 
context read-only so you can cut down on stack recursion. I don't know if that 
makes much sense; I did this once for an interpreter I wrote. Allows it to 
evaluate 2 or 3 levels of constant expressions without having to create new 
stack frames for things as simple as initialising a variable with a constant or 
from another variable **). It did require a tedious bit of code writing but it 
gave a nice extra boost (though much less than implementing a byte/JIT 
compiler).

**) Portfiles contain lots of that kind of expression, but I hope Tcl's parser 
is a little more involved than the simple contraption I wrote when so much 
younger.

Re: Allowing PortGroups to bump port revision

[Joshua Root]

On 2018-1-14 02:19 , Clemens Lang wrote:
> Hi,
> 
> On Sun, Jan 14, 2018 at 12:01:34AM +1100, Joshua Root wrote:
>> portindex isn't slow, mportopen is. You have to evaluate the changed
>> Portfiles before you can extract their info.
> 
> I haven't measures this, but I assume the majority of time in mportopen
> is spent by the sub-interpreter loading the port1.0 package (i.e.
> evaluating the PortSystem 1.0 line in the Portfile). Since this set of
> files should be in the buffer cache after the first Portfile was
> evaluated the problem is probably not the disk speed.
> 
> This could be made much faster if Tcl had a way to 'clone' an
> interpreter that already had port1.0 loaded before running the rest of
> the Portfile. If somebody wants to implement that, that would probably
> improve portindex speeds significantly.

That may indeed help in the general case. However I know that there are
some portfiles that take a lot longer than others, such as the gcc
ports. I'm not sure why.

If someone wants to work on improving this, the first step is of course
to profile.

- Josh

Re: Allowing PortGroups to bump port revision

[Clemens Lang]

Hi,

On Sun, Jan 14, 2018 at 12:01:34AM +1100, Joshua Root wrote:
> portindex isn't slow, mportopen is. You have to evaluate the changed
> Portfiles before you can extract their info.

I haven't measures this, but I assume the majority of time in mportopen
is spent by the sub-interpreter loading the port1.0 package (i.e.
evaluating the PortSystem 1.0 line in the Portfile). Since this set of
files should be in the buffer cache after the first Portfile was
evaluated the problem is probably not the disk speed.

This could be made much faster if Tcl had a way to 'clone' an
interpreter that already had port1.0 loaded before running the rest of
the Portfile. If somebody wants to implement that, that would probably
improve portindex speeds significantly.

-- 
Clemens

Re: Allowing PortGroups to bump port revision

[René J . V . Bertin]

On Sunday January 14 2018 00:01:34 Joshua Root wrote:

>portindex isn't slow, mportopen is. You have to evaluate the changed
>Portfiles before you can extract their info.

I didn't think disk I/O could be the limiting factor here. (It must be though 
during git pulls of the ports tree, judging from the size of the tree and the 
number of files it contains.)

It's orthogonal to the issue at hand, but maybe it would be worth the effort at 
some point to assess whether the indexing information that is really required 
cannot be extracted  reliably without parsing/running the entire Tcl programme 
that Portfile is. Or if the context cannot be simplified by shunting code 
that's of no interest for indexing. There don't appear to be alternative Tcl 
implementations like there are for Python, right?

Re: Allowing PortGroups to bump port revision

[Joshua Root]

On 2018-1-13 21:14 , René J.V. Bertin wrote:
> Of course this argument begs the question if the portindex command couldn't 
> be made faster, e.g. by porting it to a proper compiled language? No one ever 
> wrote the equivalent of cython or pyrex for Tcl?

portindex isn't slow, mportopen is. You have to evaluate the changed
Portfiles before you can extract their info.

- Josh

Re: Allowing PortGroups to bump port revision

[Ryan Schmidt]

On Jan 13, 2018, at 04:14, René J.V. Bertin wrote:

> Of course this argument begs the question if the portindex command couldn't 
> be made faster, e.g. by porting it to a proper compiled language? No one ever 
> wrote the equivalent of cython or pyrex for Tcl?

Lots of parts of MacPorts are compiled C code instead of Tcl. If the overhead 
of the interpreted Tcl language is a problem for portindex, it could be 
rewritten in C. I'm not convinced that's the case; I assume its speed would be 
limited by the speed of disk accesses.

Re: Allowing PortGroups to bump port revision

[René J . V . Bertin]

>My next thought was that we could add another version field for PortGroup 
>revision.  We could use the highest value set amongst all the port's groups 
>when comparing versions.  That still has the downside of causing problems if a 
>PortGroup is removed. 

Only if the removal causes a revision regression I'd say. 2 ways around that 
(at least): 
- reindex on revision change, not only revision increase
- also keep track of the number of PGs over which the maximum was determined.

> Unfortunately ports are only re-indexed when their mtime changes,

So, when a `portindex` call does seem to catch a version increase from a 
PortGroup in certain ports it is probably because the corresponding Portfiles 
were changed on disk for some other reason, since the last reindexing?

(FWIW, in my case we're talking about the version of a family of KDE ports, so 
I still need to change the checksums in each member port, taking care of the 
required mtime change. I just have to be more alert myself not to forget 
anyone, or remember to use `portindex -f`).

> ... although I don't think we'd be putting this sort of thing in the GitHub
> PortGroup ...

Or the muniversal PG. Until someone does.
And it might be more likely to happen in the libcxx PG, which also seems to be 
used increasingly often.

Of course this argument begs the question if the portindex command couldn't be 
made faster, e.g. by porting it to a proper compiled language? No one ever 
wrote the equivalent of cython or pyrex for Tcl?

R.

Re: Allowing PortGroups to bump port revision

[Joshua Root]

On 2018-1-13 13:13 , Clemens Lang wrote:
> Hi,
> 
> On Fri, Jan 12, 2018 at 03:30:50PM -0800, Jeremy Huddleston Sequoia wrote:
>> In https://trac.macports.org/ticket/54744, we've been pondering adding
>> a PortGroup to allow concurrent installation of multiple providers of
>> the OpenSSL API (eg: openssl, libressl, libressl-devel) and allow
>> ports to specify which they are compatible with.
>>
>> It occurred to me that it would be nice if we could update the
>> PortGroup when one of the dependencies changed their dylib id rather
>> than revbumping all ports.  Is this something anyone else has
>> considered?
> 
> I'd love to use this feature for updates of GHC. For every new version
> of GHC, libraries built with it are installed in a new directory and get
> a new ID, which means that I have to find and revbump every haskell port
> with such an update.
> 
> Unfortunately ports are only re-indexed when their mtime changes, so
> while you could just set revision (or epoch, or some other variable) in
> a PortGroup, that would still not lead to those ports getting reindexed.
> 
> We'd have to keep a map from PortGroup to Portfiles that use this
> PortGroup to modify portindex to correctly re-index ports when a
> PortGroup changes; note that this might also take quite long. I can
> imagine changing the GitHub PortGroup would lead to index update times
> of more than 10 minutes. :(

If I understand the problem correctly, it seems like a better solution
would be maintaining a database of linkage. You could then look up which
dylibs a port provides and which ports link against them. The rev-bump
scripting already exists as mentioned earlier. Automated alerts when a
commit causes broken linking could also be done.

We could also consider adding an archive revision. This would just
increase whenever rev-upgrade is needed, and cause the buildbot to build
a new archive. Clients could probably virtually ignore it and just rely
on rev-upgrade finding the broken linking. That way nobody has to
reinstall anything that isn't broken (e.g. if their chosen variants
don't use the changed dylib.)

- Josh

Re: Allowing PortGroups to bump port revision

[Clemens Lang]

Hi,

On Fri, Jan 12, 2018 at 03:30:50PM -0800, Jeremy Huddleston Sequoia wrote:
> In https://trac.macports.org/ticket/54744, we've been pondering adding
> a PortGroup to allow concurrent installation of multiple providers of
> the OpenSSL API (eg: openssl, libressl, libressl-devel) and allow
> ports to specify which they are compatible with.
> 
> It occurred to me that it would be nice if we could update the
> PortGroup when one of the dependencies changed their dylib id rather
> than revbumping all ports.  Is this something anyone else has
> considered?

I'd love to use this feature for updates of GHC. For every new version
of GHC, libraries built with it are installed in a new directory and get
a new ID, which means that I have to find and revbump every haskell port
with such an update.

Unfortunately ports are only re-indexed when their mtime changes, so
while you could just set revision (or epoch, or some other variable) in
a PortGroup, that would still not lead to those ports getting reindexed.

We'd have to keep a map from PortGroup to Portfiles that use this
PortGroup to modify portindex to correctly re-index ports when a
PortGroup changes; note that this might also take quite long. I can
imagine changing the GitHub PortGroup would lead to index update times
of more than 10 minutes. :(

-- 
Clemens

Re: Allowing PortGroups to bump port revision

[René J . V . Bertin]

On Friday January 12 2018 15:30:50 Jeremy Huddleston Sequoia wrote:

> It occurred to me that it would be nice if we could update the PortGroup when 
> one of the dependencies changed their dylib id rather than revbumping all 
> ports.  Is this something anyone else has considered?

I don't think I really understand what you mean, could you give an example?

If you mean something akin to setting the `version` in a PortGroup, this 
doesn't always work reliably. I have such a PG and it seems there are always 
some ports that don't show up in `port outdated` after I bump the version 
(presumably because PortIndex can fail to detect that a port imports certain 
PGs).

If you do think of setting a revision in the PortGroup: how do you propose 
coping with the fact that dependents may set their own revision, typically 
after including the PortGroup?

R.

Re: Allowing PortGroups to bump port revision

[Daniel J. Luke]

On Jan 12, 2018, at 6:30 PM, Jeremy Huddleston Sequoia  
wrote:
> In https://trac.macports.org/ticket/54744, we've been pondering adding a 
> PortGroup to allow concurrent installation of multiple providers of the 
> OpenSSL API (eg: openssl, libressl, libressl-devel) and allow ports to 
> specify which they are compatible with.
> 
> It occurred to me that it would be nice if we could update the PortGroup when 
> one of the dependencies changed their dylib id rather than revbumping all 
> ports.  Is this something anyone else has considered?

I've suggested this in the past (for p5 ports, specifically) that we could 
(ab)use revision to avoid having to rev-bump many ports. (I suggested we use a 
date-string so that individual ports that used the port-group could still set a 
higher revision if they needed to - but we'd still probably have to work out 
something to handle the case where a port wanted to set a revision and then 
later we wanted to update the revision in the portgroup). 

IIRC Ryan /hated/ that idea.

Alternatively, we could add something to base to let us specify this kind of 
dependency (which would be useful).

-- 
Daniel J. Luke

Allowing PortGroups to bump port revision

[Jeremy Huddleston Sequoia]

In https://trac.macports.org/ticket/54744, we've been pondering adding a 
PortGroup to allow concurrent installation of multiple providers of the OpenSSL 
API (eg: openssl, libressl, libressl-devel) and allow ports to specify which 
they are compatible with.

It occurred to me that it would be nice if we could update the PortGroup when 
one of the dependencies changed their dylib id rather than revbumping all 
ports.  Is this something anyone else has considered?

--Jeremy