Re: Developer mode

[Steven Smith]

This would be very helpful.

I often have to rerun failed build commands by hand to troubleshoot problems.

Also, I’d note the following comments about the use of color to provide visual 
context for quickly parsing output. Both git and brew use color effectively.

Adding a feature, say, to color errors in red, would allow developers to 
quickly pinpoint problems, rather than scanning for the word “error,” which can 
be nontrivial if one is using tools over a mobile device or through a VNC 
connection. Or using color to visually highlight different port phases.

https://saagarjha.com/blog/2019/04/26/thoughts-on-macos-package-managers/
> Homebrew aims to be simple and easy to use, and in general this is true. 
> While I’m neutral on the use of beer-related terminology and emoji, the 
> judicious use of color is quite helpful, as is the well-formatted output. 
> Most common usecases are straightforward, though occasionally commands can 
> end up doing surprising things if you use them in strange ways.
> 
> MacPorts, on the other hand, is by no means unusable, but it’s not as 
> polished as Homebrew is. The lack of color, as well as somewhat more 
> cluttered and less relevant output, makes it a bit less pleasant to work with


smime.p7s
Description: S/MIME cryptographic signature

Re: Developer mode

[Ryan Schmidt]

On Nov 19, 2019, at 18:21, Ralph Seichter wrote:

> * Ryan Schmidt:
> 
>> We might also add a way for a developer to indicate for which ports
>> developer mode should be turned on. A developer might wish, for
>> example, to be notified of issues relating to the ports they maintain
>> but not the ports that others maintain.
> 
> That can be useful. Is that not something Trac should handle? If a
> ticket references a specific port, is the designated maintainer not
> automatically added to the ticket?

As Rainer said, no, Trac does not auto-assign tickets to the ports' 
maintainers, but I'm not talking about tickets. I'm talking about deprecated 
features in MacPorts where we would like maintainers to migrate to more modern 
equivalents. We do some of this in `port lint` already (for example we already 
advise that `eval [glob ...]` is deprecated and should be replaced with 
`{*}...`) and could do more there (like recommending replacing `PortGroup cxx11 
1.1` with `compiler.cxx_standard 2011`), but developers have to run lint 
explicitly, and I'll bet that usually they don't (I don't) and so some of the 
recommendations made by lint go unheeded for years. I am proposing that 
MacPorts base should perform lint and other checks as the port is being used by 
its maintainer and alert the maintainer about any problems so that the 
maintainer can take care of it without anyone needing to file a ticket about it.

Some checks can't be done at lint time. For example, I have an experimental 
branch of base that warns if the port may be using a configure script that 
suffers from the "Yosemite libtool" bug (where OS X 10.10 and later are 
misidentified as Mac OS X 10.1). I am not personally going to examine all of 
the over 23,000 ports right now to see which of them suffer from this problem 
and file tickets for each of them; I don't have the time to do that. Instead, I 
am proposing that messages about this type of problem would be shown at the new 
"developer mode" output level which developers could opt into.

Fwd: Developer mode

[Georges Martin]

> Début du message réexpédié :
> 
> De: Georges Martin 
> Objet: Re : Developer mode
> Date: 30 novembre 2019 à 14:55:11 UTC+1
> À: Rainer Müller 
> 
> 
>> Le 26 nov. 2019 à 16:05, Rainer Müller  a écrit :
>> 
>> Sounds very useful to have such a tool. But as always, this can just be 
>> written
>> and does not necessarily have to be part of port(1). The same thing happened 
>> in
>> Gentoo, many tools were just written independently. I am just mentioning 
>> this to
>> encourage contributions for such quality-of-life scripts that do not rely on
>> becoming more familiar with macports-base first.
> 
> OK, in that spirit, I published two of my MacPorts scripts at 
> https://github.com/jrjsmrtn/macports-scripts.
> 
> 1. `mp-check` will perform a livecheck and a search of open Trac tickets for 
> a maintainer’s ports.
> 
> By default, only the maintainer’s ports are searched but additional ones can 
> be given with the « include-ports » option. Subports can also be excluded 
> from livechecks. (An « exclude-ports » option exists but is buggy at this 
> time).
> 
> With the following `~/.macports/mp-check.conf` config file:
> 
>  [DEFAULT]
>  maintainer: jrjsmrtn
>  include-ports:
>  postgresql-jdbc
> 
>  [Livecheck]
>  exclude-subports: true
> 
>  [Tickets]
>  include-ports:
>  postgresql11, postgresql11-server,
>  postgresql12, postgresql12-server,
>  python38
> 
> A `mp-check` will return:
> 
> ---> Livecheck
> agensgraph2 seems to be up to date
> cl-quicklisp seems to be up to date
> flash seems to be up to date
> postgresql-jdbc seems to be up to date
> py-zmq seems to be up to date
> wpa_passphrase seems to be up to date
> 
> ---> Tickets
>   PortSummaryStatus   
>  Type
> id
> 59518  python38  python38's distutils.sysconfig contains the wr...   new  
> defect
> 59772  python38  python38: fails on Leopard: Undefined symbols:...  assigned  
> defect
> 43272py-zmq  py33-zmq @14.1.1 +universal and arch -i386 ipy...   new  
> defect
> 54845py-zmq   py-zmq: problematic dependencies   new  
> defect
> 
> 
> 2. `mp-clean-moved` will remove all `*.mp_*` files from the MacPorts 
> directories. Those are typically created when you have to force activate PyPI 
> Python ports.
> 
> 
> If anybody wants to add those in MacPorts, be my guest :-)
> 
> 
> Enjoy,
> 
> Georges

Re: Developer mode

[Rainer Müller]

On 20/11/2019 01.21, Ralph Seichter wrote:
> * Ryan Schmidt:
> 
>> We might also add a way for a developer to indicate for which ports
>> developer mode should be turned on. A developer might wish, for
>> example, to be notified of issues relating to the ports they maintain
>> but not the ports that others maintain.
> 
> That can be useful. Is that not something Trac should handle? If a
> ticket references a specific port, is the designated maintainer not
> automatically added to the ticket?

Unfortunately we do not have any plugin in Trac to assign maintainers
automatically. There was a previous attempt at getting this into Trac a few
years ago, but it got not deployed due to constraints at macOS forge, which we
used for hosting back then.

https://trac.macports.org/ticket/40987

I am not sure if that code is still compatible with the more recent Trac
version... It would probably also need to be adapted to the PortIndex SQL schema
(or whatever the webapp is now using?).

>> You mentioned updating manifests. I don't know what a manifest is but
>> MacPorts ports only have a Portfile. We did recently introduce the
>> "port bump" developer command to assist with updating checksums
> 
> While MacPorts lists distribution archive checksums as part of each
> Portfile, Gentoo uses a separate Manifest file for that (one reason
> being that Gentoo offers multiple versions per ebuild, hence multiple
> checksums are required). "repoman manifest" adds new checksums and/or
> removes obsolete ones, for convenience. If "port bump" does the same,
> that's news to me, but a nice feature I have been waiting for.

Yes, 'port bump' is supposed to edit the Portfile with the new checksums.

>> We don't have a tool to automate commit messages. We expect developers
>> to write them manually, including referencing any related Trac
>> tickets. I'm not sure I understand how an automated tool could know
>> which ticket(s), if any, a commit is related to.
> 
> Let's use https://trac.macports.org/ticket/58921 as an example, a ticket
> I opened and you processed and closed (thanks for the update by the way).
> If this was a Gentoo ticket, you could have used the following:
> 
>   $ cd /path/to/your/ports/mongodb
>   $ repoman -c 58921 commit
> 
> This would have created the following type of Git commit message
> template and opened it in your editor:
> 
>   dev-db/mongodb:
> 
>   Closes: https://trac.macports.org/ticket/58921
>   Package-Manager: Portage-2.3.79, Repoman-2.3.18
>   Signed-off-by: Ryan Schmidt 
> 
> I am aware that MacPorts does not use sign-off lines, this is just an
> example of how repoman pre-populates commit messages. Also, the prefix
> "dev-db/mongodb:" is Gentoo-specific, it would just be "mongodb:" for
> MacPorts. The point is that repoman will choose the correct prefix for
> the user. The option "-c" adds the correct "Closes" line for a given
> ticket number, while "-b 58921 -b 1234" would add 
> 
>   Bugs: https://trac.macports.org/ticket/58921
>   Bugs: https://trac.macports.org/ticket/1234
> 
> i.e. non-closing references to existing tickets. As I mentioned before,
> repoman also performs a "port lint" equivalent. It is simply a tool to
> make recurring operations like basic QA checks and commits more
> convenient for the user (it does for example report local untracked
> files which you may have forgotten to add). The user is of course still
> responsible for providing useful commit messages, both short summary and
> verbose description.

Sounds very useful to have such a tool. But as always, this can just be written
and does not necessarily have to be part of port(1). The same thing happened in
Gentoo, many tools were just written independently. I am just mentioning this to
encourage contributions for such quality-of-life scripts that do not rely on
becoming more familiar with macports-base first.

Rainer

Re: Developer mode

[Ralph Seichter]

* Ryan Schmidt:

> We might also add a way for a developer to indicate for which ports
> developer mode should be turned on. A developer might wish, for
> example, to be notified of issues relating to the ports they maintain
> but not the ports that others maintain.

That can be useful. Is that not something Trac should handle? If a
ticket references a specific port, is the designated maintainer not
automatically added to the ticket?

> You mentioned updating manifests. I don't know what a manifest is but
> MacPorts ports only have a Portfile. We did recently introduce the
> "port bump" developer command to assist with updating checksums

While MacPorts lists distribution archive checksums as part of each
Portfile, Gentoo uses a separate Manifest file for that (one reason
being that Gentoo offers multiple versions per ebuild, hence multiple
checksums are required). "repoman manifest" adds new checksums and/or
removes obsolete ones, for convenience. If "port bump" does the same,
that's news to me, but a nice feature I have been waiting for.

> We don't have a tool to automate commit messages. We expect developers
> to write them manually, including referencing any related Trac
> tickets. I'm not sure I understand how an automated tool could know
> which ticket(s), if any, a commit is related to.

Let's use https://trac.macports.org/ticket/58921 as an example, a ticket
I opened and you processed and closed (thanks for the update by the way).
If this was a Gentoo ticket, you could have used the following:

  $ cd /path/to/your/ports/mongodb
  $ repoman -c 58921 commit

This would have created the following type of Git commit message
template and opened it in your editor:

  dev-db/mongodb:

  Closes: https://trac.macports.org/ticket/58921
  Package-Manager: Portage-2.3.79, Repoman-2.3.18
  Signed-off-by: Ryan Schmidt 

I am aware that MacPorts does not use sign-off lines, this is just an
example of how repoman pre-populates commit messages. Also, the prefix
"dev-db/mongodb:" is Gentoo-specific, it would just be "mongodb:" for
MacPorts. The point is that repoman will choose the correct prefix for
the user. The option "-c" adds the correct "Closes" line for a given
ticket number, while "-b 58921 -b 1234" would add 

  Bugs: https://trac.macports.org/ticket/58921
  Bugs: https://trac.macports.org/ticket/1234

i.e. non-closing references to existing tickets. As I mentioned before,
repoman also performs a "port lint" equivalent. It is simply a tool to
make recurring operations like basic QA checks and commits more
convenient for the user (it does for example report local untracked
files which you may have forgotten to add). The user is of course still
responsible for providing useful commit messages, both short summary and
verbose description.

Sorry if this moved a bit away from the original developer mode
subject. I just work a lot more in Gentoo than in MacPorts, and I always
miss repoman's watchful presence when I work with Portfiles. ;-)

-Ralph

Re: Developer mode

[Ryan Schmidt]

> On Nov 18, 2019, at 14:32, Ralph Seichter wrote:

> * Ryan Schmidt:
> 
>> I wonder if we should add a "developer mode" to MacPorts -- something
>> that developers could enable in macports.conf.
> 
> I for one would like to be able to peek into what is happening under the
> hood when working with Portfiles, so increased logging of technical
> details sounds useful to me.

We do already have debug mode (the -d flag), but unfortunately it is *very* 
verbose.

> That includes deprecation warnings,
> suggested update paths, etc.

And debug mode does not include this information. But if we added a developer 
mode, and started adding such developer-mode-level logging, surely we would 
also include it in debug-level logging.


We might also add a way for a developer to indicate for which ports developer 
mode should be turned on. A developer might wish, for example, to be notified 
of issues relating to the ports they maintain but not the ports that others 
maintain.


>> In "developer mode", "port lint" or other checks could be run as part
>> of every install.
> 
> Gentoo Linux provides a tool called 'repoman' [1] that is similar to
> 'port lint' in some aspects. It also provides convenience functions like
> updating manifests (mostly source tarball checksums) and a wrapper for
> VCS commits that pre-populates commit messages with the ebuild name
> (similar to Port name), sign-off lines, and links to referenced/closed
> issues in Gentoo's Bugzilla. I find repoman to be very useful, and if
> the developer mode you suggested comes with similar utility, I would
> welcome it.
> 
> [1] https://wiki.gentoo.org/wiki/Repoman

I'm not familiar with Gentoo Linux or Repoman and was not suggesting the 
introduction of any new programs or port subcommands. I was just suggesting 
that MacPorts should add a new logging level, which developers might opt in to 
having displayed on the terminal automatically, and where we could begin to add 
logging messages that would be helpful to developers to keep their ports up to 
date with our current best practices.

Not being familiar with it, I'm not sure if what repoman does is applicable to 
MacPorts. You mentioned updating manifests. I don't know what a manifest is but 
MacPorts ports only have a Portfile. We did recently introduce the "port bump" 
developer command to assist with updating checksums:

https://man.macports.org/port-bump.1.html

We don't have a tool to automate commit messages. We expect developers to write 
them manually, including referencing any related Trac tickets. I'm not sure I 
understand how an automated tool could know which ticket(s), if any, a commit 
is related to. We don't use sign-off lines in commit messages. We do have 
documentation about how we want commit messages to be written:

https://trac.macports.org/wiki/CommitMessages

Re: Developer mode

[Ralph Seichter]

* Ryan Schmidt:

> I wonder if we should add a "developer mode" to MacPorts -- something
> that developers could enable in macports.conf.

I for one would like to be able to peek into what is happening under the
hood when working with Portfiles, so increased logging of technical
details sounds useful to me. That includes deprecation warnings,
suggested update paths, etc.

> In "developer mode", "port lint" or other checks could be run as part
> of every install.

Gentoo Linux provides a tool called 'repoman' [1] that is similar to
'port lint' in some aspects. It also provides convenience functions like
updating manifests (mostly source tarball checksums) and a wrapper for
VCS commits that pre-populates commit messages with the ebuild name
(similar to Port name), sign-off lines, and links to referenced/closed
issues in Gentoo's Bugzilla. I find repoman to be very useful, and if
the developer mode you suggested comes with similar utility, I would
welcome it.

[1] https://wiki.gentoo.org/wiki/Repoman

-Ralph


P.S.: Output of 'repoman --help' in the current version:

usage: repoman [options] [mode]

Modes: ci | commit | fix | full | help | manifest | manifest-check | scan

optional arguments:
  -h, --helpshow this help message and exit
  -a, --ask Request a confirmation before commiting
  -b , --bug 
Mention a Gentoo or upstream bug in the commit footer;
takes either Gentoo bug number or full bug URL
  -c , --closes 
Adds a Closes footer to close GitHub pull request (or
compatible); takes either GitHub PR number or full PR
URL
  -m COMMITMSG, --commitmsg COMMITMSG
specify a commit message on the command line
  -M COMMITMSGFILE, --commitmsgfile COMMITMSGFILE
specify a path to a file that contains a commit
message
  --digest Automatically update Manifest digests for modified
files
  -p, --pretend don't commit or fix anything; just show what would be
done
  -q, --quiet   do not print unnecessary messages
  --echangelog 
for commit mode, call echangelog if ChangeLog is
unmodified (or regardless of modification if 'force'
is specified)
  --experimental-inherit 
Enable experimental inherit.missing checks which may
misbehave when the internal eclass database becomes
outdated
  --experimental-repository-modules 
Enable experimental repository modules
  -f, --force   Commit with QA violations
  -S, --straight-to-stable
Allow committing straight to stable
  --vcs VCS Force using specific VCS instead of autodetection
  -v, --verbose be very verbose in output
  -V, --version show version info
  -x, --xmlparseforces the metadata.xml parse check to be carried out
  --if-modifiedonly check packages that have uncommitted
modifications
  -i, --ignore-arches   ignore arch-specific failures (where arch != host)
  --ignore-default-opts
do not use the REPOMAN_DEFAULT_OPTS environment
variable
  -I, --ignore-masked   ignore masked packages (not allowed with commit mode)
  --include-arches ARCHES
A space separated list of arches used to filter the
selection of profiles for dependency checks
  -d, --include-dev include dev profiles in dependency checks
  -e , --include-exp-profiles 
include exp profiles in dependency checks
  --unmatched-removal   enable strict checking of package.mask and
package.unmask files for unmatched removal atoms
  --without-maskbehave as if no package.mask entries exist (not
allowed with commit mode)
  --output-style {column,default}
select output type
  --mode {ci,commit,fix,full,help,manifest,manifest-check,scan}
specify which mode repoman will run in (default=full)

For more help consult the man page.

Re: Developer mode

[Jackson Isaac]

Hi Ryan,

On Mon, Nov 18, 2019 at 8:57 PM Ryan Schmidt  wrote:
>
> I wonder if we should add a "developer mode" to MacPorts -- something that 
> developers could enable in macports.conf.

I like this idea.

>
> I feel that currently we print some information that is meant for developers 
> and is not helpful to regular users -- such as the "reinplace didn't change 
> anything" or "all compilers were blacklisted" warnings.
>
> And there is a lot more information that we could print that would be useful 
> to developers but that we don't print because it would be overwhelming to 
> regular users. We introduce new features, such as "compiler.cxx_standard 
> 2011", meant to replace old methods like "PortGroup cxx11 1.1", yet we have 
> no way of nudging developers to adopt these new features, short of the 
> laborious process of manually searching out the ports and filing tickets or 
> fixing it ourselves. If MacPorts could automatically nudge developers about 
> these issues as they're updating their own ports, that could be helpful.
>
> In "developer mode", "port lint" or other checks could be run as part of 
> every install.
>
> With this switch in place, over time, we could further streamline the regular 
> non-developer user experience further, such as by reducing amount of (to the 
> user probably irrelevant) output produced by installing or upgrading a port.
>

-- 
Jackson Isaac

Developer mode

[Ryan Schmidt]

I wonder if we should add a "developer mode" to MacPorts -- something that 
developers could enable in macports.conf.

I feel that currently we print some information that is meant for developers 
and is not helpful to regular users -- such as the "reinplace didn't change 
anything" or "all compilers were blacklisted" warnings.

And there is a lot more information that we could print that would be useful to 
developers but that we don't print because it would be overwhelming to regular 
users. We introduce new features, such as "compiler.cxx_standard 2011", meant 
to replace old methods like "PortGroup cxx11 1.1", yet we have no way of 
nudging developers to adopt these new features, short of the laborious process 
of manually searching out the ports and filing tickets or fixing it ourselves. 
If MacPorts could automatically nudge developers about these issues as they're 
updating their own ports, that could be helpful.

In "developer mode", "port lint" or other checks could be run as part of every 
install.

With this switch in place, over time, we could further streamline the regular 
non-developer user experience further, such as by reducing amount of (to the 
user probably irrelevant) output produced by installing or upgrading a port.