Re: nmh 1.8RC2, xlbiff, and $HOME

[David Levine]

Ralph wrote:

> Your principles soon went out the window.

Oh no, they're all still there.  One of them just got moved to the front.

David

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi David,

> Then I think that we should do the same in 1.8 for all platforms.

Your principles soon went out the window.

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[David Levine]

az wrote:

> minor data point: after following the discussion here, i've decided
> that for now _the debian packages_ of nmh should not distinguish
> between unset $HOME and set-but-empty $HOME, and that either should cause a
> fallback to getpw*.

Then I think that we should do the same in 1.8 for all platforms.

Thoughts on applying that patch (plus a test and documentation update)
to create a 1.8RC3 this weekend?

David

Re: nmh 1.8RC2, xlbiff, and $HOME

[Alexander Zangerl]

On Thu, 02 Feb 2023 16:44:05 -0800, David Levine writes:
>Anyways, I'd like to keep the 1.7
>behavior for 1.8 and change it after.  Can we agree to that?

minor data point: after following the discussion here, i've decided
that for now _the debian packages_ of nmh should not distinguish
between unset $HOME and set-but-empty $HOME, and that either should cause a
fallback to getpw*.

(1.8RC2 will hopefully/likely be part of the upcoming debian
release; the previous release shipped with 1.7.1.)

regards
az

Description: treat set-but-empty $HOME same as unset
 upstream code aborts on empty $HOME and falls back to getpw* only on unset $HOME,
 which is undesirable (affects xlbiff's testsuite and generally surprises end users).
 this patch makes it fall back for both empty and unset $HOME.
Author: Alexander Zangerl 
Bug: https://bugs.debian.org/1029752
Last-Update: 2023-02-02
--- a/sbr/path.c
+++ b/sbr/path.c
@@ -38,10 +38,7 @@ void
 set_mypath(void)
 {
 char *var = getenv("HOME");
-if (var) {
-if (!*var)
-die("environment variable HOME is empty");
-
+if (var && *var) {
 mypath = mh_xstrdup(var);
 return;
 }
-- 
Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + https://snafu.priv.at/
He who joyfully marches to music in rank and file has already earned my 
contempt. He has been given a large brain by mistake, since for him the 
spinal cord would fully suffice. -- Einstein


signature.asc
Description: Digital Signature

Re: nmh 1.8RC2, xlbiff, and $HOME

[David Levine]

I wrote:

> What case am I missing?

I get it: the non-null, empty HOME.  Anyways, I'd like to keep the 1.7
behavior for 1.8 and change it after.  Can we agree to that?

David

Re: nmh 1.8RC2, xlbiff, and $HOME

[David Levine]

Ralph wrote:

> Hi David,
>
> > And my feeling at this point is to put out an RC3 with the 1.7
> > behavior for $HOME.
>
> Which, to be clear, is not the behaviour Ken and kre want.  :-)

I was thinking that just removing the if (!*var) statement from your
code, leaving:

char *var = getenv("HOME");
if (var) {
mypath = mh_xstrdup(var);
return;
}

errno = 0;
struct passwd *pw = getpwuid(getuid());
if (!pw) {
if (errno)
adios("", "getpwuid() failed");   /* "" prints errno! */
die("password entry not found");
}
if (!*pw->pw_dir)
die("password entry has empty home directory");

mypath = mh_xstrdup(pw->pw_dir);

is the same as the 1.7 code (ignoring the string copies):

if ((mypath = getenv("HOME")) == NULL) {
if ((pw = getpwuid(getuid())) == NULL || *pw->pw_dir == '\0')
adios(NULL, "cannot determine your home directory");
else
mypath = pw->pw_dir;
}

What case am I missing?

David

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi David,

I've had one night's sleep spread over the last two nights so I'm not
going to have time for nmh until the weekend.

> And my feeling at this point is to put out an RC3 with the 1.7
> behavior for $HOME.

Which, to be clear, is not the behaviour Ken and kre want.  :-)


Stephen, I'd appreciate knowing if unsetting HOME with RC2,
e.g. ‘env -u HOME ...’, is sufficient for xlbiff's test harness.

$ HOME= MH=$PWD/mh_profile mhparam path
mhparam: environment variable HOME is empty
$
$ env -u HOME MH=$PWD/mh_profile mhparam path
/tmp

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[David Levine]

Ralph wrote:

> The behaviour cannot solidify further.  :-)  It's clear, determinate,
> predictable, and simple to document.

Those qualities can be true for the 1.7 behaviour.

> I thought RCn+1 was meant to be minimal differences from RCn?

If there's an RC3, the only code difference would be for this issue.

> That's why Andy Bradford's POP3 patch is post 1.8.

There's a key difference between the $HOME change and Andy's POP3
patch:  the behaviour has already, prior to 1.8 final release,
changed from 1.7 for the former but not the latter.

> Switching the behaviour of
> a set-but-empty HOME now seems significant given some value for HOME
> must be chosen.

1.8 hasn't been released yet.  I think that we owe users consistent
behaviour from 1.7 to 1.8 unless clearly documented.

> And will mean another round of testing by others on top of your time.

I appreciate your consideration of that but I think that we need to
resolve the consistency issue.  In any case, a new test and man page
update are called for.

And my feeling at this point is to put out an RC3 with the 1.7 behavior
for $HOME.

David

Re: nmh 1.8RC2, xlbiff, and $HOME

[Robert Elz]

Date:Wed, 01 Feb 2023 14:54:36 +
From:Ralph Corderoy 
Message-ID:  <20230201145436.1aa5222...@orac.inputplus.co.uk>

  | My stance has nothing to do with satisfying ‘user expectations’ so I'll
  | ignore this aspect.

Ralph, do you not see the contradiction between that, and:

  | If 3 is implemented then the user may have thought a different pretence
  | would occur.

which is exactly a desire to satisfy user expectations, and when you
don't know what they are, refuse to do anything.

There is plenty in nmh that doesn't act (by default) to satisfy my
expectations, but that's OK, it works as documented, and there is
generally a method by which I can achieve what I want.

Why does this need to be different?

  | After all, it's an arbitrary choice.  Perhaps he expects it
  | will carry on just like his shell does.

Perhaps.   But we cannot meet that objective, as shells do different
things, there are many to choose from, but there is just one nmh (well
two, if you count the mmh I recall hearing of).   Do you think we should
create nmh forks so users can pick the one that meets all their expectations
by default?

  | This could result in surprising files being read, e.g. /tmp/.mh_profile,

Yes, if someone's pw_dir is /tmp ... but then that is what it should do.
No-one is suggesting treating an empty HOME literally (as "") in this
context, nor as "." or anything odd - just treat it the same as an unset
HOME.   If the user doesn't like that, then HOME should be set to whatever
they want it to be.   Simple.  Clean.   Easy to document.

  | mh_profile(5) can easily say HOME must be non-empty if set for those
  | that read the fine manual.

It can.   But what's the point?

  | No, you must decide which of the many alternatives of 3 above is the
  | ‘nmh way’, document that, test it, and have some users be surprised by
  | the result.

We have already had users (the xlbiff test suite) be surprised by the
current result - which was not only unexpected, it varied from earlier
behaviour.

  | Ken, you, and David all seem irked by unset and set-but-empty being
  | treated differently, as if you'd like a binary outcome by first
  | conflating the ternary input to binary.  There isn't a clear method of
  | merging those two inputs.  Ternary's messy, as ever.  We can't stop HOME
  | being unset or being empty so there are three safe outcomes for three
  | inputs.

That's solely because that's the way you see it.   It is entirely
possible to treat two different settings (such as +0 and -0 in one's
complement arithmetic) as meaning the same thing in one context, such
as they do when using normal + - * / operations, yet a different one
in a different context (like & | ^ bit operations).   That's just the
way things work.   Treating an unset HOME and an empty but set HOME
the same in nmh is a perfectly valid choice, and seems to be the
preferred one.

kre

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ken Hornstein]

>Ken, you, and David all seem irked by unset and set-but-empty being
>treated differently, as if you'd like a binary outcome by first
>conflating the ternary input to binary.

Sigh.  I'll try to make my point clearer, but I recognize we're not going
to agree on this.  Since David is driving this release, I defer to his
decision in terms of changing the behavior for the next release candidate
(I would vote for making the behavior of unset and empty be the same, but
I consider David the God Emperor of nmh 1.8, so it's up to him).

You ask the question:

What is the user trying to achieve by explicitly setting an empty HOME? 

To me, the answer is obvious: they were trying to create a clean
environment.  You say that they could do that by unsetting HOME; well,
count me in the pool of people who did not know until now that 'unset'
even existed.  I mean, I vaguely knew that there was a difference
between a variable that was never used and one that was set to an empty
string, but as far as I could tell there wasn't a PRACTICAL difference
between those two states and I certainly wouldn't expect an application
to treat those states differently.  I suspect the person who wrote that
test for xlbiff didn't know about 'unset' either and would be bewildered
that there is any difference in behavior.  Also, some quick testing suggests
that:

% HOME='' command

Does the expected thing but:

% unset HOME command

Simply unsets the HOME and 'command' variables.  I realize you could do
that in a subshell but it just seems awkward, and the idiom of doing

% FOO=value command

to change an environment variable is very common and I see why people
would just use that.

I have read your arguments carefully and ... well, I still do not agree
that there should be a difference in behavior between an unset HOME and
an empty HOME.  To me there is no ambiguity and they are the same, even
if it's a technically different variable state.  Again, I realize we're
just not going to agree on this.

--Ken

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi,

I wrote:
> If 3 is implemented then the user may have thought a different
> pretence would occur.  After all, it's an arbitrary choice.  Perhaps
> he expects it will carry on just like his shell does.

This has overlap with DWIM, ‘do what I mean’, and Perl's TMTOWTDI,
‘there's more than one way to do it’.

DWIM obviously gets one tied to chair.
http://www.jargon.net/jargonfile/d/DWIM.html

I'm a fan of Perl, programming a lot of it in Perl 4 and early 5 days,
but having introduced it to a large site of engineers who did a bit of
programming, I came to see TMTOWTDI as a problem.  I could read all
their code because I knew all the ways.  They wrote in dialects by using
different subsets, reducing re-use and understanding.

Many saw TMTOWTDI as a problem.  It's one reason why Python has boomed.
Amongst its ‘import this’ Zen are

Explicit is better than implicit.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.

An empty HOME isn't explicit in its meaning, it's ambiguous, we
shouldn't guess, and the obvious way of specifying the home directory is
to set HOME to it.

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi David,

> > This close to a release, I think we should stick with requiring HOME
> > to be non-empty if it's set as otherwise there's too many paths to
> > consider which the test harness probably doesn't exercise.
>
> I'd rather crank out an RC3 than pass up the opportunity to solidify
> the behaviour here.

The behaviour cannot solidify further.  :-)  It's clear, determinate,
predictable, and simple to document.

I thought RCn+1 was meant to be minimal differences from RCn?  That's
why Andy Bradford's POP3 patch is post 1.8.  Switching the behaviour of
a set-but-empty HOME now seems significant given some value for HOME
must be chosen.  And will mean another round of testing by others on top
of your time.

> Ken's point that different treatmet of unset vs empty HOME is a mistake
> is fairly compelling for me.

I've covered that in my reply to kre just now.

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi kre,

> There is never a need to deliberately make anything an error to
> satisfy user expectations, if the user wants a complaint about HOME
> being an empty string, rather than some other behaviour, all they need
> is one of...

My stance has nothing to do with satisfying ‘user expectations’ so I'll
ignore this aspect.

> > No, one cannot say of an unset HOME that it may be set by accident.
>
> No, but it may have been unset by accident, where the intended value
> for mh to use was something different than the pw_dir entry.   No-one
> can possibly know in that case, any more than in the case where
> HOME=''

(I think you've over-cropped in general.)

There is a marked difference which I thought I had made very clear.

There are three cases.  All may be accidental or deliberate, obviously.

1.
- HOME is unset.
- This is valid, i.e. POSIX allows it.
- The only route for continuing is it to use a mandatory passwd.pw_dir.
- HOME=passwd.pw_dir results.

2.
- HOME is set and non-empty.
- Use it.
- If that causes errors then die, do not try again with passwd.pw_dir.
- HOME=$HOME results.

3.
- HOME is set but empty.
- There are at least three ways of continuing.
- Pretend HOME=$PWD
- Pretend HOME=/
- Pretend HOME=passwd.pw_dir
- Other alternatives include trying HOME=... until one ‘works’.
- HOME=??? results.

There is no obvious way to continue for 1 and 2 other than the above.
The result is not surprising.  Even if the manual isn't read.

If 3 is implemented then the user may have thought a different pretence
would occur.  After all, it's an arbitrary choice.  Perhaps he expects it
will carry on just like his shell does.

This could result in surprising files being read, e.g. /tmp/.mh_profile,
and unwanted commands being run, e.g. through a profile's ‘moreproc’ or
‘mhshow-show-text/plain’.  (top(1) had a similar issue.  It used $PWD
when HOME was unset or empty allowing a privilege escalation through its
configuration file; cue CVE.)

Turning it about.  What is the user trying to achieve by explicitly
setting an empty HOME?  Whatever it is, he can achieve it in an
*unambiguous* manner by unsetting HOME or setting it to a non-empty
value.  It seems you're arguing for him to instead first refer to the
manual to determine what set-but-empty means to nmh in the hope it
matches his need when he could just take and explicit short-cut?

> There is always ambiguity if you are doing something different than
> what is documented to happen.

That is not ambiguity.  That is error.

mh_profile(5) can easily say HOME must be non-empty if set for those
that read the fine manual.

> The question is what is best for nmh - and for that I agree with Ken,
> treating an unset HOME and HOME='' differently is a mistake.
> All that is required to make that OK is to document it.

No, you must decide which of the many alternatives of 3 above is the
‘nmh way’, document that, test it, and have some users be surprised by
the result.  There is no ‘All that...’.

Ken, you, and David all seem irked by unset and set-but-empty being
treated differently, as if you'd like a binary outcome by first
conflating the ternary input to binary.  There isn't a clear method of
merging those two inputs.  Ternary's messy, as ever.  We can't stop HOME
being unset or being empty so there are three safe outcomes for three
inputs.

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[David Levine]

Ralph wrote:

> This close to a release, I think we should stick with requiring HOME to
> be non-empty if it's set as otherwise there's too many paths to consider
> which the test harness probably doesn't exercise.

I'd rather crank out an RC3 than pass up the opportunity to solidify
the behaviour here.

Ken's point that different treatmet of unset vs empty HOME is a mistake
is fairly compelling for me.

David

Re: nmh 1.8RC2, xlbiff, and $HOME

[Steffen Nurpmeso]

Ralph Corderoy wrote in
 <20230131181958.1cfb121...@orac.inputplus.co.uk>:
 ...
 |But if HOME is empty we do not know their intent so to ignore it and use
 |pw_dir may not be what they think will occur.  The wrong profile could
 |be read or the wrong .netrc used, upsetting the user.

By the way my mailer does:

  ? xv var HOME
  #nodelete,import-environ-first,sync-environ,notempty:
set HOME=/home/steffen

(where xv temporarily enables *verbose* mode).
If i recall correctly you, once you were still taking part, have
been dissatisfied with TOCTOU issues for TMPDIR for which the same
holds:

  ? xv var TMPDIR
  #default-value,import-environ-first,sync-environ,notempty:
set TMPDIR=/tmp

So i (fwiw) are in the camp

  $ HOME= s-nail -R#
  s-nail: Environment variable must not be empty: HOME
  s-nail: There are messages in the error ring, manageable via `errors' command
  #?0!0/NONE#ERROR|:/dev/null? var HOME
  set HOME=/home/steffen
  #?0!0/NONE#ERROR|:/dev/null? xit

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

Re: nmh 1.8RC2, xlbiff, and $HOME

[Robert Elz]

Date:Tue, 31 Jan 2023 13:29:46 +
From:Ralph Corderoy 
Message-ID:  <20230131132946.44a5f20...@orac.inputplus.co.uk>


  | > Similarly, in XCU 4 in the description of the cd utility:

  | Yes, but also allowed there is ‘empty’ which also triggers
  | implementation-defined.

Yes, that's because of:

  | Also, POSIX's chdir(3) says
  |int chdir(const char *path);
  | ERRORS
  |The chdir() function shall fail if:
  |...
  |ENOENT  A component of path does not name an existing
  |directory or path is an empty string.

and since some shells simply do effectively
chdir(getenv(HOME));
in this case, they get an error, if HOME is unset, or empty.

Others check the value from getenv(HOME) and do something different
if it is NULL, and still others go further and check for "" as the
value, and either treat that the same as a NULL result, or in some
other way that suits them.

Note that POSIX documents what users can expect to have happen (what
can be relied upon) and only rarely mandates some specific behaviour
that the people who maintain it believe should happen, rather than
what actually does.   If HOME is unset or empty, there is no consistent
behaviour from shells, so POSIX says that.

In the case of ~ expansion, there is no system call involved (necessarily)
just string manipulation, so the case of HOME='' isn't treated specially
(though it may end up being in the next version of the standard, as this
is a case where there are some implementation differences - or perhaps
it might be decided that the implementations that don't do what the standard
currently requires are simply buggy, and leave things as they are now).
There is (as of yesterday) a POSIX bug report about this, and some other
weird case issues with ~ expansions.

  | The possible errors are now
  | environment variable HOME is empty

The current question is why that one in particular needs to be an error.

kre

Re: nmh 1.8RC2, xlbiff, and $HOME

[Robert Elz]

Date:Tue, 31 Jan 2023 13:22:19 +
From:Ralph Corderoy 
Message-ID:  <20230131132219.5e02b20...@orac.inputplus.co.uk>

  | It looks to me like code assumes mypath isn't NULL, e.g. exmaildir(),
  | so not bothering to call set_mypath() if MH is set doesn't look a goer.

Perhape not, but

void
set_mypath(void)
{
char *var = getenv("HOME");
if (var && *var != '\0') {
mypath = mh_xstrdup(var);
return;
}

errno = 0;
struct passwd *pw = getpwuid(getuid());
/* etc */

would be a fairly trivial change (making the function smaller).

It would also be possible to change

if (!*pw->pw_dir)
die("password entry has empty home directory");

to

if (*pw->pw_dir == '\0')
pw->pw_dir = "/";   /* or "/tmp" or even 
"/no/home/dir" or ... */

if that seemed desirable,

kre

Re: nmh 1.8RC2, xlbiff, and $HOME

[Robert Elz]

Date:Tue, 31 Jan 2023 18:19:58 +
From:Ralph Corderoy 
Message-ID:  <20230131181958.1cfb121...@orac.inputplus.co.uk>

  | No, one cannot say of an unset HOME that it may be set by accident.

No, but it may have been unset by accident, where the intended value
for mh to use was something different than the pw_dir entry.   No-one
can possibly know in that case, any more than in the case where HOME=''

  | There is no ambiguity over the right course.

There is always ambiguity if you are doing something different than
what is documented to happen.   Users may have expected something different
than you expect.

  | In the same way that seeing ${foo:-bar} instead of
  | ${foo:bar} in sh is a smell.

Nonsense, ${foo:-bar} (rather than ${foo-bar} - a version with just a :
would be a ksh, bash, or zsh extension - but in this case I assume just
a typo)) is the more common thing to use.

It matches the other common usage ${foo} which produces nothing when
foo is unset, or when foo=''   (and "${foo}" produces a null string
in either of those cases).   That a default value is to be used instead
of the empty string doesn't change that at all - explicitly using ${foo-bar}
is a much rarer use case (there are occasions when that's wanted, but
they are not very common).

There is no point comparing what shells do when HOME is unset or "",
whether the same thing in the two cases, or something different (and
seemingly bizarre implementations in unusual cases are often just
because the cases are unusual - don't happen often - and slowing down
the normal case to handle those specially isn't worth the effort).
A shell can do whatever it likes in these cases.   So can nmh.

The question is what is best for nmh - and for that I agree with Ken,
treating an unset HOME and HOME='' differently is a mistake.   All that
is required to make that OK is to document it.   There is never a
need to deliberately make anything an error to satisfy user expectations,
if the user wants a complaint about HOME being an empty string, rather
than some other behaviour, all they need is one of

test -z "${HOME}" && { echo Error; exit ; }
or
test -z "${HOME-x}" && { echo Error; exit ; }

(the latter being a case where omitting the ':' is wanted)

My experience is that lots of users don't know "unset" exists in the
shell.   They just do var=  (or unnecessarily var='') when then want a
variable to "go away".   Most of the time unset and empty variables mean
the same thing, so this just works.

kre

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi Ken,

> > What's the intent of an empty HOME?
> > Is it set by accident when it's meant to be unset?
> > Is it empty by accident when it's meant to be non-empty?
> > Do they want HOME=/, HOME=$PWD, or are they expecting it to error.
> > Any choice could be not what the user intended so exit.
>
> I mean ... you could say the exact same things of an unset HOME!

No, one cannot say of an unset HOME that it may be set by accident.
Nor that it is empty by accident since it is neither empty or non-empty.
This being Unix, the user knows what they're doing and POSIX seemingly
allows it so we permit the lack of HOME and use pw_dir.
There is no ambiguity over the right course.
As is often the way with an environment variable, HOME overrides pw_dir
just as an argv option overrides an environment variable which overrides
a configuration-file setting.

But if HOME is empty we do not know their intent so to ignore it and use
pw_dir may not be what they think will occur.  The wrong profile could
be read or the wrong .netrc used, upsetting the user.

> My point is that I cannot see a reason to treat unset HOME and empty
> HOME differently in nmh; if unset HOME uses pw_dir, then I would argue
> that an empty HOME should do the same thing.

I disagree.  In the same way that seeing ${foo:-bar} instead of
${foo:bar} in sh is a smell.

> I was curious and some system call traces suggest that what happens
> under the hood is that when 'cd' is called with an empty HOME sh
> explicitly calls chdir(getcwd()).

Sometimes.  I think it's more complex than that, e.g. perhaps PWD has
an effect, and it will obviously vary widely.

$ LC_ALL=C env HOME= PWD=/nonexist strace -fe getcwd,stat,chdir dash -c cd
stat("/nonexist", 0x7ffc101d1bb0)   = -1 ENOENT (No such file or 
directory)
getcwd("/home/tmp/1628693513.257008819", 4096) = 31
chdir("/home/tmp/1628693513.257008819") = 0
+++ exited with 0 +++
$
$ LC_ALL=C env HOME= PWD=$PWD strace -fe getcwd,stat,chdir dash -c cd
stat("/home/tmp/1628693513.257008819", {st_mode=S_IFDIR|0755, st_size=4096, 
...}) = 0
stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
chdir("/home/tmp/1628693513.257008819") = 0
+++ exited with 0 +++
$
$ LC_ALL=C env -u PWD HOME= strace -fe getcwd,stat,chdir dash -c cd
getcwd("/home/tmp/1628693513.257008819", 4096) = 31
chdir("/home/tmp/1628693513.257008819") = 0
+++ exited with 0 +++
$

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ken Hornstein]

>What's the intent of an empty HOME?
>Is it set by accident when it's meant to be unset?
>Is it empty by accident when it's meant to be non-empty?
>Do they want HOME=/, HOME=$PWD, or are they expecting it to error.
>Any choice could be not what the user intended so exit.

I mean ... you could say the exact same things of an unset HOME!  My
point is that I cannot see a reason to treat unset HOME and empty HOME
differently in nmh; if unset HOME uses pw_dir, then I would argue that
an empty HOME should do the same thing.

Bakul says:

>FWIW, this is how /bin/sh behaves on FreeBSD:
>[...]

Fair enough; I was curious and some system call traces suggest that
what happens under the hood is that when 'cd' is called with an empty HOME
sh explicitly calls chdir(getcwd()).  But to me that still doesn't make a
case that empty HOME and unset HOME should be treated differently in nmh.
I recognize that this is a case where reasonable people can disagree.

--Ken

Re: nmh 1.8RC2, xlbiff, and $HOME

[Bakul Shah]

On Jan 31, 2023, at 8:32 AM, Ralph Corderoy  wrote:
> 
> Hi Ken,
> 
>>> So an unset HOME is allowed by this function, it's an empty HOME
>>> which isn't.
>> 
>> It strikes me as strange that there is a difference between an unset
>> HOME and an empty HOME in terms of behavior.  I mean, yes, I can see
>> how the code is written, the historical precedent and how we got here,
>> but ... well, I'm trying to understand the justification for treating
>> those differently.
> 
> A non-empty HOME is clear; its content should be used.
> 
> An unset HOME is clear; it cannot be used.
> There's a ready fallback, getpwuid(3) and pw_dir so use that.
> 
> What's the intent of an empty HOME?
> Is it set by accident when it's meant to be unset?
> Is it empty by accident when it's meant to be non-empty?
> Do they want HOME=/, HOME=$PWD, or are they expecting it to error.
> Any choice could be not what the user intended so exit.

FWIW, this is how /bin/sh behaves on FreeBSD:

$ cd /tmp
$ sh -c 'cd; echo $PWD'
/home/bakul
$ unset HOME
$ sh -c 'cd; echo $PWD'
cd: HOME not set
/tmp
$ HOME=/bin sh -c 'cd; echo $PWD'
/bin
$ HOME='' sh -c 'cd; echo $PWD'
/tmp
$ HOME='foo' sh -c 'cd; echo $PWD'
cd: foo: No such file or directory
/tmp

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi Ken,

> > So an unset HOME is allowed by this function, it's an empty HOME
> > which isn't.
>
> It strikes me as strange that there is a difference between an unset
> HOME and an empty HOME in terms of behavior.  I mean, yes, I can see
> how the code is written, the historical precedent and how we got here,
> but ... well, I'm trying to understand the justification for treating
> those differently.

A non-empty HOME is clear; its content should be used.

An unset HOME is clear; it cannot be used.
There's a ready fallback, getpwuid(3) and pw_dir so use that.

What's the intent of an empty HOME?
Is it set by accident when it's meant to be unset?
Is it empty by accident when it's meant to be non-empty?
Do they want HOME=/, HOME=$PWD, or are they expecting it to error.
Any choice could be not what the user intended so exit.

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ken Hornstein]

>So an unset HOME is allowed by this function, it's an empty HOME which
>isn't.

It strikes me as strange that there is a difference between an unset
HOME and an empty HOME in terms of behavior.  I mean, yes, I can see how
the code is written, the historical precedent and how we got here, but
... well, I'm trying to understand the justification for treating those
differently.

--Ken

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi kre,

Thanks for your learned input on this.  As I said in another reply just
now, where I listed set_mypath(), HOME being unset is fine as getpwuid()
is the fallback in which case pw_dir must be non-empty.

> > Alexander does point out that HOME is supposed to be valid according
> > to POSIX,
>
> That's actually not correct, all the text quoted requires is that HOME
> be set at login time - nothing at all about what can be done with it
> after that.

Agreed.  Nearby, is:

8.1 Environment Variable Definition

...
Some are placed into the environment by the implementation at the
time the user logs in; all can be added or changed by the user or
any ancestor of the current process.

Does ‘changed’ include unsetting them?  I'd expect the spec. to be
precise and list ‘unset’ to clarify if they can.

> Further POSIX reading will also find, in the description of ~
> expansion in the shell (XCU 2.6.1 in Issue 7 TC2 - the current version
> from 2018):
>
> If the login name is null (that is, the tilde-prefix contains only
> the tilde), the tilde-prefix is replaced by the value of the
> variable HOME.  If HOME is unset, the results are unspecified.
...
> Setting HOME='' isn't even mentioned: in that case ~/foo expands to
> /foo (which is actually what you want

Agreed, it isn't mentioned.

> Similarly, in XCU 4 in the description of the cd utility:
>
> 1. If no directory operand is given and the HOME environment
> variable is empty or undefined, the default behavior is
> implementation-defined and no further steps shall be taken.
>
> Again, HOME being unset is allowed there, this time with
> implementation-defined results (which just means the implementation
> must say what happens in the doc).

Yes, but also allowed there is ‘empty’ which also triggers
implementation-defined.

Also, POSIX's chdir(3) says

   int chdir(const char *path);
...

ERRORS
   The chdir() function shall fail if:
   ...
   ENOENT  A component of path does not name an existing
   directory or path is an empty string.

chdir(NULL) is invalid, obviously.  But so is chdir("") so if an empty
HOME is really to be treated as the root directory then code can't do

h = getenv("HOME");
r = chdir(h ? h : "/");

but must instead

r = chdir(h && *h ? h : "/");

I expect lots of non-nmh code doesn't do that extra check so the chdir()
has no effect and the error return may not even be tested so an empty
HOME results in the current directory being used rather than /.

> Running with HOME unset is not even all that unusual

I think it is a bit.  :-)

> "env -i command" results in that

Agreed, my self-at(1)-ing shell scripts run at(1) under ‘env -i’ but
bequeath HOME, LANG, LOGNAME, and PATH at the same time.

> (and I'd even retain: relative to / if HOME is not set, which it can
> be when there as been no login - like when you're in single user
> mode).

It would be nice for nmh to work in single-user mode as reading email
might be vital in fixing problems.  As long as HOME is unset and pw_dir
is valid then all is well at the moment.

> I'd also suggest never issuing errors because the user's environment
> isn't like we think it ought to be, unless that actually prevents
> something functioning as intended.

The code I replaced mislead the user as to the cause of the exit.
That's fixed and no HOME is okay.  The possible errors are now

environment variable HOME is empty
password entry not found
[perhaps with ‘getpwuid() failed’ and errno if it did]
password entry has empty home directory

This close to a release, I think we should stick with requiring HOME to
be non-empty if it's set as otherwise there's too many paths to consider
which the test harness probably doesn't exercise.

> But I agree that a relative Path in MH should be relative to HOME

Yes, I think it's too late to alter that.
Though ‘path: mail’ would be a more Unixy default.  :-)

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi az,

> "Author: Ralph Corderoy 
> Date:   Thu May 13 13:46:20 2021 +0100
>
> sbr/path.c: add set_mypath() to factor out repeated code."

I think it's worth expanding on that.

commit d8ca46fabc26469be325b73a73dcc26e70681eb5
Author: Ralph Corderoy 
Date:   Thu May 13 13:46:20 2021 +0100

sbr/path.c: add set_mypath() to factor out repeated code.

set_mypath() sets the existing global ‘mypath’ which really holds
the home-directory's path.  As getenv(3) and getpwuid(3)'s result
can be invalidated by further library calls, a value they return is
mh_xstrdup()'d; not all the old bits of code replaced by a call to
set_mypath() were doing this.

Also add more explicit error messages if set_mypath() has trouble
finding the home directory's path.

And here's the current code I wrote for set_mypath() which is causing
the trouble.

/* set_mypath sets the global mypath to the HOME environment variable if
 * it is set and not empty, or else the password entry's home-directory
 * field if it's found and not empty.  Otherwise, the program exits
 * with an error.  The value found is copied with mh_xstrdup() as later
 * library calls may invalidate returned values. */
void
set_mypath(void)
{
char *var = getenv("HOME");
if (var) {
if (!*var)
die("environment variable HOME is empty");

mypath = mh_xstrdup(var);
return;
}

errno = 0;
struct passwd *pw = getpwuid(getuid());
if (!pw) {
if (errno)
adios("", "getpwuid() failed");   /* "" prints errno! */
die("password entry not found");
}
if (!*pw->pw_dir)
die("password entry has empty home directory");

mypath = mh_xstrdup(pw->pw_dir);
}

So an unset HOME is allowed by this function, it's an empty HOME which
isn't.

An unset HOME falls back on getpwuid() which must dish up a non-empty
pw_dir.  I think that's reasonble.

> a quick glance seems to indicate that only context_read in
> sbr/context_read.c looks at $MH for finding a home

And uip/install-mh.c in a similar manner.

> but it does (well attempts) that just a few lines AFTER set_mypath has
> bailed out on it...
>
> i think we need to fix that sequence up a little :-)

It's the long-standing arrangement going back to Git's
docs/historical/mh-6.8.5/sbr/m_getdefs.c which allows an empty HOME but
not an empty pw_dir, though it complains that the later is the lack of
a HOME ‘envariable’.  :-)

if (mypath == NULL) {
if (mypath = getenv ("HOME"))
mypath = getcpy (mypath);
else
if ((pw = getpwuid (getuid ())) == NULL
|| pw -> pw_dir == NULL
|| *pw -> pw_dir == 0)
adios (NULLCP, "no HOME envariable");
else
mypath = getcpy (pw -> pw_dir);
if ((cp = mypath + strlen (mypath) - 1) > mypath && *cp == '/')
*cp = 0;
}

It looks to me like code assumes mypath isn't NULL, e.g. exmaildir(),
so not bothering to call set_mypath() if MH is set doesn't look a goer.
At least not without a considerably more analysis which would probably
inspire changes which would ‘disappoint’ RC-maker David.  :-)

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi David,

> I'd like to get Ralph's take on what we should do.

Thanks.  If my suggestion to Stephen of unsetting HOME works and is
acceptable to him then I suggest we don't change nmh for this release.

> > A further documentation issue: mh-profile(5) does not specify the
> > treatment of a relative Path.  I expected it to be relative to the
> > directory of the profile file, but it seems to actually be relative
> > to HOME.
>
> Right, it's relative to HOME.  I'll add this clarification to the man
> page:
> A relative Path is relative to the user's $HOME directory.
> unless there's another suggestion.

As HOME needn't be set how about mh_profile(5) saying

 Path: Mail
 Locates nmh transactions in directory “Mail”.
+A relative path starts from the user's home directory.
 This is the only mandatory profile entry.
 (profile, no default)

I see the ENVIRONMENT section doesn't mention HOME.
Perhaps,

HOME  If set, it is used as the user's home directory.
  If not set then getpwuid(3)'s pw_dir is used.

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ralph Corderoy]

Hi Stephen,

> I have investigated the failure of the xlbiff tests with nmh 1.8RC2.

Thanks.

> $ printf 'Path: /tmp\n' > /tmp/mh-profile-minimal
> $ HOME= MH=/tmp/mh-profile-minimal /usr/bin/mh/mhparam path

My suggestion for a quick fix to try is to not have HOME in the
environment so getpwuid(3) is used.

$ HOME= MH=$PWD/mh_profile mhparam path
mhparam: environment variable HOME is empty
$
$ env -u HOME MH=$PWD/mh_profile mhparam path
/tmp

> This is a regression.  HOME is used only to set the default profile
> file to "$HOME/.mh_profile".  But nmh doesn't need HOME if MH is set.

I think nmh *as a whole* does need HOME to be non-empty if it's set.

-- 
Cheers, Ralph.

Re: nmh 1.8RC2, xlbiff, and $HOME

[Stephen Gildea]

Ken Hornstein  wrote:

>   >$ printf 'Path: /tmp\n' > /tmp/mh-profile-minimal
>   >$ HOME= MH=/tmp/mh-profile-minimal /usr/bin/mh/mhparam path
>   
>   Thank you for the analysis.  I am wondering, though ... WHY does xlbiff
>   set HOME to '' for this test?

To be clear, it is not xlbiff itself that does this, only an automated
unit test.  The test setup sets HOME to create a hermetic testing
environment; the result of the test should not depend on the user
running the test.  My current test sets HOME to an empty string.

The test could just as effectively set HOME to an non-existent directory.
I know of other test suites that use this approach.  This would still
give me a hermetic, repeatable test, while leaving nmh more freedom to
make, test, and enforce decisions about what values of HOME are allowed.

 < Stephen

Re: nmh 1.8RC2, xlbiff, and $HOME

[David Levine]

Stephen wrote:

> My analysis:
>
> This is a regression.  HOME is used only to set the default profile
> file to "$HOME/.mh_profile".  But nmh doesn't need HOME if MH is set.

Thanks for your analysis.  I'd like to get Ralph's take on what we
should do.

> A further documentation issue: mh-profile(5) does not specify the
> treatment of a relative Path.  I expected it to be relative to the
> directory of the profile file, but it seems to actually be relative to
> HOME.

Right, it's relative to HOME.  I'll add this clarification to the man
page:
A relative Path is relative to the user's $HOME directory.
unless there's another suggestion.

David

Re: nmh 1.8RC2, xlbiff, and $HOME

[Robert Elz]

Date:Mon, 30 Jan 2023 16:11:26 -0500
From:Ken Hornstein 
Message-ID:  <20230130211131.8e81d1df...@pb-smtp20.pobox.com>

  | Alexander does point out that HOME is supposed to be
  | valid according to POSIX,

That's actually not correct, all the text quoted requires is that
HOME be set at login time - nothing at all about what can be done
with it after that.

Further POSIX reading will also find, in the description of ~ expansion in
the shell (XCU 2.6.1 in Issue 7 TC2 - the current version from 2018):

If the login name is null (that is, the tilde-prefix contains only
the tilde), the tilde-prefix is replaced by the value of the variable
HOME. If HOME is unset, the results are unspecified.

which clearly allows the possibility that HOME not be set (which in this
case makes the result of the ~ expansion unspecified, because some shells
look up the password file, and use the home directory field from there,
others give an error, perhaps others just use "" - but that is irrelevant
to the question of whether HOME is required to be unset.)   Setting HOME=''
isn't even mentioned: in that case ~/foo expands to /foo (which is actually
what you want, HOME=/ might (should, by a strict reading of the standard)
result in //foo which is an unspecified path).

Similarly, in XCU 4 in the description of the cd utility:

 1. If no directory operand is given and the HOME environment variable
is empty or undefined, the default behavior is implementation-defined
and no further steps shall be taken.

Again, HOME being unset is allowed there, this time with implementation-defined
results (which just means the implementation must say what happens in the
doc).

Running with HOME unset is not even all that unusual, "env -i command"
results in that - I do almost all builds that way, to ensure that nothing
in my local environment, or home directory, affects the results, so they
will be repeatable.

But I agree that a relative Path in MH should be relative to HOME (and
I'd even retain: relative to / if HOME is not set, which it can be when
there as been no login - like when you're in single user mode).

I'd also suggest never issuing errors because the user's environment isn't
like we think it ought to be, unless that actually prevents something
functioning as intended.

kre

Re: nmh 1.8RC2, xlbiff, and $HOME

[Ken Hornstein]

>$ printf 'Path: /tmp\n' > /tmp/mh-profile-minimal
>$ HOME= MH=/tmp/mh-profile-minimal /usr/bin/mh/mhparam path

Thank you for the analysis.  I am wondering, though ... WHY does xlbiff
set HOME to '' for this test?

(I am neutral on whether or not this is technically a regression; I can
see it both ways.  Alexander does point out that HOME is supposed to be
valid according to POSIX, but I am not 100% that means it can't be a
zero-length string).

--Ken

Re: nmh 1.8RC2, xlbiff, and $HOME

[Alexander Zangerl]

On Mon, 30 Jan 2023 10:47:05 -0800, Stephen Gildea writes:
>I have investigated the failure of the xlbiff tests with nmh 1.8RC2.
>(This is https://bugs.debian.org/1029752)

stephen, thanks for that.

>In one of the tests, xlbiff sets environment variable HOME to an empty
>string and MH to a file containing a custom profile with an absolute Path.
>With nmh 1.7.1, this environment works.
>
>With nmh 1.8RC2, this environment fails with the error message
>"environment variable HOME is empty".

*lightbulb*

that error message comes from set_mypath in sbr/path.c, and
the changelog tells me that

"Author: Ralph Corderoy 
Date:   Thu May 13 13:46:20 2021 +0100

sbr/path.c: add set_mypath() to factor out repeated code."

started to consolidate the various places that figure out a home for mh.

however, set_mypath spot doesn't bother with $MH.

a quick glance seems to indicate that only context_read in
sbr/context_read.c looks at $MH for finding a home, but
it does (well attempts) that just a few lines AFTER set_mypath has
bailed out on it...

i think we need to fix that sequence up a little :-)

[handling of relative paths in the profile]
>Whatever your decision, the choice should be documented.  If you decide
>to keep Path relative to HOME, then HOME should be required to be
>non-empty if (and only if!) Path is relative.  (nmh 1.7.1 used "/",
>which seems wrong.)

with my nitpicker's hat on: POSIX/SUS says HOME can be depended on.

"HOME The system shall initialize this variable at the time of login to
be a pathname of the user's home directory. See ."

personally i'd say we should rely on posix and insist on not being
$HOMEless. for me that would imply relatives should be relative to that,
not the profile location.

regards
az


-- 
Alexander Zangerl + GPG Key 2FCCF66BB963BD5F + https://snafu.priv.at/
`bastard operators from hell' anagrams to `shatterproof armored balls'.
 -- Cliff Miller


signature.asc
Description: Digital Signature

Re: nmh 1.8RC2, xlbiff, and $HOME

[Kevin Cosgrove]

I also use xlbiff and would like it to continue working without needing to
fuss with it.

The next best option for me is that I might be required to set some
environment variables in a sane way that doesn't cascade into the operation
of other programs.

Thanks for uncovering this release client change/issue!

On Mon, Jan 30, 2023 at 10:47 AM Stephen Gildea 
wrote:

> I have investigated the failure of the xlbiff tests with nmh 1.8RC2.
> (This is https://bugs.debian.org/1029752)
>
> In one of the tests, xlbiff sets environment variable HOME to an empty
> string and MH to a file containing a custom profile with an absolute Path.
> With nmh 1.7.1, this environment works.
>
> With nmh 1.8RC2, this environment fails with the error message
> "environment variable HOME is empty".
>
> You can see the change in behavior as follows:
>
> $ printf 'Path: /tmp\n' > /tmp/mh-profile-minimal
> $ HOME= MH=/tmp/mh-profile-minimal /usr/bin/mh/mhparam path
>
>
> nmh 1.7.1:
> /tmp
>
> nmh 1.8RC2:
> mhparam: environment variable HOME is empty
>
>
> My analysis:
>
> This is a regression.  HOME is used only to set the default profile
> file to "$HOME/.mh_profile".  But nmh doesn't need HOME if MH is set.
>
> A further documentation issue: mh-profile(5) does not specify the
> treatment of a relative Path.  I expected it to be relative to the
> directory of the profile file, but it seems to actually be relative to
> HOME.
>
> Whatever your decision, the choice should be documented.  If you decide to
> keep Path relative to HOME, then HOME should be required to be non-empty if
> (and only if!) Path is relative.  (nmh 1.7.1 used "/", which seems wrong.)
>
>  < Stephen
>
>

nmh 1.8RC2, xlbiff, and $HOME

[Stephen Gildea]

I have investigated the failure of the xlbiff tests with nmh 1.8RC2.
(This is https://bugs.debian.org/1029752)

In one of the tests, xlbiff sets environment variable HOME to an empty
string and MH to a file containing a custom profile with an absolute Path.
With nmh 1.7.1, this environment works.

With nmh 1.8RC2, this environment fails with the error message
"environment variable HOME is empty".

You can see the change in behavior as follows:

$ printf 'Path: /tmp\n' > /tmp/mh-profile-minimal
$ HOME= MH=/tmp/mh-profile-minimal /usr/bin/mh/mhparam path


nmh 1.7.1:
/tmp

nmh 1.8RC2:
mhparam: environment variable HOME is empty


My analysis:

This is a regression.  HOME is used only to set the default profile
file to "$HOME/.mh_profile".  But nmh doesn't need HOME if MH is set.

A further documentation issue: mh-profile(5) does not specify the
treatment of a relative Path.  I expected it to be relative to the
directory of the profile file, but it seems to actually be relative to
HOME.

Whatever your decision, the choice should be documented.  If you decide to
keep Path relative to HOME, then HOME should be required to be non-empty if
(and only if!) Path is relative.  (nmh 1.7.1 used "/", which seems wrong.)

 < Stephen