Re: pipes and sub-shells

[Thorsten Glaser]

Jean Delvare dixit:

>On mer., 2017-03-22 at 10:12 +0100, Jean Delvare wrote:
>> One of our customers recently asked us about the different behavior of
>> constructs involving pipes in ksh scripts. I explained to them that
>> mksh spawns a sub-shell for the right hand side of each pipe, and
>> therefore whatever happens on that side of the pipe has no effect on
>> the main shell.
>
>I have a hard time convincing the customer that the behavior is as
>expected. They quote the manual page:
>
>"A function can be made to finish immediately using the return
>command;"
>
>Which indeed isn't what's happening.

Of course it isn’t because the return command is not called from
within a function but from within a subshell which, while contained
in a function, is not “in a function”.

This is no different to any non-last part of a pipeline in such
shells — calling return in anything BEFORE a pipe would also not
“work” as requested.

>The read loop FAQ (about pipes spawning subshells) mentions the
>inability to propagate variable changes back to the original shell, but
>says nothing about the altered effect of "return" or "exit".

It has not come up as a problem so far.

>May I suggest that the manual page gets updated in the following ways:

Yes, thank you.

>I can prepare and send a patch doing (some of) these changes if you
>agree with them.
>
>Maybe it would also make sense to add a specific paragraph or section
>somewhere about subshells. In various places, the manual page explains

Feel free to send such a patch.

>that a given construct will (or will not) spawn a subshell, but there
>is no central place explaining what a subshell is nor the limitations
>that it creates. You probably know more than I do on the topic. I know
>about the variable propagation (or more generally the separate
>environments), return and exit, but maybe there is more?

A subshell involves fork(), it’s as easy as that.

Re: pipes and sub-shells

[Jean Delvare]

On mer., 2017-03-22 at 10:12 +0100, Jean Delvare wrote:
> One of our customers recently asked us about the different behavior of
> constructs involving pipes in ksh scripts. I explained to them that
> mksh spawns a sub-shell for the right hand side of each pipe, and
> therefore whatever happens on that side of the pipe has no effect on
> the main shell.

I have a hard time convincing the customer that the behavior is as
expected. They quote the manual page:

"A function can be made to finish immediately using the return
command;"

Which indeed isn't what's happening. The description of the "return"
builtin does not say anything about subshells either. It only says that
"return" is treated as "exit" if used outside a function, but I think
they do not realize this is what's happening. (Maybe it would have been
better to stop with an error in such cases, as bash does, albeit
inconsistently, instead of falling back transparently to "exit".)

The read loop FAQ (about pipes spawning subshells) mentions the
inability to propagate variable changes back to the original shell, but
says nothing about the altered effect of "return" or "exit".

May I suggest that the manual page gets updated in the following ways:

* In the description of the exit builtin, replace "The shell exits"
with "The shell or subshell exits".

* In the description of the return builtin, explain that return can't
be used to return from a function if called in a subshell, and point to
the read loop FAQ.

* In the read loop FAQ, mention that calling return or exit in the
inner loop will only put an end to the inner loop and won't have any
immediate effect on the original shell.

I can prepare and send a patch doing (some of) these changes if you
agree with them.

Maybe it would also make sense to add a specific paragraph or section
somewhere about subshells. In various places, the manual page explains
that a given construct will (or will not) spawn a subshell, but there
is no central place explaining what a subshell is nor the limitations
that it creates. You probably know more than I do on the topic. I know
about the variable propagation (or more generally the separate
environments), return and exit, but maybe there is more?

Thanks,
-- 
Jean Delvare
SUSE L3 Support

Re: pipes and sub-shells

[Dr. Werner Fink]

On Wed, Mar 22, 2017 at 07:39:21PM +, Thorsten Glaser wrote:
> 
> Dr. Werner Fink dixit:
> 
> >Just to be mentioned, I can mksh change in such a way that without job 
> >control
> >I can do
> >
> >   ~/rpmbuild/BUILD/mksh> ./mksh -c 'echo xxx | read yyy; echo $yyy'
> >   xxx
> >
> >that is it does work similar to the lastpipe shell option of the bash
> >I had initiated now 7 years back. The trick is to avouid forking the
> >last element in a pipe chain.
> 
> I’ve thought about this… years ago, too. But seeing as POSIX allows
> either behaviour, I decided that this (running all sides of pipelines
> in subshells) is a language feature now, for simplicity of user scripts
> (as they can just use co-processes when they don’t want it).
> 
> So, thanks but no. (I’ve got scripts that actually rely on that now.)

OK .. nevertheless, only for the records, the change might look
similar to that in the attachment I guess.

Werner

-- 
  "Having a smoking section in a restaurant is like having
  a peeing section in a swimming pool." -- Edward Burr
--- mksh/exec.c
+++ mksh/exec.c	2017-03-27 10:33:11.0 +
@@ -156,6 +156,8 @@ execute(struct op * volatile t,
 			}
 		}
 
+	flags &= ~XPIPELS;
+
 	switch (t->type) {
 	case TCOM:
 		rv = comexec(t, tp, (const char **)ap, flags, xerrok);
@@ -192,8 +194,23 @@ execute(struct op * volatile t,
 		restfd(1, e->savefd[1]);
 		/* no need to re-restore this */
 		e->savefd[1] = 0;
-		/* Let exchild() close 0 in parent, after fork, before wait */
-		i = exchild(t, flags | XPCLOSE | XPIPEST, xerrok, 0);
+
+		tp = NULL;
+		if (Flag(FLASTPIPE) && !Flag(FPIPEFAIL) && t->args) {
+			up = eval(t->args, t->u.evalflags | DOBLANK | DOGLOB | DOTILDE);
+			ap = (const char **)up;
+			if (ap[0])
+tp = findcom(ap[0], FC_BI|FC_FUNC);
+		}
+
+		if (tp && tp->type == CSHELL) {
+			flags &= ~XFORK;
+			flags |= XPIPELS;
+			i = execute(t, flags | XPIPELS | XERROK, xerrok);
+		} else
+			/* Let exchild() close 0 in parent, after fork, before wait */
+			i = exchild(t, flags | XPCLOSE | XPIPEST, xerrok, 0);
+
 		if (!(flags&XBGND) && !(flags&XXCOM))
 			rv = i;
 		break;
@@ -453,6 +470,12 @@ execute(struct op * volatile t,
 
 	/* restores IO */
 	quitenv(NULL);
+
+#ifndef MKSH_UNEMPLOYED
+	if (flags & XPIPELS)
+		j_restore(flags);
+#endif
+
 	if ((flags&XEXEC))
 		/* exit child */
 		unwind(LEXIT);
--- mksh/jobs.c
+++ mksh/jobs.c	2017-03-27 10:34:26.0 +
@@ -641,6 +641,23 @@ exchild(struct op *t, int flags,
 	return (rv);
 }
 
+#ifndef MKSH_UNEMPLOYED
+void
+j_restore(int flags)
+{
+	if (!Flag(FMONITOR) || (flags&XXCOM))
+		return;
+	/* job control set up */
+
+	if (!ttypgrp_ok)
+		return;
+
+	if (kshpgrp == kshpid && kshpgrp != tcgetpgrp(tty_fd))
+		if (tcsetpgrp(tty_fd, kshpid) < 0)
+			ttypgrp_ok = false;
+}
+#endif
+
 /* start the last job: only used for $(command) jobs */
 void
 startlast(void)
--- mksh/main.c
+++ mksh/main.c	2017-03-27 10:45:27.0 +
@@ -250,6 +250,15 @@ main_init(int argc, const char *argv[],
 	 */
 	Flag(FBRACEEXPAND) = 1;
 
+#if 1
+	/*
+	 * Turn on lastpipe by default ??
+	 * If set the shell runs the last command of a pipeline  not
+	 * executed in the background in the current shell environment.
+	 */
+	Flag(FLASTPIPE) = 1;
+#endif
+
 	/*
 	 * Turn on "set -x" inheritance by default.
 	 */
--- mksh/mksh.1
+++ mksh/mksh.1	2017-03-27 10:58:39.000 +
@@ -4291,6 +4291,9 @@ Do not reset
 .Fl o Ic xtrace
 upon entering functions.
 This is enabled by default.
+.It Fl o Ic lastpipe
+If set the shell runs the last command of a pipeline not executed in the
+background in the current shell environment.
 .It Fl o Ic nohup
 Do not kill running jobs with a
 .Dv SIGHUP
--- mksh/sh.h
+++ mksh/sh.h	2017-03-27 10:47:08.0 +
@@ -1653,6 +1653,7 @@ struct op {
 #define TTIME		20	/* time pipeline */
 #define TEXEC		21	/* fork/exec eval'd TCOM */
 #define TCOPROC		22	/* coprocess |& */
+#define TLPIPE		23	/* last in pipe chain  */
 
 /*
  * prefix codes for words in command tree
@@ -1714,6 +1715,7 @@ struct ioword {
 #define XCOPROC BIT(9)		/* starting a co-process */
 #define XTIME	BIT(10)		/* timing TCOM command */
 #define XPIPEST	BIT(11)		/* want PIPESTATUS */
+#define XPIPELS	BIT(12)		/* Last element in pipe chain */
 
 /*
  * flags to control expansion of words (assumed by t->evalflags to fit
@@ -2142,6 +2144,7 @@ void j_init(void);
 void j_exit(void);
 #ifndef MKSH_UNEMPLOYED
 void j_change(void);
+void j_restore(int);
 #endif
 int exchild(struct op *, int, volatile int *, int);
 void startlast(void);
--- mksh/sh_flags.opt
+++ mksh/sh_flags.opt	2017-03-27 10:40:06.0 +
@@ -79,6 +79,10 @@ FN("interactive", FTALKING, OF_CMDLINE
 >k|
 FN("keyword", FKEYWORD, OF_ANY
 
+/* ./.	if set the shell runs the last command of a pipeline not executed in the background in the current shell environment */
+>|
+FN("lastpipe", FLASTPIPE, OF_ANY
+
 /* -l	login shell */
 >l|!SHFLAGS_NOT_CMD
 FN("login", FLOGIN, OF_CMDLINE


signature.asc

Re: POSIX character classes (was Re: pipes and sub-shells)

[Martijn Dekker]

Op 23-03-17 om 22:02 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> * BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
>> [expressions] are not supported in glob patterns.
> 
> I really really REALLY hate that this will make mksh really big.
> We’re talking about 36K .rodata even without titlecase conversion
> and BMP-only (16-bit Unicode) here.

I sympathise.

Even fnmatch(3) is not compliant on all systems; the BSDs don't seem to
have caught up yet. :(  I don't suppose using that is an option in any
case because of mksh's extended globbing functionality.

Is adding 36k really that much in 2017? On my system, the current
development binary of mksh is 283k after stripping when built with -O2,
235k with -Os. Adding 36k would make it 316k/271k, still quite small.

If that's too much, I guess you should continue to not support them. The
reason modernish detects BUG_NOCHCLASS is not to make some sort of
statement, but to enable programs using the library to easily check for
the presence of the issue and implement alternative methods (such as
falling back to external commands, or just matching ASCII only without
character classes).

> Can I get by making them match ASCII only even in UTF-8 mode?

IMHO, that would defeat their primary purpose, namely locale-dependent
class matching, so no, not really. :)

If Greeks or Russians (or Germans, for that matter) can't count on
[:upper:] matching an upper case letter in their alphabets, then I'd say
for them it would be better to have no support than broken support.

> Strictly speaking, POSIX requires only support for the C locale,
[...]

Yes, but on systems supporting other locales (e.g. UTF-8), it would not
be conforming for character classes to match ASCII only. You either
support UTF-8 or you don't.

- M.

Re: pipes and sub-shells

[Jean Delvare]

Hi Martijn,

On Thu, 23 Mar 2017 16:49:33 +0100, Martijn Dekker wrote:
> Op 23-03-17 om 10:49 schreef Jean Delvare:
> > Apparently it requires a more recent version of mksh than we are
> > shipping:
> > 
> > $ echo $KSH_VERSION
> > @(#)MIRBSD KSH R50 2014/06/29 openSUSE
> 
> That version is quite ancient, so you should consider upgrading it to
> the latest. FYI, modernish 

I know, we are working up upgrading to version 50f. Upgrading to the
latest version on a released product is not an option, the risk of
regression is too high.

> currently detects the following bugs on it that are relevant for
> cross-shell programming. All except BUG_LNNOALIAS, BUG_LNNOEVAL and
> BUG_NOCHCLASS have been fixed in the current release. The former two are
> fixed in current cvs. The latter is a design decision from Thorsten that
> is nonetheless a bug in POSIX terms.

The only bugs I care about are the ones which my customers complain
about. Which sometimes are not even bugs ;-)

-- 
Jean Delvare
SUSE L3 Support

POSIX character classes (was Re: pipes and sub-shells)

[Thorsten Glaser]

Martijn Dekker dixit:

>* BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
>[expressions] are not supported in glob patterns.

I really really REALLY hate that this will make mksh really big.
We’re talking about 36K .rodata even without titlecase conversion
and BMP-only (16-bit Unicode) here.

Can I get by making them match ASCII only even in UTF-8 mode?

Strictly speaking, POSIX requires only support for the C locale,
and our UTF-8 mode is only close to POSIX anyway, and currently
(though this will change, there have been good points made for
locale tracking) enabled using a mksh-specific set flag.

If I implemented that, we could then say that “lksh -o posix”
is, in the C locale, fully POSIX conformant. (Perhaps — but
certainly a goal to work for, even despite the uselessness
of standards.)

bye,
//mirabilos
-- 
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

Re: pipes and sub-shells

[Dan Douglas]

On Mar 23, 2017 10:16 AM, Martijn Dekker  wrote:
> Op 23-03-17 om 15:44 schreef Dan Douglas:
> > On 03/23/2017 04:49 AM, Jean Delvare wrote:
> >> Apparently it requires a more recent version of mksh than we are
> >> shipping:
> >
> > Herestrings are old. Should work in just about any ksh.
>  pdksh (including OpenBSD ksh) doesn't have them.

Ew

> On those (as on POSIX), you need a here-document to get the same effect:
>
> while stuff; do
> stuff
> done < $(cmd)
> EOF
>
> >> $ echo $KSH_VERSION @(#)MIRBSD KSH R50 2014/06/29 openSUSE $
> >> ./test_return.sh ./test_return.sh[10]: syntax error: '('
> >> unexpected
> >>
> >> But indeed works find with the latest upstream version, thanks
> >> for the pointer.
> >
> > It's also not at all equivalent to a pipe. The shell reads the
> > command output from a pipe, stripping trailing newlines, and dumps
> > the result into a temporary file, which then adds a trailing
> > newline, then reads the file back into whatever runs in the loop.
>
> In most use cases, that seems like a distinction without a difference.

It matters when there are zero, or more than one newlines at the end. Besides 
that, it just has no advantage over `out=$(cmd); printf %s -- "$out" | ...', or 
better, `read -rd "" out' to avoid mangling the data. In all cases we can't 
begin processing the loop until we've read the entire possibly large output, so 
I don't suggest that method unless the intention is to take advantage of its 
side-effects.

> > It will also fail if it expands to more than one word due to lack
> > of quotes. (That likely caused your error)
>
> That is true on bash, but not on mksh. I should probably have added
> the quotes anyway though, just to avoid that confusion.

I like that feature. Exclusive to mksh though.

Re: pipes and sub-shells

[Martijn Dekker]

Op 23-03-17 om 10:49 schreef Jean Delvare:
> Apparently it requires a more recent version of mksh than we are
> shipping:
> 
> $ echo $KSH_VERSION
> @(#)MIRBSD KSH R50 2014/06/29 openSUSE

That version is quite ancient, so you should consider upgrading it to
the latest. FYI, modernish 
currently detects the following bugs on it that are relevant for
cross-shell programming. All except BUG_LNNOALIAS, BUG_LNNOEVAL and
BUG_NOCHCLASS have been fixed in the current release. The former two are
fixed in current cvs. The latter is a design decision from Thorsten that
is nonetheless a bug in POSIX terms.

* BUG_CMDPV: 'command -pv' does not find builtins.

* BUG_CMDSPEXIT: preceding a special builtin with 'command' does not
stop it from exiting the shell if the builtin encounters an error.

* BUG_CMDVRESV: 'command -v' does not find reserved words such as "if".

* BUG_LNNOALIAS: $LINENO is always expanded to 0 when used within an alias.

* BUG_LNNOEVAL: $LINENO is always expanded to 0 when used in 'eval'.

* BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
[expressions] are not supported in glob patterns.

* BUG_PP_01: POSIX says that empty "$@" generates zero fields but empty
'' or "" or "$emptyvariable" generates one empty field. This means
concatenating "$@" with one or more other, separately quoted, empty
strings (like "$@""$emptyvariable") should still produce one empty
field. With this bug, this erroneously produces zero fields.

* BUG_PP_02: Like BUG_PP_01, but with unquoted $@ and only with
"$emptyvariable"$@, not $@"$emptyvariable".

* BUG_PP_03: Assigning the positional parameters to a variable using
either var=$* or var="$*" or both doesn't work as expected, using either
default, empty, unset or custom settings of $IFS.

* BUG_PP_04: Like BUG_PP_03, but for a default assignment within a
parameter substitution, i.e. ${var=$*} or ${var="$*"}.

* BUG_SELECTRPL: In a 'select' loop, input that is not a menu item is
not stored in the REPLY variable as it should be.

* BUG_TESTERR0: test/[ exits successfully (exit status 0) if an invalid
argument is given to an operator.

Hope this helps,

- M.

Re: pipes and sub-shells

[Martijn Dekker]

Op 23-03-17 om 15:44 schreef Dan Douglas:
> On 03/23/2017 04:49 AM, Jean Delvare wrote:
>> Apparently it requires a more recent version of mksh than we are 
>> shipping:
> 
> Herestrings are old. Should work in just about any ksh.

They're a ksh93-ism, not a ksh88-ism. pdksh (including OpenBSD ksh)
doesn't have them.

On those (as on POSIX), you need a here-document to get the same effect:

while stuff; do
stuff
done <> $ echo $KSH_VERSION @(#)MIRBSD KSH R50 2014/06/29 openSUSE $
>> ./test_return.sh ./test_return.sh[10]: syntax error: '('
>> unexpected
>> 
>> But indeed works find with the latest upstream version, thanks
>> for the pointer.
> 
> It's also not at all equivalent to a pipe. The shell reads the
> command output from a pipe, stripping trailing newlines, and dumps
> the result into a temporary file, which then adds a trailing
> newline, then reads the file back into whatever runs in the loop.

In most use cases, that seems like a distinction without a difference.

> It will also fail if it expands to more than one word due to lack
> of quotes. (That likely caused your error)

That is true on bash, but not on mksh. I should probably have added
the quotes anyway though, just to avoid that confusion.

- M.

Re: pipes and sub-shells

[Dan Douglas]

On 03/23/2017 04:49 AM, Jean Delvare wrote:
> Apparently it requires a more recent version of mksh than we are
> shipping:

Herestrings are old. Should work in just about any ksh.

> $ echo $KSH_VERSION
> @(#)MIRBSD KSH R50 2014/06/29 openSUSE
> $ ./test_return.sh
> ./test_return.sh[10]: syntax error: '(' unexpected
> 
> But indeed works find with the latest upstream version, thanks for the
> pointer.

It's also not at all equivalent to a pipe. The shell reads the command
output from a pipe, stripping trailing newlines, and dumps the result
into a temporary file, which then adds a trailing newline, then reads
the file back into whatever runs in the loop. It will also fail if it
expands to more than one word due to lack of quotes. (That likely
caused your error)

There are a few legitimate uses for <<<"$(cmd)", which mostly have to
do with wanting that temp file for some reason, but typically don't use
it without good reason.




signature.asc
Description: OpenPGP digital signature

Re: pipes and sub-shells

[Jean Delvare]

Hi Thorsten,

On mer., 2017-03-22 at 19:39 +, Thorsten Glaser wrote:
> Martijn Dekker dixit:
> 
> >Op 22-03-17 om 10:12 schreef Jean Delvare:
> >> Concretely, the customer's code looks like this:
> >>
> >> command | while read line
> >> do
> >>  if 
> >>  then
> >>  exit
> >>  fi
> >>  process $line
> >> done
> >[...]
> >> I was wondering if there is any other trick you can suggest that would 
> >> work in mksh?
> 
> This is actually a FAQ ;-) Use co-processes!
> 
> command |& while read -p line; do …

Looks good, thank you very much :-)

-- 
Jean Delvare
SUSE L3 Support

Re: pipes and sub-shells

[Jean Delvare]

Hi Martijn,

On mer., 2017-03-22 at 10:26 +0100, Martijn Dekker wrote:
> Op 22-03-17 om 10:12 schreef Jean Delvare:
> > 
> > Concretely, the customer's code looks like this:
> > 
> > command | while read line
> > do
> > if 
> > then
> > exit
> > fi
> > process $line
> > done
> [...]
> > 
> > I was wondering if there is any other trick you can suggest that would work 
> > in mksh?
> 
> A here-string with a command substitution:
> 
> while read line
> do
>   if 
>   then
>   exit
>   fi
>   process $line
> done <<<$(command)

Apparently it requires a more recent version of mksh than we are
shipping:

$ echo $KSH_VERSION
@(#)MIRBSD KSH R50 2014/06/29 openSUSE
$ ./test_return.sh
./test_return.sh[10]: syntax error: '(' unexpected

But indeed works find with the latest upstream version, thanks for the
pointer.

Thanks,
-- 
Jean Delvare
SUSE L3 Support

Re: pipes and sub-shells

[Dan Douglas]

On 03/22/2017 04:12 AM, Jean Delvare wrote:
> command | while read line
> do
>   if 
>   then
>   ex
> it 42
>   fi
>   process $line
> done
> 
> if [ $? -eq 42 ]
> then
>   exit
> fi

Splitting things into functions sometimes helps. Possibly just one:

 ~ $ mksh <<\EOF; echo "status: $?"
# Take a command and read a random number of lines
function reader {
  if ${1+'false'}; then
while read -r line; do
  if (( ! (RANDOM % 10) )); then
echo
return 42
  fi
  print -rn -- "$line "
done
  else
"$@" 2>/dev/null | reader || exit
  fi
}

reader yes I read my mind
EOF

I read my mind I read my mind I read my mind I read my mind
status: 42

Coprocesses are probably the most direct ksh alternative to  <() / >()
and lastpipe. Example (mksh / ksh93):

 ~ $ mksh <<\EOF; echo "status: $?"

LC_CTYPE=C tr -cd "[:digit:]" &p 3>&-  
> I was wondering if there is any other trick you can suggest that would work 
> in mksh?

I usually nag at Thorsten til I get $FEATURE. Worked for ksh93.
https://www.mail-archive.com/ast-users@lists.research.att.com/msg00900.html

Optionally disabling the subshell at runtime is useful for many things.

Re: pipes and sub-shells

[Thorsten Glaser]

Martijn Dekker dixit:

>Op 22-03-17 om 10:12 schreef Jean Delvare:
>> Concretely, the customer's code looks like this:
>>
>> command | while read line
>> do
>>  if 
>>  then
>>  exit
>>  fi
>>  process $line
>> done
>[...]
>> I was wondering if there is any other trick you can suggest that would work 
>> in mksh?
>
>A here-string with a command substitution:

That works but if not very performant.

This is actually a FAQ ;-) Use co-processes!

command |& while read -p line; do …


Dr. Werner Fink dixit:

>Just to be mentioned, I can mksh change in such a way that without job control
>I can do
>
>   ~/rpmbuild/BUILD/mksh> ./mksh -c 'echo xxx | read yyy; echo $yyy'
>   xxx
>
>that is it does work similar to the lastpipe shell option of the bash
>I had initiated now 7 years back. The trick is to avouid forking the
>last element in a pipe chain.

I’ve thought about this… years ago, too. But seeing as POSIX allows
either behaviour, I decided that this (running all sides of pipelines
in subshells) is a language feature now, for simplicity of user scripts
(as they can just use co-processes when they don’t want it).

So, thanks but no. (I’ve got scripts that actually rely on that now.)

bye,
//mirabilos
-- 
Solange man keine schmutzigen Tricks macht, und ich meine *wirklich*
schmutzige Tricks, wie bei einer doppelt verketteten Liste beide
Pointer XORen und in nur einem Word speichern, funktioniert Boehm ganz
hervorragend.   -- Andreas Bogk über boehm-gc in d.a.s.r

Re: pipes and sub-shells

[Dr. Werner Fink]

On Wed, Mar 22, 2017 at 10:12:37AM +0100, Jean Delvare wrote:
> Hi Thorsten,
> 
> One of our customers recently asked us about the different behavior of
> constructs involving pipes in ksh scripts. I explained to them that
> mksh spawns a sub-shell for the right hand side of each pipe, and
> therefore whatever happens on that side of the pipe has no effect on
> the main shell.
> 
> While I believe this is a perfectly sane implementation (and as a
> matter of fact, bash and dash behave the same, at least by default), I
> would like to make suggestions to the customer on how they can modify
> their code so that it keeps working.
> 
> With bash, I would have a number of ideas. For example, they could set
> option "lastpipe", which causes the last element of a pipe to be run in
> the main shell, instead of the first. Or they could use process
> substitution (the <() or >() syntax) to explicitly decide which part
> goes in a sub-shell. But it doesn't seem mksh implements any of these?
> 
> One approach which works in all shells is to use temporary files
> instead of pipes, however it could lower the performance of the
> scripts, so I consider it a last resort solution.
> 
> Concretely, the customer's code looks like this:
> 
> command | while read line
> do
>   if 
>   then
>   exit
>   fi
>   process $line
> done
> 
> ... more stuff ...
> 
> The "exit" doesn't work in mksh as it ends the sub-shell, not the main
> shell. One thing that would work in this case is exiting the sub-shell
> with a specific error code, and checking for that code afterward:
> 
> command | while read line
> do
>   if 
>   then
>   ex
> it 42
>   fi
>   process $line
> done
> 
> if [ $? -eq 42 ]
> then
>   exit
> fi
> 
> ... more stuff ...
> 
> I was wondering if there is any other trick you can suggest that would work 
> in mksh?

Just to be mentioned, I can mksh change in such a way that without job control
I can do

   ~/rpmbuild/BUILD/mksh> ./mksh -c 'echo xxx | read yyy; echo $yyy'
   xxx

that is it does work similar to the lastpipe shell option of the bash
I had initiated now 7 years back. The trick is to avouid forking the
last element in a pipe chain.

Don't know if this could help.

Werner

-- 
  "Having a smoking section in a restaurant is like having
  a peeing section in a swimming pool." -- Edward Burr


signature.asc
Description: PGP signature

Re: pipes and sub-shells

[Martijn Dekker]

Op 22-03-17 om 10:12 schreef Jean Delvare:
> Concretely, the customer's code looks like this:
> 
> command | while read line
> do
>   if 
>   then
>   exit
>   fi
>   process $line
> done
[...]
> I was wondering if there is any other trick you can suggest that would work 
> in mksh?

A here-string with a command substitution:

while read line
do
if 
then
exit
fi
process $line
done <<<$(command)

Hope this helps,

- M.

pipes and sub-shells

[Jean Delvare]

Hi Thorsten,

One of our customers recently asked us about the different behavior of
constructs involving pipes in ksh scripts. I explained to them that
mksh spawns a sub-shell for the right hand side of each pipe, and
therefore whatever happens on that side of the pipe has no effect on
the main shell.

While I believe this is a perfectly sane implementation (and as a
matter of fact, bash and dash behave the same, at least by default), I
would like to make suggestions to the customer on how they can modify
their code so that it keeps working.

With bash, I would have a number of ideas. For example, they could set
option "lastpipe", which causes the last element of a pipe to be run in
the main shell, instead of the first. Or they could use process
substitution (the <() or >() syntax) to explicitly decide which part
goes in a sub-shell. But it doesn't seem mksh implements any of these?

One approach which works in all shells is to use temporary files
instead of pipes, however it could lower the performance of the
scripts, so I consider it a last resort solution.

Concretely, the customer's code looks like this:

command | while read line
do
if 
then
exit
fi
process $line
done

... more stuff ...

The "exit" doesn't work in mksh as it ends the sub-shell, not the main
shell. One thing that would work in this case is exiting the sub-shell
with a specific error code, and checking for that code afterward:

command | while read line
do
if 
then
ex
it 42
fi
process $line
done

if [ $? -eq 42 ]
then
exit
fi

... more stuff ...

I was wondering if there is any other trick you can suggest that would work in 
mksh?

Thanks,
-- 
Jean Delvare
SUSE L3 Support