Dear Chet Ramey, thanks for the explanations, I assume you were able to reproduce the issue.
First a warning that I forgot to include in the previous email: the quoted excerpts of the gprof tables correspond to a slightly modified test script in which I tried to magnify the overhead in order to more easily spot performance differences. Therefore, the absolute values displayed, e.g. number of function calls, are exaggerated. The relative values (and hopefully your conclusions) remain largely the same, though. Sorry for any potential confusion. A few short comments prompted by your reply (probably not very useful, feel free to ignore them). Chet Ramey: >> - $() seems generally slightly slower than ``, but becomes pathologically >> so when preceded with "eval set -- ...". > > It is slightly slower -- POSIX requires that the shell parse the contents > of $(...) to determine that it's a valid script as part of finding the > closing `)'. The rules for finding the closing "`" don't have that > requirement. > >> - "eval set -- ..." itself doesn't seem slow at all, but obviously it has >> side-effects not captured by the "time" measurement tool. > > What happens is you end up with a 4900-character command string that you > have to parse multiple times. But that's not the worst of it. Since this statement ought to run exactly once, naïvely I would expect that by "multiple times" you really mean at most "twice": once for the top-level script, another time "inside" the eval "sub-script". > The gprof output provides a clue. > > >> case 1 1 0 (pathological): >> % cumulative self self total >> time seconds seconds calls us/call us/call name >> 38.89 0.21 0.21 28890 7.27 7.27 set_line_mbstate > > set_line_mbstate() runs through each command line before parsing, creating > a bitmap that indicates whether each element is a single-byte character or > part of a multi-byte character. The scanner uses this to determine whether > a shell metacharacter should act as a delimiter or get skipped over as part > of a multibyte character. For a single run with args `1 1 0', it gets > called around 7300 times, with around 2400 of them for the 4900-character > string with all the arguments. > > When you're in a multibyte locale (en_US.UTF-8 is one such), each one of > those characters requires a call to mbrlen/mbrtowc. So that ends up being > 2400 * 4900 calls to mbrlen. I am indeed using an UTF-8 locale, but I tested also with export LC_ALL=C and the behaviour did not change, I should have mentioned that. Also, it occurs to me that since all shell metacharacters are surely 7-bit ASCII, and all UTF-8 code units are strictly and deliberately non-ASCII 8-bit bytes, at least in the case of UTF-8 it may not be necessary to construct such bitmap: there cannot be a shell-metacharacter byte part of a UTF-8 sequence representing something else, so there's no need for the distinction. Of course, this observation applies only to such specially crafted multibyte encodings as UTF-8 (which nonetheless is surely by far the most common). Furthermore, it does not explain the issue at hand. > There is something happening here -- there's no way there should be that > many calls to set_line_mbstate(), Notice that there are almost as many calls (only 2 fewer) in case "0 1 0" (in which eval is not used) yet in that case the performance is not harmed. Quoting from the previous email: case 0 1 0: 2.08 0.32 0.01 28888 0.00 0.00 set_line_mbstate case 1 1 0 (pathological): 38.89 0.21 0.21 28890 7.27 7.27 set_line_mbstate OTOH, all other test cases show less than 100 calls (these numbers correspond to a profiling run of the given script, unmodified): $ grep -m1 set_line_mbstate gmon*.table gmon.out.0-0-X.table: 0.00 0.00 0.00 88 0.00 0.00 set_line_mbstate gmon.out.0-1-0.table: 0.00 0.13 0.00 7288 0.00 0.00 set_line_mbstate gmon.out.0-1-1.table: 0.00 0.18 0.00 88 0.00 0.00 set_line_mbstate gmon.out.1-0-X.table: 0.00 0.01 0.00 90 0.00 0.00 set_line_mbstate gmon.out.1-1-0.table: 37.50 0.06 0.06 7290 8.23 8.23 set_line_mbstate gmon.out.1-1-1.table: 0.00 0.07 0.00 90 0.00 0.00 set_line_mbstate > even when you have to save and restore > the input line because you have to parse the contents of $(). There must > be some combination of the effect of `eval' on the line bitmap and the > long string. I'll see what I can figure out. > > Chet Cheers.