Re: Possibly a bug
On 4/20/15 5:01 PM, Valentin Bajrami wrote: Now when running ./history | his where his is not an existing command it fails and adds an entry in the job list. $ ./history | his [2]+ Stopped ./history | his bash: his: command not found... Running twice the output becomes: $ jobs -l [1]- 6556 Killed ./history 6557 Stopped (tty input) | his [2]+ 6830 Stopped (tty input) ./history 6831 | his I seem not to be able to clean up the jobs. Can anyone explain what is really happening? It's most likely that you have a shell function defined for your command-not-found hook (command_not_found_handle) and that function attempts to read from the tty, or calls a program that does, causing the process group to stop due to SIGTTIN. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Possibly a bug
Hi everyone,
While testing an awk script and piping the script to a command the
following seem to occur. Not sure if this is even a bash problem. Here are
some details about my configuration:
$ echo $BASH_VERSION
4.3.33(1)-release
The file called 'history' contains the following code
$ cat history
#!/bin/awk -f
BEGIN { OFS=\t ; print COMMAND, COUNT, HASHING }
{
a[$2]++
}
END { for (i in a) print a[i], i | sort -rn | head -n5; $1 max; max = $1
bar=
i=s=20*$1/max;while(i--0)bar=bar#; printf %5s %5d %s %s, $2,
$1,bar, \n
}
Now when running ./history | his where his is not an existing command it
fails and adds an entry in the job list.
$ ./history | his
[2]+ Stopped ./history | his
bash: his: command not found...
Running twice the output becomes:
$ jobs -l
[1]- 6556 Killed ./history
6557 Stopped (tty input) | his
[2]+ 6830 Stopped (tty input) ./history
6831 | his
I seem not to be able to clean up the jobs.
Can anyone explain what is really happening?
--
Met vriendelijke groet,
Valentin Bajrami
Re: [PATCH] circular buffer + hash for jobs.c:bgpids
On 4/19/15, 5:24 PM, Chet Ramey chet.ra...@case.edu wrote: On 4/17/15 4:55 PM, John Fremlin wrote: Did some benchmarks, for the while true; do (:) (:); done simple example this goes from 215 to 313 iterations/s, and changes sys+user CPU from 152% to 45% Any long running bash script will tend to exhibit this issue -- I disagree that `any' long-running bash script will create thousands of asychronous jobs and never call `wait' with no arguments. It's not clear that your example above is realistic. Let's see what happens with your patch to change bash to save statuses of asynchronous jobs instead of all jobs before we see whether another change is warranted. It¹s true that with the first patch then this one is less important and there will only be this long slowdown for bash sessions that have many asynchronous processes and that do not call wait with no arguments. However, why should there be a performance penalty of high order for using asynchronous processes? Calling wait with no arguments breaks modularity boundaries in that the caller of it precludes other callers within the same bash context from using asynchronous processes in an independent way. Independently of the real world probability of someone running while true; do : :; done (which we all must agree is low) this diff improves the performance of workloads with asynchronous processes, and also improves things by making sure we don¹t slavishly follow the CHILD_MAX limit which can be unusably low :)
Re: [Help-bash] make function local
On 4/20/15 4:04 PM, Peng Yu wrote: I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. Why do you assume that one only calls each of these functions once? What if a function is called thousands of time? Your example script called each function once. If you want to make a different argument, use a different example. Let's deconstruct your argument. Your assertion is that execution time will be dominated by the time it takes to look up a function in the shell's hash table, and that lookup time can be avoided in the case where there are huge numbers of functions -- on the order of more than 10,000 -- by adding local functions that are stored separately from the set of functions at the global scope. Correct? The performance difference is 10x different (0.490s vs 0.040s). To me, this is a huge performance problem. The performance difference is 10x when you define 10x functions. The difference is between 10,000, which is highly unlikely, to 100,000, which is truly outrageous. Are you seriously proposing that there are scripts in actual use that define 10,000 or 100,000 functions? I'd suggest that any script with that many functions deserves a rewrite, maybe in a different language. Functions are usually called for more times they are defined. Therefore, even if the function creation time is acceptable, it doesn't mean its call time is acceptable. That's true, but you haven't demonstrated that call time is unacceptable. The main point that Linda and I are making is that we suggest add the local function feature. I believe that we've made the case --- it makes the code cleaner and run faster. You've not made any such case. There's nothing other than your assertion that real-world benefits will accrue from adding local functions, and there is no reason to believe that any script in actual use will benefit from local functions. The sole advantage is the namespace issue: you would be able to define a function at a local scope with the same name as a function at the global scope. That would not appear to make code any cleaner, and there's certainly no evidence that there are performance problems that this would solve. Increasing the number of buckets in the hash table used to store shell functions, which I already did, will have a greater performance benefit. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Possibly a bug
Hi Chet,
I see. But how would I avoid this? Using Fedora 21 here and my
command_not_found_handle() is
command_not_found_handle ()
{
local runcnf=1;
local retval=127;
[[ $- =~ i ]] || runcnf=0;
[ ! -S /var/run/dbus/system_bus_socket ] runcnf=0;
[ ! -x /usr/libexec/packagekitd ] runcnf=0;
if [ $runcnf -eq 1 ]; then
/usr/libexec/pk-command-not-found $@;
retval=$?;
else
echo bash: $1: command not found;
fi;
return $retval
}
Would I then use kill -ttin %1 for example to remove the specified job
from the jobs list? Is this the behaviour of command_not_found_handle
function?
Thanks!
On Mon, Apr 20, 2015 at 11:13 PM, Chet Ramey chet.ra...@case.edu wrote:
On 4/20/15 5:01 PM, Valentin Bajrami wrote:
Now when running ./history | his where his is not an existing command it
fails and adds an entry in the job list.
$ ./history | his
[2]+ Stopped ./history | his
bash: his: command not found...
Running twice the output becomes:
$ jobs -l
[1]- 6556 Killed ./history
6557 Stopped (tty input) | his
[2]+ 6830 Stopped (tty input) ./history
6831 | his
I seem not to be able to clean up the jobs.
Can anyone explain what is really happening?
It's most likely that you have a shell function defined for your
command-not-found hook (command_not_found_handle) and that function
attempts to read from the tty, or calls a program that does, causing the
process group to stop due to SIGTTIN.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.edu
http://cnswww.cns.cwru.edu/~chet/
--
Met vriendelijke groet,
Valentin Bajrami
Re: Possibly a bug
On 4/20/15 5:24 PM, Valentin Bajrami wrote: Hi Chet, I see. But how would I avoid this? Using Fedora 21 here and my command_not_found_handle() is It's hard to say without knowing what pk-command-not-found does. What does this do when you just run `his' from the command line and let it execute in the foreground? What happens when you run `fg' to try and bring that pipeline back into the foreground? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
On Mon, Apr 20, 2015 at 03:04:00PM -0500, Peng Yu wrote: Hi Chet, I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. Why do you assume that one only calls each of these functions once? What if a function is called thousands of time? The point is, the performance is *perfectly acceptable* if your script only defines 10,000 functions instead of 100,000 functions. You are arguing to make a performance tuning adjustment for a situation that will never arise in real life. The performance difference is 10x different (0.490s vs 0.040s). To me, this is a huge performance problem. It would be a huge performance problem (maybe) if your script actually declared 100,000 functions. Do you have any scripts that declare 100,000 functions? The main point that Linda and I are making is that we suggest add the local function feature. I believe that we've made the case --- it makes the code cleaner and run faster. What makes you think adding function namespaces (or whatever means of localization) would make things run FASTER instead of slower?
Re: [Help-bash] make function local
On 4/19/15 10:38 PM, Peng Yu wrote: That's the difference: if you're careful with naming and rigorous about your calling conventions, your one-time-use functions are about as close as you can get to local functions in bash, but you have to pay attention to the declaration's side effects. There is at least a runtime overhead for having too many unused functions that are supposed to be local or in your word lambda. Maybe so, but this is not the discussion we were having. Despite that one can rename supposedly internal functions to names that are unlikely to cause name collision via good naming convention, it still can incur a significant performance overhead. I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. (And the example you chose to illustrate this is not what Linda is talking about or using.) In this sense, I think that it is still necessary to consider make the supposedly internal function local so that they would not slow down function search in the global namespace. Look, you can make the same argument about function creation at any scope, since there is one function namespace. Acceptable performance is subjective: if the technique that Linda uses for data encapsulation results in performance that's acceptable for her application, then it's ok for her to use it. You are certainly free to use any methodology you find comfortable and satisfies your constraints. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: readline-6.3: update config.h.in
On 4/20/15 2:59 AM, Andreas Schwab wrote: +#undef __EXTENSIONS__ +#undef _ALL_SOURCE +#undef _GNU_SOURCE +#undef _POSIX_SOURCE +#undef _POSIX_1_SOURCE +#undef _POSIX_PTHREAD_SEMANTICS +#undef _TANDEM_SOURCE + Readline configure.ac doesn't call AC_USE_SYSTEM_EXTENSIONS. Yes, it does, even if indirectly. The configure script defines them, so you need to add the templates. We'll try it, then. If you would use autoheader you'll just get what you ask for, no second hand guessing which will be wrong sooner or later. I'll take another look at it. It was no help when I looked at it previously. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: associative arrays and [[ -v
On 4/20/15, Chet Ramey chet.ra...@case.edu wrote: On 4/17/15 6:45 PM, isabella parakiss wrote: This seems the way to go, but I'm not sure I understand why: $ declare -A arr=([a]=b) $ [[ -v arr['$var'] ]]; echo $? 1 $ declare -A arr=(['*']=x) $ [[ -v arr['$var'] ]]; echo $? 0 What's happening? Well, as I said before, `*' is special: it expands to all elements of the array, in the same way as $*. So you have to protect it through all word expansions. First, each word in the conditional expression is expanded as described in the manual page tilde expansion, parameter and variable expansion, arithmetic expansion, command substitution, process substitution, and quote removal are performed. That leaves the word as 'arr[$var]' (without the quotes, of course). The subscript in an associative array reference also undergoes expansions, the same ones as the rhs of an assignment statement (pretty much the same as above). That expands the $var, leaving arr[*]. Since the check for `*' or `@' as a subscript happens before expansion, the shell looks for the `*' as an element of the array. In the first case, it doesn't find it; in the second, it does. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/ Thanks a lot for the explanation, that was tricky. --- xoxo iza
Re: [Help-bash] make function local
Hi Chet, That's the difference: if you're careful with naming and rigorous about your calling conventions, your one-time-use functions are about as close as you can get to local functions in bash, but you have to pay attention to the declaration's side effects. There is at least a runtime overhead for having too many unused functions that are supposed to be local or in your word lambda. Maybe so, but this is not the discussion we were having. Despite that one can rename supposedly internal functions to names that are unlikely to cause name collision via good naming convention, it still can incur a significant performance overhead. I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. Why do you assume that one only calls each of these functions once? What if a function is called thousands of time? The performance difference is 10x different (0.490s vs 0.040s). To me, this is a huge performance problem. (And the example you chose to illustrate this is not what Linda is talking about or using.) In this sense, I think that it is still necessary to consider make the supposedly internal function local so that they would not slow down function search in the global namespace. Look, you can make the same argument about function creation at any scope, since there is one function namespace. Functions are usually called for more times they are defined. Therefore, even if the function creation time is acceptable, it doesn't mean its call time is acceptable. Acceptable performance is subjective: if the technique that Linda uses for data encapsulation results in performance that's acceptable for her application, then it's ok for her to use it. You are certainly free to use any methodology you find comfortable and satisfies your constraints. The main point that Linda and I are making is that we suggest add the local function feature. I believe that we've made the case --- it makes the code cleaner and run faster. What makes you reluctant to consider this feature be added? Is it takes too much time to implement such a feature? -- Regards, Peng
Re: readline-6.3: update config.h.in
Chet Ramey chet.ra...@case.edu writes: On 4/19/15 2:00 PM, Andreas Schwab wrote: The config.h.in file in the standalone readline distribution is missing a lot of templates. It should really be generated by autoheader. --- readline-6.3/config.h.in.~1~ 2012-01-18 16:22:13.0 +0100 +++ readline-6.3/config.h.in 2015-04-19 19:39:13.215290092 +0200 @@ -6,6 +6,14 @@ #undef _FILE_OFFSET_BITS +#undef __EXTENSIONS__ +#undef _ALL_SOURCE +#undef _GNU_SOURCE +#undef _POSIX_SOURCE +#undef _POSIX_1_SOURCE +#undef _POSIX_PTHREAD_SEMANTICS +#undef _TANDEM_SOURCE + Readline configure.ac doesn't call AC_USE_SYSTEM_EXTENSIONS. Yes, it does, even if indirectly. The configure script defines them, so you need to add the templates. Readline doesn't actually call any of these functions, and even though BASH_CHECK_MULTIBYTE tests for them, it doesn't matter whether or not it finds them. If you would use autoheader you'll just get what you ask for, no second hand guessing which will be wrong sooner or later. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 And now for something completely different.
