Re: [Help-bash] make function local
On 4/20/15 4:04 PM, Peng Yu wrote: I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. Why do you assume that one only calls each of these functions once? What if a function is called thousands of time? Your example script called each function once. If you want to make a different argument, use a different example. Let's deconstruct your argument. Your assertion is that execution time will be dominated by the time it takes to look up a function in the shell's hash table, and that lookup time can be avoided in the case where there are huge numbers of functions -- on the order of more than 10,000 -- by adding local functions that are stored separately from the set of functions at the global scope. Correct? The performance difference is 10x different (0.490s vs 0.040s). To me, this is a huge performance problem. The performance difference is 10x when you define 10x functions. The difference is between 10,000, which is highly unlikely, to 100,000, which is truly outrageous. Are you seriously proposing that there are scripts in actual use that define 10,000 or 100,000 functions? I'd suggest that any script with that many functions deserves a rewrite, maybe in a different language. Functions are usually called for more times they are defined. Therefore, even if the function creation time is acceptable, it doesn't mean its call time is acceptable. That's true, but you haven't demonstrated that call time is unacceptable. The main point that Linda and I are making is that we suggest add the local function feature. I believe that we've made the case --- it makes the code cleaner and run faster. You've not made any such case. There's nothing other than your assertion that real-world benefits will accrue from adding local functions, and there is no reason to believe that any script in actual use will benefit from local functions. The sole advantage is the namespace issue: you would be able to define a function at a local scope with the same name as a function at the global scope. That would not appear to make code any cleaner, and there's certainly no evidence that there are performance problems that this would solve. Increasing the number of buckets in the hash table used to store shell functions, which I already did, will have a greater performance benefit. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
On Mon, Apr 20, 2015 at 03:04:00PM -0500, Peng Yu wrote: Hi Chet, I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. Why do you assume that one only calls each of these functions once? What if a function is called thousands of time? The point is, the performance is *perfectly acceptable* if your script only defines 10,000 functions instead of 100,000 functions. You are arguing to make a performance tuning adjustment for a situation that will never arise in real life. The performance difference is 10x different (0.490s vs 0.040s). To me, this is a huge performance problem. It would be a huge performance problem (maybe) if your script actually declared 100,000 functions. Do you have any scripts that declare 100,000 functions? The main point that Linda and I are making is that we suggest add the local function feature. I believe that we've made the case --- it makes the code cleaner and run faster. What makes you think adding function namespaces (or whatever means of localization) would make things run FASTER instead of slower?
Re: [Help-bash] make function local
On 4/19/15 10:38 PM, Peng Yu wrote: That's the difference: if you're careful with naming and rigorous about your calling conventions, your one-time-use functions are about as close as you can get to local functions in bash, but you have to pay attention to the declaration's side effects. There is at least a runtime overhead for having too many unused functions that are supposed to be local or in your word lambda. Maybe so, but this is not the discussion we were having. Despite that one can rename supposedly internal functions to names that are unlikely to cause name collision via good naming convention, it still can incur a significant performance overhead. I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. (And the example you chose to illustrate this is not what Linda is talking about or using.) In this sense, I think that it is still necessary to consider make the supposedly internal function local so that they would not slow down function search in the global namespace. Look, you can make the same argument about function creation at any scope, since there is one function namespace. Acceptable performance is subjective: if the technique that Linda uses for data encapsulation results in performance that's acceptable for her application, then it's ok for her to use it. You are certainly free to use any methodology you find comfortable and satisfies your constraints. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
Hi Chet, That's the difference: if you're careful with naming and rigorous about your calling conventions, your one-time-use functions are about as close as you can get to local functions in bash, but you have to pay attention to the declaration's side effects. There is at least a runtime overhead for having too many unused functions that are supposed to be local or in your word lambda. Maybe so, but this is not the discussion we were having. Despite that one can rename supposedly internal functions to names that are unlikely to cause name collision via good naming convention, it still can incur a significant performance overhead. I disagree that performance overhead in typical use is `significant'. This point is more or less identical to the one I discussed Friday in regards to creating huge numbers of variables. Who creates 100,000 shell functions in a single script? The overhead, such as it is, of creating and calling even 10,000 functions is negligible. Why do you assume that one only calls each of these functions once? What if a function is called thousands of time? The performance difference is 10x different (0.490s vs 0.040s). To me, this is a huge performance problem. (And the example you chose to illustrate this is not what Linda is talking about or using.) In this sense, I think that it is still necessary to consider make the supposedly internal function local so that they would not slow down function search in the global namespace. Look, you can make the same argument about function creation at any scope, since there is one function namespace. Functions are usually called for more times they are defined. Therefore, even if the function creation time is acceptable, it doesn't mean its call time is acceptable. Acceptable performance is subjective: if the technique that Linda uses for data encapsulation results in performance that's acceptable for her application, then it's ok for her to use it. You are certainly free to use any methodology you find comfortable and satisfies your constraints. The main point that Linda and I are making is that we suggest add the local function feature. I believe that we've made the case --- it makes the code cleaner and run faster. What makes you reluctant to consider this feature be added? Is it takes too much time to implement such a feature? -- Regards, Peng
Re: [Help-bash] make function local
On 4/17/15 6:27 PM, Linda Walsh wrote: Eduardo A. Bustamante López wrote: Well, if your scripts are so simple, why use local functions at all? --- Cleanliness, Hygiene... Please, let's not have this argument again. I think you're all using the term `local function' to mean different things. You seem to be using the term to describe one-time-use functions (they're almost lambdas, but they have names). If you define a function within another function's body, have it unset itself when it executes, and call it only from within the function where it's defined, you have something very close to one-time-use functions with function-only scope. However you use them, they share the name namespace as every other defined function, regardless of whether or not they are defined as part of the body of another function. If you had a function defined in the global scope with the same name as your one-time-use function, it would be removed when the one-time-use function was declared. I think this is what Greg and Eduardo mean, and in this sense they are correct: bash doesn't have local functions with separate namespaces from other defined functions. That's the difference: if you're careful with naming and rigorous about your calling conventions, your one-time-use functions are about as close as you can get to local functions in bash, but you have to pay attention to the declaration's side effects. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
Hi Chet,
Eduardo A. Bustamante López wrote:
Well, if your scripts are so simple, why use local functions at all?
---
Cleanliness, Hygiene...
Please, let's not have this argument again. I think you're all using the
term `local function' to mean different things.
You seem to be using the term to describe one-time-use functions (they're
almost lambdas, but they have names). If you define a function within
another function's body, have it unset itself when it executes, and call
it only from within the function where it's defined, you have something
very close to one-time-use functions with function-only scope.
However you use them, they share the name namespace as every other defined
function, regardless of whether or not they are defined as part of the body
of another function. If you had a function defined in the global scope
with the same name as your one-time-use function, it would be removed when
the one-time-use function was declared. I think this is what Greg and
Eduardo mean, and in this sense they are correct: bash doesn't have local
functions with separate namespaces from other defined functions.
That's the difference: if you're careful with naming and rigorous about
your calling conventions, your one-time-use functions are about as close
as you can get to local functions in bash, but you have to pay attention
to the declaration's side effects.
There is at least a runtime overhead for having too many unused
functions that are supposed to be local or in your word lambda.
Despite that one can rename supposedly internal functions to names
that are unlikely to cause name collision via good naming convention,
it still can incur a significant performance overhead.
In this sense, I think that it is still necessary to consider make the
supposedly internal function local so that they would not slow down
function search in the global namespace.
~$ cat main_many.sh
#!/usr/bin/env bash
n=$1
for i in $(seq -w $n)
do
eval function f$i { echo '$i'; }
done
function g {
for i in $(seq -w $n | head -n 1000)
do
f$i /dev/null
done
}
time g
~$ ./main_many.sh 1000
real 0m0.032s
user 0m0.020s
sys 0m0.010s
~$ ./main_many.sh 1
real 0m0.040s
user 0m0.029s
sys 0m0.011s
~$ ./main_many.sh 10
real 0m0.490s
user 0m0.461s
sys 0m0.031s
--
Regards,
Peng
Re: [Help-bash] make function local
On Thu, Apr 16, 2015 at 10:38:22PM -0700, Linda Walsh wrote:
It's a trite example, but I do something like:
sub gvim () {
array orig_args=($@) gv_files=() gv_ops=()
int use_tab=0 look_for_ops=1
sub _exec_gvim() {
array args
((use_tab)) args=(-p)
(( ${#gv_ops[@]:-0} )) args+=(${gv_ops[@]})
(( $# )) args+=($@)
command gvim ${args[@]}
unset -f _exec_gvim
}
AFAIK, _exec_gvim, can only be called from within function gvim, no?
That is not correct. In bash, all function definitions are global.
imadev:~$ f1() { f2() { echo I'm f2; }; }; f1; f2
I'm f2
Re: [Help-bash] make function local
On Thu, Apr 16, 2015 at 10:38:22PM -0700, Linda Walsh wrote:
[...]
AFAIK, _exec_gvim, can only be called from within function gvim, no?
No. Doing this:
outerfunc() {
func() { ...; }
func args
unset -f func
}
outerfunc
Doesn't guarantee that `func' will be only called from outerfunc. If you have
traps defined they can switch to other contexts where that funtion will be
available.
It's sad. Why do people keep pretending bash is something it isn't :-)?
--
Eduardo Bustamante
https://dualbus.me/
Re: [Help-bash] make function local
Greg Wooledge wrote:
AFAIK, _exec_gvim, can only be called from within function gvim, no?
That is not correct. In bash, all function definitions are global.
imadev:~$ f1() { f2() { echo I'm f2; }; }; f1; f2
I'm f2
You left out a few important parts, like f2 being called inside f1 before
it returns, and unsetting f2:
f1() { f2() { echo I'm f2 $1; unset -f f2; }; f2 internal; }; f1; f2 external
I'm f2 internal
-bash: f2: command not found
Hmmm...
Seems to work for me.
Re: [Help-bash] make function local
On Fri, Apr 17, 2015 at 11:58:15AM -0700, Linda Walsh wrote:
outerfunc() {
func() { ...; }
func args
unset -f func
}
outerfunc
The problem is, ALL function definitions (and un-definitions) are global.
If there was a func defined at the global scope, you've obliterated it.
I don't understand why you don't understand this.
imadev:~$ f() { echo global f; }
imadev:~$ f2() { f() { echo local f; }; unset -f f; }
imadev:~$ f2
imadev:~$ f
bash: f: command not found
Bash is just like C in many ways. This is one of them -- there is only
a single global namespace for all functions.
Doing trickery to make it LOOK LIKE you have a local function just leads
to confusion. The person who inherits this script from you after you're
gone may think the function is local, when it isn't.
So, what is the point of doing this convoluted trickery, when it
accomplishes nothing (it doesn't create a local function, and it doesn't
prevent contamination of the global function namespace)? Just to
see how clever you can be?
Re: [Help-bash] make function local
Eduardo A. Bustamante López wrote:
On Thu, Apr 16, 2015 at 10:38:22PM -0700, Linda Walsh wrote:
[...]
AFAIK, _exec_gvim, can only be called from within function gvim, no?
No. Doing this:
outerfunc() {
func() { ...; }
func args
unset -f func
}
outerfunc
Doesn't guarantee that `func' will be only called from outerfunc. If you have
traps defined they can switch to other contexts where that funtion will be
available.
It's sad. Why do people keep pretending bash is something it isn't :-)?
It's sad, when you have to work at breaking people's example.
The program doesn't use traps.
Primarily, I didn't want to leave '_exec_vim' left in the namespace
after it had run. It accomplishes that.
You can try hard enough to break anything. Java was designed to
be a secure computing language from the start, yet it's track
record in security doesn't set it apart from any other language.
If you can't start with a 'fresh' language to design something
secure, how can you expect to add duct-tape and bailing wire to
30-40 year-old technology to make it secure.
Most security experts have advanced beyond the stage of the
silver bullet, and have finally come into agreement with where
I was, at least, 15 years ago -- security in layers. None of the
layers are secure, but by the time a cracker finishes peeling off
the layers of the onion, they'll be in tears.
Re: [Help-bash] make function local
Greg Wooledge wrote:
The problem is, ALL function definitions (and un-definitions) are global.
---
yeah...
If there was a func defined at the global scope, you've obliterated it.
I don't understand why you don't understand this.
---
I don't understand why you think I don't understand this.
I showed an example where I wanted to use a local function in
a stand-alone script -- there were no libraries involved. There was
no previous definition. The function was deleted if it was called
once. No other interactions were specified.
I don't get why you toss in all sorts of other possible interactions
that I didn't specify nor that existed. Sure, if I pull the plug from
the wall all of your examples on your website won't work. So should
I question your examples because they don't work under all possible
circumstances? That's ridiculous.
Doing trickery to make it LOOK LIKE you have a local function just leads
to confusion. The person who inherits this script from you after you're
gone may think the function is local, when it isn't.
---
I'm quite clear about how I use throw-away functions.
So, what is the point of doing this convoluted trickery, when it
accomplishes nothing (it doesn't create a local function, and it doesn't
prevent contamination of the global function namespace)? Just to
see how clever you can be?
---
No, as an example where a local function would be of use --
and the clever trickery (I said it as a trite example) involved.
If I wanted clever trickery, I'd use a function generator
that generated function names with a GUID ending -- then test if that
exists and repeat if there is a collision. That would be more my idea
of clever trickery, since Guids are considered to be Globally Unique
(even though they really aren't), it would be as solid as a true local
functions as much as GUIDS are really globally unique -- sorta something
like:
sub uuid_name {
my uuid=$(uuidgen)
echo ${uuid//-/_}
}
sub filterarray {
my arrayname=${1:?}
my filter=${2:?}
my t=$(type -t $filter)
if [[ $t!=function $t!=alias ]]; then
my uname=F_$(uuid_name)
eval sub $uname { $filter }
fi
}
---
Would be more my idea of clever trickery -- and would be just
as globally unique as GUID's are taken to be (regen GUID's until
you find one that doesn't collide (note the above code doesn't
loop on a test, but was a random scriplet written a year ago
to capture the idea of wrapping some 'filter' code in a sub
if it wasn't already an alias or a sub.
Of course, I'm sure you already thought of creating function
names on the fly using GUID's to ensure they don't collide
globally, tricky guy that you are... ;-)
Re: [Help-bash] make function local
Well, if your scripts are so simple, why use local functions at all? You're claiming we invent stuff to make your examples fail, but I don't know anyone that writes such complex code for very simple tasks that can even be done without functions. So, the burden to prove these convoluted approaches are justified is on *your* side. The local function you provided is clearly a fake case. Provide real world cases, so that we can make real world criticism and take decisions that affect people writing actual useful code. If that's the complete script, why unset the functions? Security? That's the whole script, what other functions could there be that are affected by a left over function name in the script's global namespace? -- Eduardo Bustamante https://dualbus.me/
Re: [Help-bash] make function local
Eduardo A. Bustamante López wrote: Well, if your scripts are so simple, why use local functions at all? --- Cleanliness, Hygiene... You're claiming we invent stuff to make your examples fail, but I don't know anyone that writes such complex code for very simple tasks that can even be done without functions. --- You know of me -- but you know nothing of my use case. So, the burden to prove these convoluted approaches are justified is on *your* side. --- I don't need to justify my code to you. Someone asked for a use case, and I supplied one. I didn't write that code for this discussion. Last mod-time was 1 month ago (Mar 17). I wrote it BEFORE any of this discussion on local functions. The local function you provided is clearly a fake case. --- If it is clearly a fake case, you are clearly an idiot. I've been using that code for it's purpose, unchanged for over a month. That's what is clear. Provide real world cases, so that we can make real world criticism and take decisions that affect people writing actual useful code. Your definition of real-world cases are ones that you can provide real world criticism to shoot down any example provided. I'm not the only one who notices that tendency on this list: Peng Yu wrote: On Tue, Apr 7, 2015 at 8:49 PM, Chet Ramey chet.ra...@case.edu wrote: [Show us your valid real world example] One could also ask the same question for local variables. Any limited cases that show local variable is need, by definition, can be changed to ones in which global variables can also work. Therefore, ***no matter what small examples that I should show here, it will *** ***always be criticized as can be solved by an alternative solution ***. - [Emphasis mine]. Peng Yu says the same thing. No matter what case someone comes up with, there are the rigid-thinkers who believe they, and they alone can judge what are real world and not fake cases. GAG! If that's the complete script... Now you are setting up your own strawman cases to shoot down. I showed 12 lines with an *ellipses* after the code out of a 70 line script and you start making arguments based on those 12 lines being the entirety of the script. I deliberately didn't show the rest of the script because it was not important to show a use case developed and in use long before this discussion was started --- so unless you have some evidence that it is a 'fake' script, I'd say you are purposely lying to support your case. You don't read what we write. Just like with my note IFS=: splitting paths -- (maybe fixed in 4.3?), where you answered what the fuck. If you had read the entire note -- I pointed out 4 examples of behavior that exists, asking rhetorical questions about how the behavior was justified -- because the last example contradicted the previous examples. You skipped that last question in your response, and totally missed the point FLAMING me for my observations and questioning of behavior on the 1st 4.Then you again go off on me saying: I guess you think that you look smart by obfuscating your code with aliases and weird names, but it's the opposite effect. Also, it annoys people that are trying to understand what you say to 'help'[sic] you. Another example of your twisting words and not reading what is there: I talked about IFS being 'thrashed' -- i.e. it no longer has it's initial default value, and there is no way to set it to default other than reinitializing it with some arbitrary hardcoded default. You go off and say what do you mean by 'IFS is trashed'? Notice thrash -- from goog: define thrashed 1st entry urban dictionary: thrashed -- destroyed or hurt really badly, usually used to refer to someone after they have tried a huge gap and died. I was referring to a built-in variable that had it's default meaning (contents) overwritten to the point of not being able to recover it (except by hardcoding the current default into an assignment). Please stop looking to pick apart my words. It doesn't matter WHAT type of example I come up with. You will find some reason to _not_ understand it and call it fake, unreal or annoying. Please, Eduardo: learn some new way of helping people -- calling them liars (accusing them of writing fake code) isn't helpful.
Re: [Help-bash] make function local
Hey Linda. I do remember that thread, and I apologize for my words. I honestly try my best here. But it seems that I cannot reply to you without offending you, so, to avoid further offenses I will not reply to any further email from you in the future. I'm sorry that I offended you, it was not my intention, though that doesn't matter much. On Fri, Apr 17, 2015 at 03:27:12PM -0700, Linda Walsh wrote: Eduardo A. Bustamante López wrote: Well, if your scripts are so simple, why use local functions at all? --- Cleanliness, Hygiene... You're claiming we invent stuff to make your examples fail, but I don't know anyone that writes such complex code for very simple tasks that can even be done without functions. --- You know of me -- but you know nothing of my use case. So, the burden to prove these convoluted approaches are justified is on *your* side. --- I don't need to justify my code to you. Someone asked for a use case, and I supplied one. I didn't write that code for this discussion. Last mod-time was 1 month ago (Mar 17). I wrote it BEFORE any of this discussion on local functions. The local function you provided is clearly a fake case. --- If it is clearly a fake case, you are clearly an idiot. I've been using that code for it's purpose, unchanged for over a month. That's what is clear. Provide real world cases, so that we can make real world criticism and take decisions that affect people writing actual useful code. Your definition of real-world cases are ones that you can provide real world criticism to shoot down any example provided. I'm not the only one who notices that tendency on this list: Peng Yu wrote: On Tue, Apr 7, 2015 at 8:49 PM, Chet Ramey chet.ra...@case.edu wrote: [Show us your valid real world example] One could also ask the same question for local variables. Any limited cases that show local variable is need, by definition, can be changed to ones in which global variables can also work. Therefore,***no matter what small examples that I should show here, it will *** ***always be criticized as can be solved by an alternative solution ***. - [Emphasis mine]. Peng Yu says the same thing. No matter what case someone comes up with, there are the rigid-thinkers who believe they, and they alone can judge what are real world and not fake cases. GAG! If that's the complete script... Now you are setting up your own strawman cases to shoot down. I showed 12 lines with an *ellipses* after the code out of a 70 line script and you start making arguments based on those 12 lines being the entirety of the script. I deliberately didn't show the rest of the script because it was not important to show a use case developed and in use long before this discussion was started --- so unless you have some evidence that it is a 'fake' script, I'd say you are purposely lying to support your case. You don't read what we write. Just like with my note IFS=: splitting paths -- (maybe fixed in 4.3?), where you answered what the fuck. If you had read the entire note -- I pointed out 4 examples of behavior that exists, asking rhetorical questions about how the behavior was justified -- because the last example contradicted the previous examples. You skipped that last question in your response, and totally missed the point FLAMING me for my observations and questioning of behavior on the 1st 4.Then you again go off on me saying: I guess you think that you look smart by obfuscating your code with aliases and weird names, but it's the opposite effect. Also, it annoys people that are trying to understand what you say to 'help'[sic] you. Another example of your twisting words and not reading what is there: I talked about IFS being 'thrashed' -- i.e. it no longer has it's initial default value, and there is no way to set it to default other than reinitializing it with some arbitrary hardcoded default. You go off and say what do you mean by 'IFS is trashed'? Notice thrash -- from goog: define thrashed 1st entry urban dictionary: thrashed -- destroyed or hurt really badly, usually used to refer to someone after they have tried a huge gap and died. I was referring to a built-in variable that had it's default meaning (contents) overwritten to the point of not being able to recover it (except by hardcoding the current default into an assignment). Please stop looking to pick apart my words. It doesn't matter WHAT type of example I come up with. You will find some reason to _not_ understand it and call it fake, unreal or annoying. Please, Eduardo: learn some new way of helping people -- calling them liars (accusing them of writing fake code) isn't helpful. -- Eduardo Bustamante https://dualbus.me/
Re: [Help-bash] make function local
On 4/12/15 5:56 PM, Eduardo A. Bustamante López wrote: Oh, you already have lots of things to do to bother with this :-) Anyways, I'll expand them. On Fri, Apr 10, 2015 at 04:35:25PM -0400, Chet Ramey wrote: On 4/10/15 10:13 AM, Eduardo A. Bustamante López wrote: - a faster implementation of the variable lookup code What does this mean, exactly? Optimizing the existing code paths? (Have at it.) Different semantics? Static as opposed to dynamic scoping? Yes. I've been using gprof to study the code paths of some basic functions, and it seems like it spends quite some time in the find_variable() and related functions (IIRC, there was an mt_hash or something function taking up some precious time). I knew that rang a bell somewhere. mt_hash is a function in the bash malloc library that keeps track of all allocations and deallocations in a table. It's part of the debugging that is enabled when you build from the devel code. It's been well-known for a long time that the debugging code in malloc slows bash down considerably; that's why it's not enabled as part of bash releases. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
On 4/12/15 5:56 PM, Eduardo A. Bustamante López wrote: Oh, you already have lots of things to do to bother with this :-) Anyways, I'll expand them. On Fri, Apr 10, 2015 at 04:35:25PM -0400, Chet Ramey wrote: On 4/10/15 10:13 AM, Eduardo A. Bustamante López wrote: - a faster implementation of the variable lookup code What does this mean, exactly? Optimizing the existing code paths? (Have at it.) Different semantics? Static as opposed to dynamic scoping? Yes. I've been using gprof to study the code paths of some basic functions, and it seems like it spends quite some time in the find_variable() and related functions (IIRC, there was an mt_hash or something function taking up some precious time). I'm not sure if it might be better to have other kind of data structure for this. TBH, I'm not sure if there's even enough justification for this, other than to make bash startup faster. - a shopt to disable evaluation of shell code in places like arithmetic expansion Remember this thread? http://lists.gnu.org/archive/html/bug-bash/2014-12/msg00158.html Sure, of course. Here's how I summarized the concern: assignment statements in arithmetic expressions that contain array references are also word expanded, almost as if they were executed in an assignment statement context At one point, this was brought up: dualbus@hp ~/t % bash -c 'var=a[\$(ls)]; a=(); a[var]=x; declare -p a' bash: bar baz foo: syntax error in expression (error token is baz foo) I understand the reasons behind it. This time I don't want to debate that :-) But, wouldn't it be nice to have a `arith_expand' or something shopt that when turned off, this happened: OK, but you're going to have to specify it more tightly than that. The first question is how bash treat tokens that look like identifiers in arithemtic expression contexts: do you treat them as variables that may specify expressions, or do you treat them as variables whose values must be integer constants? Then you have to specify which word expansions you'd like expressions to undergo, and which word expansions you'd like array subscripts to undergo in case they're different, and in which contexts you'd like that to happen. The answer to the first question should determine whether and why a[var]=x and a[a[\$(ls)]]=x from your example should behave differently. Or is it some middle ground you want: that identifiers are expanded and the expanded values are treated as expressions, but those expressions don't undergo any word expansions. That still leaves the question of what to do about array subscripts in these expressions. That should be enough to get a discussion started. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Bash performance when declaring variables (was: Re: [Help-bash] make function local)
On Thu, Apr 16, 2015 at 11:07:34AM -0400, Chet Ramey wrote: [...] I knew that rang a bell somewhere. mt_hash is a function in the bash malloc library that keeps track of all allocations and deallocations in a table. It's part of the debugging that is enabled when you build from the devel code. It's been well-known for a long time that the debugging code in malloc slows bash down considerably; that's why it's not enabled as part of bash releases. Actually, this is the post that motivated me to look into this: (yes, the conclusion is idiotic, but I guess the rest of the post is pretty okay). http://spencertipping.com/posts/2013.0814.bash-is-irrecoverably-broken.html Now, there is some truth to what he says: dualbus@yaqui ...src/gnu/bash % time ./bash -c 'i=0; while ((i++1000)); do declare a$RANDOM$RANDOM=1; done' ./bash -c 'i=0; while ((i++1000)); do declare a$RANDOM$RANDOM=1; done' 0.01s user 0.06s system 93% cpu 0.077 total dualbus@yaqui ...src/gnu/bash % time ./bash -c 'i=0; while ((i++1)); do declare a$RANDOM$RANDOM=1; done' ./bash -c 'i=0; while ((i++1)); do declare a$RANDOM$RANDOM=1; done' 0.16s user 0.48s system 98% cpu 0.643 total dualbus@yaqui ...src/gnu/bash % time ./bash -c 'i=0; while ((i++10)); do declare a$RANDOM$RANDOM=1; done' ./bash -c 'i=0; while ((i++10)); do declare a$RANDOM$RANDOM=1; done' 15.44s user 6.51s system 99% cpu 21.959 total I built bash like this: CFLAGS='-pg -g -O0' ./configure --silent make -sj4 DEBUG= MALLOC_DEBUG= To make sure the malloc debugging code doesn't interfere. I got a gprof profile with that last run, which gave: Each sample counts as 0.01 seconds. % cumulative self self total time seconds secondscalls s/call s/call name 71.42 12.0712.07 1100104 0.00 0.00 hash_search 21.18 15.65 3.58 275435 0.00 0.00 morecore 1.63 15.93 0.28 6800525 0.00 0.00 internal_malloc 0.71 16.05 0.12 6200116 0.00 0.00 internal_free 0.59 16.15 0.10 31 0.00 0.00 expand_word_internal 0.24 16.19 0.04 6800474 0.00 0.00 sh_xmalloc 0.18 16.22 0.03 7203779 0.00 0.00 is_basic 0.18 16.25 0.03 1932530 0.00 0.00 is_basic 0.18 16.28 0.03 22 0.00 0.00 subexpr 0.18 16.31 0.03 18 0.00 0.00 find_special_var 0.15 16.33 0.03 pagealign Notice how it spends most of the time in these two functions. Yeah, it's not mt_* like I said, because I did this a time ago and forgot to take notes. Does this matter much? I don't know. Having 100,000 variables declared does seem like something stupid. Still, it shouldn't have that quadratic increase in performance (I didn't even try for 1,000,000 because it was very slow), because it is a hash table. -- Eduardo Bustamante https://dualbus.me/
Re: [Help-bash] make function local
On 4/16/15 11:43 AM, Dan Douglas wrote: I thought Bash always first splits the identifier from the subscript, then checks which attributes the variable has set. If it has the associative array attribute plus a subscript then the subscript is only processed for expansions and the resulting string is used as the key. If the associative array attribute is not set then the subscript is processed for expansions and the resulting string is passed on to arithmetic evaluation. Am I following the discussion correctly? i.e. if you have `a[b[text]]`, the treatment of `text` is entirely determined by b's attributes. Yes, that's correct. In the case I'm talking about, we're only concerned with indexed arrays and the consequent arithmetic evaluation. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
Pierre Gaston wrote:
Is there a particular problem you're trying to solve for which local
functions would be the appropriate solution?
Cleanliness.
Not polluting the global namespace.
Ensuring the function can't be called from outside a function.
It's a trite example, but I do something like:
sub gvim () {
array orig_args=($@) gv_files=() gv_ops=()
int use_tab=0 look_for_ops=1
sub _exec_gvim() {
array args
((use_tab)) args=(-p)
(( ${#gv_ops[@]:-0} )) args+=(${gv_ops[@]})
(( $# )) args+=($@)
command gvim ${args[@]}
unset -f _exec_gvim
}
AFAIK, _exec_gvim, can only be called from within function gvim, no?
Re: [Help-bash] make function local
On 4/12/15 5:56 PM, Eduardo A. Bustamante López wrote: Yes. I've been using gprof to study the code paths of some basic functions, and it seems like it spends quite some time in the find_variable() and related functions (IIRC, there was an mt_hash or something function taking up some precious time). I'm not sure if it might be better to have other kind of data structure for this. TBH, I'm not sure if there's even enough justification for this, other than to make bash startup faster. mt_hash isn't a bash function. You might be running into an internal malloc function from some libc implementation, or you might be running into the devel branch's malloc debugging code. That code has a significant effect on performance, and always has. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
On Sat, Apr 11, 2015 at 01:27:53PM -0400, Chet Ramey wrote: On 4/10/15 11:09 AM, Greg Wooledge wrote: - Fix the $... security hole (I tried and failed). http://www.gnu.org/software/gettext/manual/html_node/bash.html Yeah, I didn't like the all-or-nothing choice the patch implemented. If command substitution is the problem, a better approach would have been to inhibit command substitution instead of every word expansion. That's just not easy to do at the point where locale transformation gets done -- it requires processing the translated string to insert some kind of quoting. I'm skeptical about any substitutions being performed in a translated string. While I don't have real-life experience writing localized shell scripts, I would *think* the correct way to put variables in a translated string is: printf $Hello, %s. Welcome to %s. $LOGNAME $HOSTNAME As the script writer, I would want some guarantee that the translated string won't undergo any substitutions at all (especially not command substitutions, but even something like $1 in the translation, expanded to whatever garbage is in the positional parameters, would make the output appear wrong). But then I suppose I would also want some guarantee that the translated string won't contain any extra % or \ characters for printf to trip over. That may be outside of bash's scope. It's a messy problem.
Re: [Help-bash] make function local
On 4/13/15 8:33 AM, Greg Wooledge wrote: On Sat, Apr 11, 2015 at 01:27:53PM -0400, Chet Ramey wrote: On 4/10/15 11:09 AM, Greg Wooledge wrote: - Fix the $... security hole (I tried and failed). http://www.gnu.org/software/gettext/manual/html_node/bash.html Yeah, I didn't like the all-or-nothing choice the patch implemented. If command substitution is the problem, a better approach would have been to inhibit command substitution instead of every word expansion. That's just not easy to do at the point where locale transformation gets done -- it requires processing the translated string to insert some kind of quoting. I'm skeptical about any substitutions being performed in a translated string. While I don't have real-life experience writing localized shell scripts, I would *think* the correct way to put variables in a translated string is: printf $Hello, %s. Welcome to %s. $LOGNAME $HOSTNAME Maybe, but your original post on the subject contained this sentence: But people I've talked with said there were using $foo $bar in practice, so this definitely affects them. so the problem is real regardless of our skepticism. As the script writer, I would want some guarantee that the translated string won't undergo any substitutions at all (especially not command substitutions, but even something like $1 in the translation, expanded to whatever garbage is in the positional parameters, would make the output appear wrong). Unless you want the substitution to take place. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
Oh, you already have lots of things to do to bother with this :-) Anyways, I'll expand them. On Fri, Apr 10, 2015 at 04:35:25PM -0400, Chet Ramey wrote: On 4/10/15 10:13 AM, Eduardo A. Bustamante López wrote: - a faster implementation of the variable lookup code What does this mean, exactly? Optimizing the existing code paths? (Have at it.) Different semantics? Static as opposed to dynamic scoping? Yes. I've been using gprof to study the code paths of some basic functions, and it seems like it spends quite some time in the find_variable() and related functions (IIRC, there was an mt_hash or something function taking up some precious time). I'm not sure if it might be better to have other kind of data structure for this. TBH, I'm not sure if there's even enough justification for this, other than to make bash startup faster. - a shopt to disable evaluation of shell code in places like arithmetic expansion Remember this thread? http://lists.gnu.org/archive/html/bug-bash/2014-12/msg00158.html At one point, this was brought up: dualbus@hp ~/t % bash -c 'var=a[\$(ls)]; a=(); a[var]=x; declare -p a' bash: bar baz foo: syntax error in expression (error token is baz foo) I understand the reasons behind it. This time I don't want to debate that :-) But, wouldn't it be nice to have a `arith_expand' or something shopt that when turned off, this happened: bash +O arith_expand -c 'var=a[\$(ls)]; a=(); a[var]=x; declare -p a' declare -a a='([0]=x)' More or less how awk treats non-numeric strings as zero. -- Eduardo Bustamante https://dualbus.me/
Re: [Help-bash] make function local
On 4/10/15 11:09 AM, Greg Wooledge wrote: On Fri, Apr 10, 2015 at 09:13:17AM -0500, Eduardo A. Bustamante López wrote: Now, for the features in bash that'd be actually useful: - discipline functions and compound datatypes (like in ksh) - a way to do wrap arbitrary OS system calls with a builtin, so that instead of having thousands of builtins each interacting with the system, we could just do: `syscall lstat foo' and get something useful. (Even adding *just* examples/loadables/finfo as a standard builtin would be so helpful! But we also get a huge number of requests for readlink.) - a faster implementation of the variable lookup code - a shopt to disable evaluation of shell code in places like arithmetic expansion - a better way to deal with binary input (specially NUL bytes) - and many more, these are the ones I can think of right now Yeah, same here. I didn't know I was going to be quizzed on this, or I would have studied. - Fix the $... security hole (I tried and failed). http://www.gnu.org/software/gettext/manual/html_node/bash.html Yeah, I didn't like the all-or-nothing choice the patch implemented. If command substitution is the problem, a better approach would have been to inhibit command substitution instead of every word expansion. That's just not easy to do at the point where locale transformation gets done -- it requires processing the translated string to insert some kind of quoting. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: [Help-bash] make function local
On Thu, Apr 9, 2015 at 5:40 PM, Peng Yu pengyu...@gmail.com wrote: However, the real problem is that global variable make larger programs difficult to maintain. Therefore, the problem becomes severe for people using bash for making bash libraries. Any ad hoc use of bash can always get away from this problem. The solution is namespaces. ksh93 supports namespace blocks (including nested namespaces), though it's still quite buggy. Note C doesn't have namespaces either and gets by despite being used as a library language to a much greater extent than bash. It's worked around with naming conventions. There are many prefix conventions even within the C standard library and POSIX for various headers - atomic_*, mtx_*, pthread_*, sem_*, etc. On Fri, Apr 10, 2015 at 8:54 AM, Peng Yu pengyu...@gmail.com wrote: You don't know functional programming, do you? That example is functional program (not full fledged though) but not OO. It seems that you don't know its importance, either? With functional programming, you basically can make OO on top of it, but not the other way around. R has examples on how to make OO using functional programing. Functional programming is a style, not a language feature. Features that facilitate it are common in OO langs. C# implements first-class functions with closures by translating them in the IL to inner classes that maintain the lexical state. I'm pretty sure Java 8 does essentially the same with its lambdas. People have been using the inner class trick for ages prior to that. function as first class citizen is just one of the feature that makes javascript so successful now days. The type of function you're talking about isn't merely a local function. JavaScript doesn't exactly have local functions, and probably isn't the best example because of its numerous ways of defining functions. It also doesn't have a separate namespace for functions. No matter how you define them, you're likely using sugar that in the end results in a function object that's accessed through an identifier that's in the same namespace as any other variable (which might be local). Bash needs several major features before it could even think about supporting function objects. First it needs yet another type of function (probably with yet another syntax) which supports returning a value, and declaration as an expression so it can be passed around and treated as a value. At the same time it needs to add some sort of type system so that data other than strings (objects or functions) can be bound to variables and formal parameters. ksh93 has the beginnings of an object system which supports user-defined types, but functions are still insufficient for doing anything functional. Some of its basic ideas are nice but I have pretty much given up on understanding David Korn's design. It's obviously half-baked in its current state and hard to use for getting real work done. It's better than nothing I guess. Anyway, you can probably do something resembling FP with some combination of `typeset -T` wrappers to contain your functions, and `typeset -M` and `-C` to move and copy objects around by reference. It's not pretty. -- Dan Douglas
Re: [Help-bash] make function local
On 4/10/15 10:13 AM, Eduardo A. Bustamante López wrote: - a faster implementation of the variable lookup code What does this mean, exactly? Optimizing the existing code paths? (Have at it.) Different semantics? Static as opposed to dynamic scoping? - a shopt to disable evaluation of shell code in places like arithmetic expansion This has a chance to get implemented if you can provide enough detail about how you want it to work. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
