Re: proposed BASH_SOURCE_PATH
On Tue, May 14, 2024 at 3:35 AM Koichi Murase wrote:
>
> 2024年5月14日(火) 15:09 Martin D Kealey :
> > 1. I therefore propose that where a relative path appears in
> > BASH_SOURCE_PATH, it should be taken as relative to the directory
> > containing $0 (after resolving symlinks), rather than relative to $PWD.
> >
> > As an interim step until that's implemented, please ignore any relative
> > entries in BASH_SOURCE_PATH, so that users who really want the cwd in
> > BASH_SOURCE_PATH get used to writing $PWD or /proc/self/cwd instead.
>
> I totally agree that we should have a way to specify the files in
> subdirectories by the path relative to the current script file.
I currently use something like this to source script files relatively.
declare -g -A __sourced__files__
__sourced__files__[$BASH_SOURCE]=$(realpath "$BASH_SOURCE")
# This is to ensure to the relative path is to the actual bash file
instead of a symlink to the bash file.
source
"${__sourced__files__[$BASH_SOURCE]%/*}/dirname1/dirname2/${BASH_SOURCE##*/}"
# My convention is that the file sourced in the subdirectory has the
same filename as the parent file.
# The difference between them are expressed in the directory path,
i.e., dirname1/dirname2
My concern with adding too many features in bash is that it may have a
maintenance problem and backward compatibility issues.
Python has this problem. Python2 was good, but they introduced python3
which supposedly to make it better. But python3 introduced too many
problems. One problem is it is very difficult to import a python file
at an arbitrary location in Python3. There are too many redundant
features in Python 3, it is unclear which one is more stable then the
other. Sometimes, a feature is removed in newer version of Python
which breaks my programs that used to work. This increases maintenance
costs.
In this case, my above method works fine for many years. It sounds to
me like adding a new feature for a relative source is unnecessary.
> * One concern with treating relative paths relative to the current
> script location is that the user might put `.' in BASH_SOURCE_PATH to
> mean the current directory. I don't think it is a good practice, but I
> anticipate that users expect that. For example, one can observe the
> default values of PATH [1] and BASH_LOADABLES_PATH [2] in the Bash
> codebase. It explicitly contains `.' to mean the current working
> directory. It is very confusing that `.' in BASH_SOURCE_PATH points to
> the script location ($(dirname ${BASH_SOURCE[0]})), while `.' in PATH
> and BASH_LOADABLES_PATH points to the current working directory
> ($PWD).
>
> [1]
> https://git.savannah.gnu.org/cgit/bash.git/tree/config-top.h?h=devel&id=b3d8c8a4e7c5417d986f93f646ea740cb13c08d7#n61
> [2]
> https://git.savannah.gnu.org/cgit/bash.git/tree/config-top.h?h=devel&id=b3d8c8a4e7c5417d986f93f646ea740cb13c08d7#n78
>
> One solution might be to treat `.' as a special case to mean $PWD, and
> the other relative paths would mean the path relative to the script
> location. However, the problem is that it is confusing that the base
> location changes depending on its form. In particular, the difference
> in the behavior between `.' and `..' would be very confusing.
>
> Another solution would be to introduce a special name to mean the
> current script location. For example, we could introduce a tilde
> expansion to mean the current script location since the only expansion
> processed in `find_path_in()' and related functions is the tilde
> expansion. We already have ~+ and ~+N (equivalently, ~N) to mean
> ${DIRSTACK[N]}. Similarly, we can have e.g. ~@ and ~@N to mean
> $(dirname ${BASH_SOURCE[N]}). This can also be useful outside
> BASH_SOURCE_PATH since determining the directory name of a path isn't
> just ${path%/*} and one needs to care about edge cases to properly do
> that.
>
> * Another point to discuss is whether we should resolve the symbolic
> links. Maybe it's just fine to resolve the symbolic links fully (as
> suggested), but another possibility can be to search all the
> directories appearing in the intermediate steps of the link
> resolution.
>
> > 2. Search BASH_SOURCE_PATH when any relative path is given, not just a path
> > that lacks a '/', so that libraries can be organized into subdirectories.
>
> I agree. I actually thought about raising the discussion about this
> after the current patch set is merged.
>
> > 3. To avoid accidentally loading a script rather than a library, while
> > searching BASH_SOURCE_PATH, ignore any files that have exec permission,
> > inverting the check normally made for executables in PATH. This would keep
> > executables and libraries functionally separate, even if they're commingled
> > in the same directories.
>
> I disagree with this. Since the original intent of having
> BASH_SOURCE_PATH would be to have the list of directories that do not
> contain files not intended to be sourced, I don't see a reason to
> perform additional filtering. I generally think
Re: performance bug of [[ $x ]]
My OS is Mac OS X. I don't have perf. Is it only on linux? Could you
show me the output of your perf?
On 3/7/20, Chris Down wrote:
> Peng Yu writes:
>>Could you show me how you do the profiling for this specific case?
>>Based on what proof that you can conclude that it is not the `[[`
>>performance problem?
>
> Like I said, `perf` is perfectly adequate.
>
> bash -c 'x=$(printf "%.sx" {1..1000}); perf record -g -p $$ & sleep
> 2; time [[ $x ]]'
>
> You might as well just use `:`.
>
--
Regards,
Peng
Re: performance bug of [[ $x ]]
Could you show me how you do the profiling for this specific case? Based on what proof that you can conclude that it is not the `[[` performance problem? On 3/7/20, Chris Down wrote: > Peng Yu writes: >>[[ $x ]] just tests whether the variable $x is of length 0 or not. So >>its performance should not depend on how long the variable is. > > Who said it has anything to do with the [[ builtin's performance? A shell > does > a lot more than just running one command. > > For this and the last issue you posted, you could quite trivially find out > the > answer yourself using `perf` or another profiler. > -- Regards, Peng
performance bug of [[ $x ]]
[[ $x ]] just tests whether the variable $x is of length 0 or not. So
its performance should not depend on how long the variable is.
But the following test case shows that the run time does depend on the
length of the variable.
Should it be considered as a performance bug of bash?
$ x=$(printf '%.sx' {1..100})
$ time [[ $x ]]
real0m0.004s
user0m0.003s
sys 0m0.000s
$ x=$(printf '%.sx' {1..1000})
$ time [[ $x ]]
real0m0.043s
user0m0.039s
sys 0m0.004s
--
Regards,
Peng
Performance bug of {1..1000000}?
See the following run time comparison. {1..100} is slower than
$(seq 100).
Since seq involves an external program, I'd expect the latter to be
slower. But the comparison shows the opposite.
I guess seq did some optimization?
Can the performance of {1..100} be improved so that it is faster
than $(seq 100)?
$ time builtin printf %.sx {1..100} > /dev/null
real0m2.614s
user0m2.361s
sys 0m0.166s
$ time builtin printf %.sx $(seq 100) > /dev/null
real0m1.516s
user0m1.317s
sys 0m0.158s
--
Regards,
Peng
Use high bits of the raw random number?
Hi, Bash uses the low 16 bits for $RANDOM. https://git.savannah.gnu.org/cgit/bash.git/tree/variables.c#n1321 https://git.savannah.gnu.org/cgit/bash.git/tree/variables.c#n1356 It seems that the high bits should be more random. If so, maybe the high 16 bits should be kept if $RANDOM must stay in 16bits? https://stackoverflow.com/questions/13104478/uniformity-of-random-numbers-taken-modulo-n -- Regards, Peng
About ARITH_CMD
Hi, yylex() still gives the token ARITH_CMD for the following command. The error seems to be raised at the parsing stage. Shouldn't the error be caught in the lexical analysis stage? $ ((x = 10 + 5; ++x; echo $x)) bash: ((: x = 10 + 5; ++x: syntax error: invalid arithmetic operator (error token is "; ++x") Why the parsing of the arithmetic expression is in the lexical analysis. Why not introduce token `((` and `))` and handle arithmetic expression in the bison parsing code? Also, I don't find that POSIX specifies `((`. (Let me know if I miss anything.) If `((` is a bash-specific thing, why not allow it to handle multiple arithmetic expressions instead of just one? Thanks. http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html -- Regards, Peng
Why { is parsed differently depending on the context?
Hi,
`echo {` treats `{` as WORD.
`{ echo; }` treats `{` as a token of `{`.
`{a` treats `{a` as a WORD.
I don't see the point why yylex() treat `{` context dependently.
Wouldn't it better just treat a bare `{` as a token of `{`?
What is the reasoning behind the current design of the syntax?
--
Regards,
Peng
What is the purpose of ASSIGNMENT_WORD?
Hi,
I know that ASSIGNMENT_WORD in parse.y is for assignments like x=10.
But in the grammar rules. I don't see any difference between them in
terms of actions to take. Where is the code that deals with them
differently?
Also, why parse x=10 as a single token. Why not parse it as three
tokens "x" "=" "10"? Is it because one wants to control the complexity
of the grammar? Thanks.
https://github.com/mfragkoulis/bash/blob/master/parse.y#L710
simple_command_element: WORD
{ $$.word = $1; $$.redirect = 0; }
| ASSIGNMENT_WORD
{ $$.word = $1; $$.redirect = 0; }
https://github.com/mfragkoulis/bash/blob/master/parse.y#L3098
if (token_to_read == WORD || token_to_read == ASSIGNMENT_WORD)
{
yylval.word = word_desc_to_read;
word_desc_to_read = (WORD_DESC *)NULL;
}
https://github.com/mfragkoulis/bash/blob/master/parse.y#L5748
case WORD:
case ASSIGNMENT_WORD:
if (yylval.word)
t = savestring (yylval.word->word);
break;
https://github.com/mfragkoulis/bash/blob/master/parse.y#L6030
if (tok != WORD && tok != ASSIGNMENT_WORD)
--
Regards,
Peng
Re: What is the purpose of wdcache and wlcache?
On Fri, Feb 8, 2019 at 10:50 AM Chet Ramey wrote:
>
> On 2/8/19 10:52 AM, Peng Yu wrote:
> > On Fri, Feb 8, 2019 at 9:42 AM Chet Ramey wrote:
> >>
> >> On 2/8/19 10:39 AM, Peng Yu wrote:
> >>>> Yes: ocache_free.
> >>>
> >>> Could you please help explain what wdcache and wlcache actually do.
> >>> Why is it essential to have them? Why not just alloc and free them
> >>> without the caches? Thanks.
> >>
> >> To avoid potentially-expensive calls to malloc and free, the same as
> >> any cache.
> >
> > There are already many mallocs and frees used in other places in the
> > program, why it is essential to use cache here. Is this decision based
> > on profiling or it is just based on some abstract concept but without
> > actual runtime data?
>
> It was based on profiling at the time I implemented it.
>
> >
> > Where are these parameters coming from?
> >
> > #define WDCACHESIZE 128
>
> That's a reasonable size for a cache, and it provided a nice balance
> between memory use and speed for the most common cases.
>
> > if ((nbytes) <= 32) {
>
> That's the byte threshold for this implementation of Duff's Device. If
> the object is larger than that, it's better to use memset, with whatever
> assist the compiler can give you.
>
> > I made the following test program. I don't understand why a longer
> > word "xyzabc" can still use the cache. What is actually stored in
> > "data"?
>
> Come on. If you're not going to read the code, at least read the comments.
This is what I don't understand. Why caching pointers not caching the
actual memory allocated to the string can improve the performance?
> /* Create an object cache C of N pointers to OTYPE. */
>
> /* Free all cached items, which are pointers to OTYPE, in object cache C. */
--
Regards,
Peng
Re: What is the purpose of wdcache and wlcache?
On Fri, Feb 8, 2019 at 9:42 AM Chet Ramey wrote:
>
> On 2/8/19 10:39 AM, Peng Yu wrote:
> >> Yes: ocache_free.
> >
> > Could you please help explain what wdcache and wlcache actually do.
> > Why is it essential to have them? Why not just alloc and free them
> > without the caches? Thanks.
>
> To avoid potentially-expensive calls to malloc and free, the same as
> any cache.
There are already many mallocs and frees used in other places in the
program, why it is essential to use cache here. Is this decision based
on profiling or it is just based on some abstract concept but without
actual runtime data?
Where are these parameters coming from?
#define WDCACHESIZE 128
if ((nbytes) <= 32) {
I made the following test program. I don't understand why a longer
word "xyzabc" can still use the cache. What is actually stored in
"data"?
$ cat main.c
/* vim: set noexpandtab tabstop=2: */
/* A structure which represents a word. */
typedef struct word_desc {
char *word; /* Zero terminated string. */
int flags;/* Flags associated with this word. */
} WORD_DESC;
WORD_DESC *make_word(const char*);
void dispose_word(WORD_DESC *w);
#include
#define PTR_T char*
typedef struct objcache {
PTR_T data;
int cs; /* cache size, number of objects */
int nc; /* number of cache entries */
} sh_obj_cache_t;
extern sh_obj_cache_t wlcache;
extern sh_obj_cache_t wdcache;
void print_WORD_DESC(const WORD_DESC* w) {
printf("{ word: %s, flags: %d }\n", w->word, w->flags);
}
void print_sh_obj_cache_t(const sh_obj_cache_t cache) {
printf("{ data: %s, cs: %d, nc: %d }\n", cache.data, cache.cs, cache.nc);
}
void cmd_init(void);
int main(int argc, char **argv, char **env) {
cmd_init ();
WORD_DESC *w = make_word("abc");
print_WORD_DESC(w);
print_sh_obj_cache_t(wdcache);
dispose_word(w);
print_sh_obj_cache_t(wdcache);
WORD_DESC* w1 = make_word("abc");
print_WORD_DESC(w1);
print_sh_obj_cache_t(wdcache);
WORD_DESC* w2 = make_word("abc");
print_WORD_DESC(w2);
print_sh_obj_cache_t(wdcache);
dispose_word(w1);
print_sh_obj_cache_t(wdcache);
dispose_word(w2);
print_sh_obj_cache_t(wdcache);
WORD_DESC* w3 = make_word("xyzabc");
print_sh_obj_cache_t(wdcache);
}
$./main.exe
{ word: abc, flags: 0 }
{ data:
,
cs: 128, nc: 0 }
{ data:?s, cs: 128, nc: 1 }
{ word: abc, flags: 0 }
{ data:?s, cs: 128, nc: 0 }
{ word: abc, flags: 0 }
{ data:?s, cs: 128, nc: 0 }
{ data:?s, cs: 128, nc: 1 }
{ data:?s, cs: 128, nc: 2 }
{ data:?s, cs: 128, nc: 1 }
--
Regards,
Peng
Re: What is the purpose of wdcache and wlcache?
> Yes: ocache_free. Could you please help explain what wdcache and wlcache actually do. Why is it essential to have them? Why not just alloc and free them without the caches? Thanks. -- Regards, Peng
What is the purpose of wdcache and wlcache?
Hi,
I don't understand the purpose of wdcache and wlcache. The "nc" field
seems to be always 0 (as initialized in ocache_create(), and I don't
find where it is increased. But `ocache_alloc()` just call xmalloc
without using the cache since nc is 0. So wdcache and wlcache seem to
be useless.
Do I miss something?
http://git.savannah.gnu.org/cgit/bash.git/tree/make_cmd.c#n52
sh_obj_cache_t wdcache = {0, 0, 0};
sh_obj_cache_t wlcache = {0, 0, 0};
http://git.savannah.gnu.org/cgit/bash.git/tree/make_cmd.c#n64
void
cmd_init ()
{
ocache_create (wdcache, WORD_DESC, WDCACHESIZE);
ocache_create (wlcache, WORD_LIST, WLCACHESIZE);
}
http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n55
typedef struct objcache {
PTR_T data;
int cs; /* cache size, number of objects */
int nc; /* number of cache entries */
} sh_obj_cache_t;
http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n29
# define PTR_T char *
http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n61
/* Create an object cache C of N pointers to OTYPE. */
#define ocache_create(c, otype, n) \
do { \
(c).data = xmalloc((n) * sizeof (otype *)); \
(c).cs = (n); \
(c).nc = 0; \
} while (0)
http://git.savannah.gnu.org/cgit/bash.git/tree/include/ocache.h#n90
#define ocache_alloc(c, otype, r) \
do { \
if ((c).nc > 0) { \
(r) = (otype *)((otype **)((c).data))[--(c).nc]; \
} else \
(r) = (otype *)xmalloc (sizeof (otype)); \
} while (0)
--
Regards,
Peng
Re: What is the purpose of parser-built?
On Wed, Feb 6, 2019 at 4:49 PM Eric Blake wrote: > > On 2/6/19 4:18 PM, Peng Yu wrote: > > Hi, > > > > I deleted the file parser-built, and bash still compiles and an empty > > parser-built file will be generated upon compilation. What is the > > purpose of this file? Should it be deleted? Thanks. > parser-built is a witness that $(YACC) was run, even if the timestamps > did not change (because the generated file did not change compared to > last time). It exists in the file system so as to let make compute > timestamp dependencies where we know the parser is up-to-date, even > though the actual file we depend on has a timestamp that does NOT change > (because we intentionally don't override it when there is no > difference), all in order to minimize the time spent rebuilding the > project when making a tweak to parse.y (for example, being able to tell > the difference between a minor edit that changes a comment but not the > generated parser, vs. a major edit that requires rebuilding other files > to pick up the changes implied by the changed parser). I don't get the point. Should parser-built be deleted? If bash can be compiled without it, why put it in the source tar.gz file? -- Regards, Peng
Where is yacc_EOF defined?
Hi, yacc_EOF is mentioned in parse.y in something like this %left '&' ';' '\n' yacc_EOF | error yacc_EOF But I don't find where it is defined similarly to other tokens like BAR_AND. %token GREATER_BAR BAR_AND Where is yacc_EOF defined? (y.tab.c and y.tab.h are files generated by bison. so yacc_EOF must be defined before bison is called?) -- Regards, Peng
What is the purpose of parser-built?
Hi, I deleted the file parser-built, and bash still compiles and an empty parser-built file will be generated upon compilation. What is the purpose of this file? Should it be deleted? Thanks. -- Regards, Peng
Re: The use of register keyword in bash source code
> No, that is what volatile means. The register keyword is just an > optimisation hint, and is mostly ignored by the compiler. If it is ignored anyway, why "register" is used in many places in the code? Thanks. -- Regards, Peng
The use of register keyword in bash source code
Hi, I see many variables are declared with the "register" keyword. I know its purpose is to tell compile always access the corresponding memory without assuming the previously accessed values are preserved. This is usually to deal with some external devices. But I don't understand why it is useful in bash. Could anybody help explain? Thanks. -- Regards, Peng
Re: Where is GLOBAL_COMMAND?
> That's a documentation convention - the all-caps in the docstring calls > your attention to the need to search case-insensitively for the actual > variable, while spelling it case-sensitively would make it blend into > the sentence and make it harder to realize that the sentence is indeed > pointing you to an external variable. Why not just use ` to quote it literally? Given that markdown is very common nowadays. Quoting it with ` should be a better choice rather than making it capitalized. I guess this probably because the original codebase predates markdown and its variants? > If you think the existing convention is confusing, then submit a patch > to change it instead of asking someone else to take on the grunt work. > Otherwise, learn to live with the existing convention (which doesn't > bother me, so I won't be submitting a patch). If it was hosted on github, then I would. But I am not used to savannah. Personally, I don't think it is as convenient. -- Regards, Peng
Re: Where is GLOBAL_COMMAND?
> grep global_command *.? GLOBAL_COMMAND is uppercase. But the actual variable name global_command is in lowercase. I think that GLOBAL_COMMAND should be changed to global_command in the comment. -- Regards, Peng
Where is GLOBAL_COMMAND?
Hi, GLOBAL_COMMAND is mentioned as a global variable. But I don't find it. Is it renamed to something else? eval.c 276-/* Call the YACC-generated parser and return the status of the parse. 277- Input is read from the current input stream (bash_input). yyparse 278: leaves the parsed command in the global variable GLOBAL_COMMAND. 279- This is where PROMPT_COMMAND is executed. */ 280-int -- 310- 311-/* Read and parse a command, returning the status of the parse. The command 312: is left in the globval variable GLOBAL_COMMAND for use by reader_loop. 313- This is where the shell timeout code is executed. */ 314-int execute_cmd.c 373- 374-/* Execute the command passed in COMMAND. COMMAND is exactly what 375: read_command () places into GLOBAL_COMMAND. See "command.h" for the 376- details of the command structure. 377- -- 540- 541-/* Execute the command passed in COMMAND, perhaps doing it asynchronously. 542: COMMAND is exactly what read_command () places into GLOBAL_COMMAND. 543- ASYNCHROUNOUS, if non-zero, says to do this command in the background. 544- PIPE_IN and PIPE_OUT are file descriptors saying where input comes -- Regards, Peng
loadable cat bug?
When I use the loadable cat, I may get the following error. The input is a fifo in this specific case. cat: cannot open /tmp/tmp.VXkbqFlPtH: Interrupted system call So far, I can not make a minimal script to demonstrate the problem. But if I replace it with coreutils cat in my code, the problem is gone. Does anybody know what could cause this error? -- Regards, Peng
~60 global variables use by only one function in bash source
Hi, I see these global or static variables (1st column) used only by one function (2nd column). Some are from bash, some are from the libraries that bash depends. It seems to be problematic to declare variables global/static but only to use them in one function. Should these variables be made local? Thanks. BC_rl_init_terminal_io PC_rl_init_terminal_io UP_rl_init_terminal_io _rl_color_indicator_rl_parse_colors _rl_complete_mark_symlink_dirsset_completion_defaults _rl_completion_case_mapcomplete_fncmp _rl_echo_control_charsrl_echo_signal_char _rl_enable_bracketed_pasterl_prep_terminal _rl_enable_metareadline_internal_setup _rl_interrupt_immediatelyrl_signal_handler _rl_mark_modified_linesrl_redisplay _rl_match_hidden_filesrl_filename_completion_function _rl_menu_complete_prefix_firstrl_menu_complete _rl_optimize_typeaheadrl_insert _rl_page_completionsrl_display_match_list _rl_print_completions_horizontallyrl_display_match_list _rl_revert_all_at_newlinereadline_internal_teardown _rl_skip_completed_textinsert_match _rl_term_clrscroll_rl_init_terminal_io _rl_term_pc_rl_init_terminal_io alias_expand_allalias_expand bash_copyrightshow_manpage bash_getcwd_errstrget_working_directory bash_licenseshow_manpage build_versionmake_vers_array current_command_subst_pidcommand_substitute debugging_login_shellmain dircomplete_expand_relpathbash_directory_completion_hook dont_save_function_defsreset_shell_options from_return_trapexecute_function funmap_program_specific_entry_startrl_initialize_funmap history_multiline_entriesread_history_range history_no_expand_charshistory_expand line_number_baseassign_lineno localvar_unsetmakunbound max_input_historystifle_history optflagsinitialize_flags pretty_print_modemain progcomp_aliasprogrammable_completions release_statusmake_vers_array rl_already_promptedreadline_internal_setup rl_change_environment_rl_get_screen_size rl_complete_with_tilde_expansionrl_filename_completion_function rl_completion_word_break_hook_rl_find_completion_word rl_event_hookrl_read_key rl_executing_key_rl_dispatch_subseq rl_executing_keymap_rl_dispatch_subseq rl_inhibit_completionrl_complete rl_input_available_hook_rl_input_available rl_pre_input_hookreadline_internal_setup rl_prefer_env_winsize_rl_get_screen_size rl_visible_statsprint_filename saved_command_line_countedit_and_execute_command sh_syntabsizlocale_setblanks shell_script_filenamemain sourcenest_maxexecute_builtin terminate_immediatelytermsig_sighandler the_printed_command_sizethe_printed_command_resize this_shell_functionexecute_function tilde_expansion_failure_hooktilde_expand_word vi_insertion_keymaprl_vi_replace yylvalyyparse yynerrsyyparse -- Regards, Peng
Re: Difference of extglob between 5.0.0(1)-release and 4.4.23(1)-release
> The bash-4.4 code only worked the way you want it by chance. There was a bug that was fixed in January, 2017, the result of > http://lists.gnu.org/archive/html/bug-bash/2017-01/msg00018.html > that uncovered the behavior you're complaining about. This only explains where the change of behavior for my example comes from. I assume that this is what you meant and you did not mean to use that email to justify the change for the example that I showed. > In a filename context, or a context where a leading `.' must be matched > explicitly, a pattern must only match a filename that starts with a `.' if > `.' is the first character in the pattern. A pattern that begins with a > null extglob pattern (especially one that is defined to perform at least > one match) followed by a dot, quoted or unquoted, does not fulfill that > criterion. I don't follow your explanation. Could you please use my specific examples and break it down into steps to show what globbing pattern matching is involved in bash5? And why the logic in bash4 was wrong? > I wouldn't be so quick to declare this a bug. Other shells (e.g, ksh93, > mksh, and zsh) that implement extended globbing patterns behave like > bash-5.0 does. > > There is a question of whether or not an extglob pattern that is allowed > to make zero matches followed by a `.' should succeed, and the existing > implementations are mixed on that point. I don't think it is a good idea to introduce such kind of special cases. If @() should match an empty string, the least surprising definition is that it should match empty string everywhere. Weight the surprise that it could introduce, does the benefit of introducing special cases large enough to allow those special cases? -- Regards, Peng
Difference of extglob between 5.0.0(1)-release and 4.4.23(1)-release
See the following for the difference. I'd consider the behavior of 4.4.23 should be correct. How was this bug introduced? Should there be a test case to cover this case? $ cat main_debug.sh #!/usr/bin/env bash # vim: set noexpandtab tabstop=2: echo $BASH_VERSION declare -- null="@()" declare -- dir="." shopt -s extglob set +f matched_paths=(./$null"$dir"/main_debug.sh) declare -p matched_paths $ ./main_debug.sh 4.4.23(1)-release declare -a matched_paths=([0]="././main_debug.sh") $ ./main_debug.sh 5.0.0(1)-release declare -a matched_paths=([0]="./@()./main_debug.sh") -- Regards, Peng
Re: Identical function names in bash source code
> > > > https://pastebin.com/cV1jP41Y > > Really? What is your analysis? There are 100 duplicate global symbols > shared between bash and other libraries? Or is it your assertion that > one should never use the same symbol names, unconditionally? You're not > making much of a point here. It is the duplicated definitions among all the files in the source directory, not just the ones in the package root directory versus the ones below the root directory. The first column is the symbol, the second column is the files where a symbol is defined. You can check some examples to see if the analysis result makes sense. > > -- Regards, Peng
Re: Identical function names in bash source code
> "Not uncommon" is stretching it, since it happens in only one place: > lib/readline/shell.c. No. it is not uncommon. See the analysis of duplicated function/macro names and where they appear. There are around ~100 of them. Note that this analysis is not very accurate. But the balkpark estimate should be OK. https://pastebin.com/cV1jP41Y > Readline is now distributed separately, and used in hundreds of other > applications in addition to bash, but it was not always this way. Then, since it has been made into a separate package long time ago, why readline has not been removed from bash source code? Thanks. -- Regards, Peng
Re: [Help-bash] What are the regex spec for function names?
> There is probably no easy regex to match strings bash will tolerate as > a function name without error. The accepted names vary in several > contexts. > > http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05 > > "The function is named fname; the application shall ensure that it is > a name (see XBD Name) and that it is not the name of a special > built-in utility. An implementation may allow other characters in a > function name as an extension." > > http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_235 > > So: > > ^[_[:alpha:]][_[:alnum:]]+$ > > is close enough for most purposes. There is no easy regex to filter > the set of special builtin names. Could anybody help explain How this is done at the parsing? I am still learning the code. It seems that bash does not rely on flex to tokenize. But how to recognize function that can not be expressed in regex? -- Regards, Peng
Re: Identical function names in bash source code
> What would you say the "suggested improvement" is here? This is implied. If it is agreed that identical function names are not good by the majority of bash developers, then what I found could be turned into an explicit suggestion. Since maybe there is a good reason, I don't want to pretend that I knew why it was designed in this way. So far I don't see a good reason. So I want to confirm it first. Or this problem has never been noted before (I don't think it is very likely, given bash has been been developed for a long time). Then, it worth a discussion. To me, help-bash is more for using bash for scripting. bug-bash is more appropriate for bash development questions. -- Regards, Peng
Identical function names in bash source code
Hi,
It is not uncommon to see the same name is used to defined functions
in different .c files in bash source code.
For example, sh_single_quote is defined in both lib/readline/shell.c
and lib/sh/shquote.c with the exact same signature. The two pieces of
code are slightly different. Do they do the exact same things or do
something different?
In either case, is having the same name for different functions a good
practice? This will make the linked binary dependent on the order of
the corresponding .a files specified. Or if linked via .o files, then
one function will shadow the others. See 1) and 2) below for minimal
working examples. Neither cases seem to be good and could be avoided
easily by giving the functions unique names.
So, should such functions with the same name be named differently? Thanks.
// lib/readline/shell.c
/* Does shell-like quoting using single quotes. */
char *
sh_single_quote (char *string)
{
register int c;
char *result, *r, *s;
result = (char *)xmalloc (3 + (4 * strlen (string)));
r = result;
*r++ = '\'';
for (s = string; s && (c = *s); s++)
{
*r++ = c;
if (c == '\'')
{
*r++ = '\\'; /* insert escaped single quote */
*r++ = '\'';
*r++ = '\''; /* start new quoted string */
}
}
*r++ = '\'';
*r = '\0';
return (result);
}
// lib/sh/shquote.c
/* Return a new string which is the single-quoted version of STRING.
Used by alias and trap, among others. */
char *
sh_single_quote (string)
const char *string;
{
register int c;
char *result, *r;
const char *s;
result = (char *)xmalloc (3 + (4 * strlen (string)));
r = result;
if (string[0] == '\'' && string[1] == 0)
{
*r++ = '\\';
*r++ = '\'';
*r++ = 0;
return result;
}
*r++ = '\'';
for (s = string; s && (c = *s); s++)
{
*r++ = c;
if (c == '\'')
{
*r++ = '\\'; /* insert escaped single quote */
*r++ = '\'';
*r++ = '\''; /* start new quoted string */
}
}
*r++ = '\'';
*r = '\0';
return (result);
}
# 1 ##
==> libprint1.c <==
// vim: set noexpandtab tabstop=2:
#include
void print() {
puts("Hello World1!\n");
}
==> libprint2.c <==
// vim: set noexpandtab tabstop=2:
#include
void print() {
puts("Hello World2!\n");
}
==> main.c <==
// vim: set noexpandtab tabstop=2:
void print();
int main() {
print();
return 0;
}
==> main.sh <==
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:
set -v
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint1.a libprint1.o
ar cr libprint2.a libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint1.a libprint2.a
./main.exe
gcc -o main.exe main.o libprint2.a libprint1.a
./main.exe
$ ./main.sh
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint1.a libprint1.o
ar cr libprint2.a libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint1.a libprint2.a
./main.exe
Hello World1!
gcc -o main.exe main.o libprint2.a libprint1.a
./main.exe
Hello World2!
# 2 ##
==> libprint1.c <==
// vim: set noexpandtab tabstop=2:
#include
void print() {
puts("Hello World1!\n");
}
==> libprint2.c <==
// vim: set noexpandtab tabstop=2:
#include
void print() {
puts("Hello World2!\n");
}
==> main.c <==
// vim: set noexpandtab tabstop=2:
void print();
int main() {
print();
return 0;
}
==> main.sh <==
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:
set -v
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint.a libprint1.o libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint.a
./main.exe
ar cr libprint.a libprint2.o libprint1.o
gcc -o main.exe main.o libprint.a
./main.exe
$ ./main.sh
gcc -g -Wall -pedantic -c -o libprint1.o libprint1.c
gcc -g -Wall -pedantic -c -o libprint2.o libprint2.c
ar cr libprint.a libprint1.o libprint2.o
gcc -g -Wall -pedantic -c -o main.o main.c
gcc -o main.exe main.o libprint.a
./main.exe
Hello World1!
ar cr libprint.a libprint2.o libprint1.o
gcc -o main.exe main.o libprint.a
./main.exe
Hello World1!
--
Regards,
Peng
JIT in bash
The following test cases show that the variable length can significantly affect the runtime in bash. But the variable length doesn't seem to have a significant effect in some other interpreted languages, such as python. I can understand that variables length will slow down the runtime, as bash is interpreted. But I didn't expect that it should slow down the runtime so much. Does anybody happen to know the cause of the problem on why bash is slowed down so much compared with python when the variable length is increased? One way that I think can mitigate (but not solve the original problem) the problem is to use just-in-time (JIT) compilation. Python has a JIT compiler pypy. However, I don't see that bash has a JIT compiler. Is it because people think that bash is just a glue language so that making it run faster is not important, therefore there is no need to make a JIT compiler for bash? Or it is because the current bash code started very early and has certain limitation which makes it hard to create a new JIT compiler based on it? $ cat ./main.sh #!/usr/bin/env bash # vim: set noexpandtab tabstop=2: TIMEFORMAT=%R set -v time for ((i=0;i<1;++i)); do : done > /dev/null time for ((i=0;i<1;++i)); do : done > /dev/null time ./main.py 100 time ./main.py 1000 time ./main_long.py 100 time ./main_long.py 1000 $ cat main.py #!/usr/bin/env python # vim: set noexpandtab tabstop=2 shiftwidth=2 softtabstop=-1 fileencoding=utf-8: import sys for i in xrange(int(sys.argv[1])): pass $ ./main.sh time for ((i=0;i<1;++i)); do : done > /dev/null 0.148 time for ((i=0;i<1;++i)); do : done > /dev/null 0.321 time ./main.py 100 0.188 time ./main.py 1000 0.822 time ./main_long.py 100 0.164 time ./main_long.py 1000 0.945 -- Regards, Peng
Re: [Help-bash] How to run tests/?
> Have you tried 'make test'? No, I didn't. I didn't know it was a target. I just followed the README in that directory. -- Regards, Peng
Re: bug in dirname loadable?
On Wed, Dec 26, 2018 at 11:35 AM Chet Ramey wrote: > > On 12/24/18 10:35 PM, Peng Yu wrote: > > dirname loadable gives the following error. I think the coreutils' > > direname's convention is better. Should it be considered as a bug to > > fix? > > > > $ dirname -- -a > > dirname: usage: dirname string > > $(type -P dirname) -- -a > > . > > Yes, dirname should skip over a `--' denoting the end of options. Thanks > for the report. There is a similar problem in `basename`. Is it due to a common bug on how loadables are programmed? $ builtin basename -a -bash: basename: -a: invalid option basename: usage: basename string [suffix] $ builtin basename -- -a -- -- Regards, Peng
The usage of `cd builtins && $(MAKE) ...`
Hi, I see things like `cd builtins && $(MAKE) ...` in the Makefiles in bash source code. GNU Make has the option of -C for entering a directory and make. Is the reason to cd then make for compatibility with other make's that don't support -C? Thanks. -- Regards, Peng
Re: How to compile hashlib.c for testing?
> That code hasn't really changed in almost twenty years. All the testing > was done long ago. Do you keep all the testing code in the bash repository? Or you keep the testing code separately from the bash source? Given the fugal testing code that is in the bash source, it is doesn't seem that the current testing code can cover all the branches of the source code. I am wondering how to ensure the code contains as few bugs as possible without full coverage testing cases? Thanks. -- Regards, Peng
Re: Should [[ -v 1 ]] be supported?
> A profiler is exactly what you need here. You should profile your > script and understand the stuff that actually matters for your goals. > Otherwise you're just chasing unimportant things. Again, my goal is not to profile a specific bash script. The goal is to see what features make bash only fit into a shell language but cannot make into other domains that other languages (e.g. python) are popular at. -- Regards, Peng
Re: Should [[ -v 1 ]] be supported?
> You're whacking moles. Use a profiler. That's what they're for. I've already shown that $() is a major problem to slow down the speed and I have reduced using its usage in my code and significantly improved the performance. Nevertheless, it doesn't mean that it is not necessary to systematically evaluating features that are used frequently. > Yes, I can. You need to identify where bash is _actually_ spending most > of its execution time, and a profiler can help you do that. Yes and no. For a particular bash script, you can quantify which bash features are the most time-consuming. But you can not profile all the bash scripts that have ever been written. Since there are only limited features in bash, in this case, a logical way to go is to at least profile each commonly used feature with minimal code (as just for and repetitive calling that features as I do) and understand its pros and cons. A profiler is an overkill in this case. -- Regards, Peng
Re: How to compile hashlib.c for testing?
We are talking about unit testing in the bash C source code, not bash scripts. On Thu, Dec 27, 2018 at 8:03 PM G. Branden Robinson wrote: > > At 2018-12-27T17:34:49-0800, Eduardo Bustamante wrote: > > On Thu, Dec 27, 2018 at 5:15 PM Peng Yu wrote: > > (...) > > > Since the main() function is already there, why there is not already > > > an easy way to compile it? How do you do unit-testing then for the > > > code? > > > > This is very easy to figure out from the source code, right :)? > > > > (Hint: there is no "unit" testing in bash) > > Au contraire! :-D > > https://github.com/kward/shunit2 > > Regards, > Branden -- Regards, Peng
Re: Should [[ -v 1 ]] be supported?
On Thu, Dec 27, 2018 at 7:37 PM G. Branden Robinson wrote: > > At 2018-12-27T18:39:26-0600, Peng Yu wrote: > > What I meant in my original email is that I want something for testing > > if there is a command line argument (one or more, the exact number > > does not matter). $# gives more than that info, because it tells not > > only whether is any command line argument, but also how many. This > > could lead to slower performance if the goal is to just test if there > > is an argument. > > You should look into how integer comparisons are done in hardware. > > For instance, comparison and subtraction operations are often comparable > (or even identical) in cycle count because they both perform a > subtraction "under the hood". You need to be programming in assembly > language to influence execution at such depths. > > See, e.g., > > https://www.quora.com/What-is-the-difference-between-cmp-and-sub-instruction-in-8086microprocessor > > As others have noted, if you are worried about marginal performance > impacts this small, margin you are probably writing in the wrong > language, or distracting yourself with tiny details when you do not even > know the cyclomatic complexity of your code or the big-O classification > of your algorithms. > > Attack those problems first, and see what you discover. The problem is that bash is not systematically profiled for performance at all. I am doing it step it by step. There are many more that I have already tested. You can not dismiss this one just because it may not have a large impact. -- Regards, Peng
Re: Should [[ -v 1 ]] be supported?
> I don't believe that at all. The number of positional parameters is kept > anyway. It's not recalculated when you compare it to another number, so > it's just as fast as a simple comparison of two integers. Getting the number $# is slow. > And even if it weren't -- if performance is *that* important to you, > you're using the wrong language altogether. There is no reason why bash has to be slow. Javascript was slow. But it has been optimized to be much faster than it was before. What I see bash is that it is syntax is much better than other languages in terms of interactive use and not use () for arguments? It could be made faster and with some additional features, it could be used for other more serious applications rather than just a glue of other tools. -- Regards, Peng
Re: How to compile hashlib.c for testing?
On Thu, Dec 27, 2018 at 12:27 PM Chet Ramey wrote: > > On 12/26/18 4:31 PM, Peng Yu wrote: > > Hi, > > > > I'd like to compile hashlib.c to try its main(). But I got the > > following error. What is the correct commands to compile it? Thanks. > > Think about the error message: > > > "_xmalloc", referenced from: > > _hash_create in hashlib.o > > _hash_copy in hashlib.o > > _hash_search in hashlib.o > > _hash_insert in hashlib.o > > _main in hashlib.o > > ld: symbol(s) not found for architecture x86_64 > > So you need a definition for xmalloc. The easiest thing to do is to add > an xmalloc function in the TEST_HASHING block, or you could follow the > directions in the comment there. > > Sooner or later, you're going to have to pick this stuff up on your own. I run make in lib/malloc, then I got the following error. What is wrong? ~/Downloads/bash-4.4$ gcc hashlib.o xmalloc.o lib/malloc/libmalloc.a -o hashlib.exe Undefined symbols for architecture x86_64: "_libintl_gettext", referenced from: _xmalloc in xmalloc.o _xrealloc in xmalloc.o _internal_malloc in libmalloc.a(malloc.o) _internal_realloc in libmalloc.a(malloc.o) _internal_free in libmalloc.a(malloc.o) _xbotch in libmalloc.a(malloc.o) "_running_trap", referenced from: _internal_malloc in libmalloc.a(malloc.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) Then I add ./lib/intl/libintl.a, I got additional errors. This does not seem to be an efficient way to resolve the problem. Is there an easy way to figure out all the necessary libraries (but no more) to link. $ gcc hashlib.o xmalloc.o ./lib/intl/libintl.a lib/malloc/libmalloc.a -o hashlib.exe Undefined symbols for architecture x86_64: "_iconv", referenced from: __nl_find_msg in libintl.a(dcigettext.o) "_iconv_close", referenced from: __nl_free_domain_conv in libintl.a(loadmsgcat.o) "_iconv_open", referenced from: __nl_init_domain_conv in libintl.a(loadmsgcat.o) "_libiconv_set_relocation_prefix", referenced from: _libintl_set_relocation_prefix in libintl.a(relocatable.o) "_running_trap", referenced from: _internal_malloc in libmalloc.a(malloc.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) Since the main() function is already there, why there is not already an easy way to compile it? How do you do unit-testing then for the code? -- Regards, Peng
Re: Should [[ -v 1 ]] be supported?
On Thu, Dec 27, 2018 at 3:19 PM Martijn Dekker wrote:
>
> Op 27-12-18 om 19:22 schreef Chet Ramey:
> > On 12/26/18 10:49 PM, Peng Yu wrote:
> >
> >> Although [[ -z ${1+s} ]] and (($#)) works for testing if $1 is set,
> >> neither of them are uniformly better performance wise. In this case,
> >> should [[ -v 1 ]] be supported?
> >
> > So you're saying that neither of the existing options performs better
> > than the other, though they both perform well, so we should add some
> > new capability just because? That's a particularly poor argument.
>
> Consistency might be a better argument. If [[ -v foo ]] is equivalent to
> [[ -n ${foo+s} ]] for variables (with the advantage that you don't need
> 'eval' to handle arbitrary values of 'foo'), then perhaps it's not
> unreasonable to expect [[ -v 1 ]] to be equivalent to [[ -n ${1+s} ]].
>
> FWIW, zsh and mksh do support this; ksh93 doesn't.
The above are additions arguments that why [[ -v 1 ]] should be supported.
What I meant in my original email is that I want something for testing
if there is a command line argument (one or more, the exact number
does not matter). $# gives more than that info, because it tells not
only whether is any command line argument, but also how many. This
could lead to slower performance if the goal is to just test if there
is an argument.
[[ -z ${1+s} ]] does something also more than necessary too, because
it not only tests for whether $1 is set, it also replaced with a
string "s". This also does more than just testing whether $1 is set.
So both cases would be slower than [[ -v 1 ]] if it were supported.
As of now, because (($#)) or [[ -z ${1+s} ]] are not consistently
faster than the other, there is no way to write a program that is
consistently fastest. To achieve this goal, one has to implement [[ -v
1 ]] or something similar that just test whether $1 but no more.
--
Regards,
Peng
Should [[ -v 1 ]] be supported?
Hi,
[[ -v 1 ]] does not work for $1.
$ [ -v 1 ]; echo "$?"
1
$ set -- a
$ [ -v 1 ]; echo "$?"
1
Although [[ -z ${1+s} ]] and (($#)) works for testing if $1 is set,
neither of them are uniformly better performance wise. In this case,
should [[ -v 1 ]] be supported?
set -- $(seq 1)
time for ((i=0;i<1;++i)); do
(($#))
done
real0m0.140s
user0m0.132s
sys0m0.001s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done
real0m0.216s
user0m0.151s
sys0m0.000s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done
real0m0.336s
user0m0.156s
sys0m0.009s
set -- $(seq 10)
time for ((i=0;i<1;++i)); do
(($#))
done
real0m0.224s
user0m0.130s
sys0m0.001s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done
real0m0.256s
user0m0.147s
sys0m0.000s
set -- $(seq 100)
time for ((i=0;i<1;++i)); do
(($#))
done
real0m0.260s
user0m0.131s
sys0m0.001s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done
real0m0.244s
user0m0.146s
sys0m0.001s
set -- $(seq 1000)
time for ((i=0;i<1;++i)); do
(($#))
done
real0m0.338s
user0m0.167s
sys0m0.002s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done
real0m0.277s
user0m0.144s
sys0m0.000s
set -- $(seq 1)
time for ((i=0;i<1;++i)); do
(($#))
done
real0m1.012s
user0m0.540s
sys0m0.011s
time for ((i=0;i<1;++i)); do
[[ -z ${1+s} ]]
done
real0m0.292s
user0m0.148s
sys0m0.003s
--
Regards,
Peng
How to compile hashlib.c for testing?
Hi, I'd like to compile hashlib.c to try its main(). But I got the following error. What is the correct commands to compile it? Thanks. $ gcc -DPROGRAM='"bash"' -DCONF_HOSTTYPE='"x86_64"' -DCONF_OSTYPE='"darwin17.7.0"' -DCONF_MACHTYPE='"x86_64-apple-darwin17.7.0"' -DCONF_VENDOR='"apple"' -DLOCALEDIR='"/usr/local/share/locale"' -DPACKAGE='"bash"' -DSHELL -DTEST_HASHING -DHAVE_CONFIG_H -DMACOSX -I. -I. -I./include -I./lib -I./lib/intl -I/Users/pengy/Downloads/bash-4.4/lib/intl -g -O2 -Wno-parentheses -Wno-format-security -c hashlib.c $ gcc -c hashlib.o hashlib.exe Undefined symbols for architecture x86_64: "_xmalloc", referenced from: _hash_create in hashlib.o _hash_copy in hashlib.o _hash_search in hashlib.o _hash_insert in hashlib.o _main in hashlib.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) -- Regards, Peng
difference between /tmp and other directory for loadable mkdir?
Hi, I can not mkdir -p . in /tmp/ via the loadable mkdir. What is the difference between /tmp/ and other directories? I am on Mac OS X. Is this a bug in mkdir? $ cd /tmp $ mkdir -p -- . -bash: mkdir: .: Operation not permitted $ cd ~/ $ mkdir -p -- . -- Regards, Peng
Re: Why does ctrl-c behave differently for builtin sleep in a subshell?
If I add a command after the builtin sleep, then the EXIT trap will be
triggered upon typing ctrl-C.
If the last command is removed, then the EXIT trap will not be
triggered upon typing ctrl-C.
Is this a bug in the built-in sleep? I am trying to inspect the C code
of sleep builtin, but it is very simple. I am not sure where the
problem comes from.
Could anybody take a look? Thanks.
$ cat main_INT.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:
source ../../../trapexit.sh
(
trapexit -q <<-'EOF'
echo Hello World!
EOF
source enable.sh
enable sleep
sleep 10
echo "$?"
)
$ ./main_INT.sh
^CHello World!
$ cat main_INT.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:
source ../../../trapexit.sh
(
trapexit -q <<-'EOF'
echo Hello World!
EOF
source enable.sh
enable sleep
sleep 10
#echo "$?"
)
$ ./main_INT.sh
^C
$ source enable.sh
$ enable sleep
$ help sleep
sleep: sleep seconds[.fraction]
Suspend execution for specified period.
sleep suspends execution for a minimum of SECONDS[.FRACTION] seconds.
On Mon, Dec 17, 2018 at 1:57 PM Peng Yu wrote:
>
> Note that when SIGINT is set, whether an external or built-in sleep is
> used will not result in any difference.
>
> If I use some other method (say a for loop) instead of built-in sleep
> to introduce a delay, the EXIT can be reached upon ctrl-c. So this
> sounds like that the builtin sleep somehow causes the problem. Is it
> the case?
>
> $ cat main_builtin_INT.sh
> #!/usr/bin/env bash
> trap 'echo at EXIT' EXIT
> trap 'echo at SIGINT' SIGINT
> enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> sleep 10
> $ cat \(/main_builtin_INT.sh
> #!/usr/bin/env bash
> (
> trap 'echo at EXIT' EXIT
> enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> sleep 10
> )
> $ ./main_builtin_INT.sh
> ^Cat SIGINT
> at EXIT
> $ '('/main_builtin_INT.sh
> ^Cat SIGINT
> at EXIT
>
> $ cat ./main_builtin_for.sh
> #!/usr/bin/env bash
> (
> trap 'echo at EXIT' EXIT
> for ((i=0;i<10;++i)); do
> :
> done
> )
> $ ./main_builtin_for.sh
> ^Cat EXIT
>
> On Mon, Dec 17, 2018 at 1:41 PM Peng Yu wrote:
> >
> > Hi,
> >
> > See the follow code, '(/main_builtin.sh' behaves differently from the
> > other three scripts upon receiving ctrl-c. This is counter-intuitive.
> > I'd expect whether a built-in command called the result should be the
> > same.
> >
> > How to understand why there is such a difference? Is there an easy
> > workaround to make the result the same as the other three?
> >
> > $ ./main.sh
> > ^Cat EXIT
> >
> > $ ./main_builtin.sh
> > ^Cat EXIT
> >
> > $ \(/main.sh
> > ^Cat EXIT
> >
> > $ \(/main_builtin.sh
> > ^C
> >
> > $ cat main.sh
> > #!/usr/bin/env bash
> > trap 'echo at EXIT' EXIT
> > sleep 10
> >
> > $ cat main_builtin.sh
> > #!/usr/bin/env bash
> > trap 'echo at EXIT' EXIT
> > enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> > sleep 10
> >
> > $ cat '(/main.sh'
> > #!/usr/bin/env bash
> > (
> > trap 'echo at EXIT' EXIT
> > sleep 10
> > )
> >
> > $ cat '(/main_builtin.sh'
> > #!/usr/bin/env bash
> > (
> > trap 'echo at EXIT' EXIT
> > enable -f ~/Downloads/bash-4.4/examples/loadables/sleep sleep
> > sleep 10
> > )
> >
> > --
> > Regards,
> > Peng
>
>
>
> --
> Regards,
> Peng
--
Regards,
Peng
bug in dirname loadable?
dirname loadable gives the following error. I think the coreutils' direname's convention is better. Should it be considered as a bug to fix? $ dirname -- -a dirname: usage: dirname string $(type -P dirname) -- -a . -- Regards, Peng
Re: general loadable integration
> I suppose the first thing needed to make that work, and maybe the only > thing needed to make that work, is agreement on the name of a search path > environment variable that enable can use to find loadable builtins. Why not just use an environment variable such as LOADABLES_PATH (just like the way how PATH is used)? Is it too difficult to reach an agreement on this? As of 4.4.23, I still don't see such an environment variable. Anyway, any variable is better than no variable. -- Regards, Peng
Re: Why does bash use xmalloc?
The artificial ulimit is to tigger the error. My point is why bash terminates when it runs an external command that requires a large memory. Shouldn't bash return an exit code on behalf of the failed command and continue to the next command? On Sunday, November 6, 2016, Eduardo Bustamante wrote: > Hi Peng. Read the link you provided again. xmalloc is not an > alternative version of malloc. It's just a common wrapper function > around malloc. You can go and see for yourself, the definition is > here: http://git.savannah.gnu.org/cgit/bash.git/tree/xmalloc.c#n97 > > If you want the rest of the commands execute properly, then make sure > that there's enough memory in the system, and remove artificial > restrictions (ulimit -v?). There's no trick here, and certainly no bug > in bash. > -- Regards, Peng
Why does bash use xmalloc?
Hi, The following example shows that bash uses xmalloc. But it seems
that using xmalloc is not a good practice. Is it better to use malloc
instead of xmalloc? In this test case, after `./main 100` failed I
still want to run the rest commands. So it sounds like malloc is
better.
http://stackoverflow.com/questions/7590254/what-is-the-difference-between-xmalloc-and-malloc
~$ cat main.sh
#!/usr/bin/env bash
# vim: set noexpandtab tabstop=2:
ulimit -Sv 1000
set -v
./main 10
./main 100
./main 1000
./main 1
~$ cat main.c
// vim: set noexpandtab tabstop=2:
#include
#include
int main(int argc, char *argv[]) {
//printf("%s\n", "Hello World!");
size_t size = atoi(argv[1]);
void *ptr;
if((ptr = malloc(size)) == NULL) {
perror("malloc");
exit(2);
} else {
printf("%p\n", ptr);
free(ptr);
}
return 0;
}
~$ ./main.sh
./main 10
./main.sh: xmalloc: .././variables.c:3997: cannot allocate 1313 bytes
(53248 bytes allocated)
--
Regards,
Peng
Re: How bash do tokenization?
OK. I see it, which check emails and print prompt. What factors people need to consider to decide whether to use flex to perform tokenization and write a customize tokenizer? Checking emails and printing prompt strictly speaking is not related with tokenization. Is there an alternative way to organize the code so that it will be more decoupled? On Tue, Oct 18, 2016 at 10:22 PM, Eduardo Bustamante wrote: > On Tue, Oct 18, 2016 at 10:18 PM, Eduardo Bustamante > wrote: >> Check parser.y > Sorry, I meant parse.y, inside it you will find read_token and yylex. -- Regards, Peng
How bash do tokenization?
Hi, I have been checking bash source code. But it is not clear to me how bash do tokenization as I don't find a lex file. Could anybody point to me where I should look for the information about tokenization in bash source code? -- Regards, Peng
Re: [Help-bash] make function local
Hi Chet, >>> That's the difference: if you're careful with naming and rigorous about >>> your calling conventions, your one-time-use functions are about as close >>> as you can get to local functions in bash, but you have to pay attention >>> to the declaration's side effects. >> >> There is at least a runtime overhead for having too many unused >> functions that are supposed to be "local" or in your word "lambda". > > Maybe so, but this is not the discussion we were having. > >> Despite that one can rename supposedly internal functions to names >> that are unlikely to cause name collision via good naming convention, >> it still can incur a significant performance overhead. > > I disagree that performance overhead in typical use is `significant'. > This point is more or less identical to the one I discussed Friday in > regards to creating huge numbers of variables. Who creates 100,000 > shell functions in a single script? The overhead, such as it is, of > creating and calling even 10,000 functions is negligible. Why do you assume that one only calls each of these functions once? What if a function is called thousands of time? The performance difference is 10x different (0.490s vs 0.040s). To me, this is a huge performance problem. > (And the example you chose to illustrate this is not what Linda is > talking about or using.) >> In this sense, I think that it is still necessary to consider make the >> supposedly internal function "local" so that they would not slow down >> function search in the global namespace. > > Look, you can make the same argument about function creation at any > scope, since there is one function namespace. Functions are usually called for more times they are defined. Therefore, even if the function creation time is acceptable, it doesn't mean its call time is acceptable. > Acceptable performance is > subjective: if the technique that Linda uses for data encapsulation > results in performance that's acceptable for her application, then it's > ok for her to use it. You are certainly free to use any methodology you > find comfortable and satisfies your constraints. The main point that Linda and I are making is that we suggest add the local function feature. I believe that we've made the case --- it makes the code cleaner and run faster. What makes you reluctant to consider this feature be added? Is it takes too much time to implement such a feature? -- Regards, Peng
Re: [Help-bash] make function local
Hi Chet,
>> Eduardo A. Bustamante López wrote:
>>> Well, if your scripts are so simple, why use local functions at all?
>> ---
>> Cleanliness, Hygiene...
>
> Please, let's not have this argument again. I think you're all using the
> term `local function' to mean different things.
>
> You seem to be using the term to describe one-time-use functions (they're
> almost lambdas, but they have names). If you define a function within
> another function's body, have it unset itself when it executes, and call
> it only from within the function where it's defined, you have something
> very close to one-time-use functions with function-only scope.
>
> However you use them, they share the name namespace as every other defined
> function, regardless of whether or not they are defined as part of the body
> of another function. If you had a function defined in the global scope
> with the same name as your one-time-use function, it would be removed when
> the one-time-use function was declared. I think this is what Greg and
> Eduardo mean, and in this sense they are correct: bash doesn't have local
> functions with separate namespaces from other defined functions.
>
> That's the difference: if you're careful with naming and rigorous about
> your calling conventions, your one-time-use functions are about as close
> as you can get to local functions in bash, but you have to pay attention
> to the declaration's side effects.
There is at least a runtime overhead for having too many unused
functions that are supposed to be "local" or in your word "lambda".
Despite that one can rename supposedly internal functions to names
that are unlikely to cause name collision via good naming convention,
it still can incur a significant performance overhead.
In this sense, I think that it is still necessary to consider make the
supposedly internal function "local" so that they would not slow down
function search in the global namespace.
~$ cat main_many.sh
#!/usr/bin/env bash
n=$1
for i in $(seq -w $n)
do
eval "function f$i { echo '$i'; }"
done
function g {
for i in $(seq -w $n | head -n 1000)
do
f$i > /dev/null
done
}
time g
~$ ./main_many.sh 1000
real 0m0.032s
user 0m0.020s
sys 0m0.010s
~$ ./main_many.sh 1
real 0m0.040s
user 0m0.029s
sys 0m0.011s
~$ ./main_many.sh 10
real 0m0.490s
user 0m0.461s
sys 0m0.031s
--
Regards,
Peng
Re: How to deal with errors in <()?
On Sat, Mar 14, 2015 at 8:46 AM, Linda Walsh wrote: > > > Peng Yu wrote: >> >> Hi, >> >> http://mywiki.wooledge.org/ProcessSubstitution >> >> The above webpage says the following. >> >> commandA <(commandB; [commandB's exit code is available here from $?]) >> [commandB's exit code cannot be obtained from here. $? holds >> commandA's exit code] > > >> >> Does anybody have a good solution for this situation? Thanks. > > -- > It's not a pretty solution, but how about some variation of: > >> alias cmda='cat >/dev/tty'## 1st cmd >> alias cmdb='(echo "foo";exit 2)' ## command executes, but exits w/err=2 >> read status < <(cmda <(cmdb; echo "status=$?" >&2) 2>&1) Stdout and stderr are commonly used for other purposes. But using fd 3 and above seems to be a good walkaround. Does anybody have a cleaner solution? > status=2 > foo > --- > not ideal, but it does get you the status. > Instead of echoing it you could assigned it to a var, > and test the value and only print status if non-zero, > something like: > > ((status=$?)) && echo "status=$status > > instead of just the echo > > -- Regards, Peng
Re: -i option of set missing in man bash
On Friday, March 13, 2015, Chet Ramey wrote: > On 3/12/15 2:13 PM, Peng Yu wrote: > >>> The -i option obviously works with set. But it is missing in the man > >>> page. Should this be added? > >> > >> No. It's really only there for completeness, so things like `set $-' > >> work as expected without error. > > > > But if something is in the implementation, it should be also in the > > documentation, right? > > There is a level of absurdity about this. I can put a sentence in saying > that the -i, -c, and -s options are only effective at invocation and no-ops > when used with `set' -- if you've ever wondered how you get an 80-page man > `page', that's how -- but would that really have made a difference? It's > only there for convenience, after all, and has no effect. You can avoid these surprises by making -i just as other options, i.e., working within the shell not just when a shell is started. The document should be complete. Why it matters you have a 8, 80, or 800 manpages? Few people prints it. After all, most people just search it and it is better to make it complete. I believe the best way to make the manpage short is to make the implementation intuitive so that people can derive the behavior in complex cases based on what are described in simpler cases in the manual. With this in mind, one avoids the description of complexes cases in the manual. >> -i is described in the OPTIONS section of the man page, since invocation > >> is the only place using it makes sense. > > > > Not necessarily. See my other example sent recently on the mailing > > list of trying to get COLUMNS. > > With regards to COLUMNS, one doesn't need interactive mode to do that. > There are lots of ways to set COLUMNS to the screen width. It's much more > productive to state your requirements clearly so people can suggest > solutions -- people here are more than happy to help -- than to complain > that the shell isn't satisfying your assumptions. > > Chet > -- > ``The lyf so short, the craft so long to lerne.'' - Chaucer > ``Ars longa, vita brevis'' - Hippocrates > Chet Ramey, ITS, CWRUc...@case.edu > http://cnswww.cns.cwru.edu/~chet/ > -- Regards, Peng
Re: -i option of set missing in man bash
On Thu, Mar 12, 2015 at 1:29 PM, Greg Wooledge wrote: > On Thu, Mar 12, 2015 at 01:13:18PM -0500, Peng Yu wrote: >> One may want to manually set -i option in a bash script for whatever >> reason. (In this case, it is to check COLUMNS.) > > http://mywiki.wooledge.org/BashFAQ/091 This does not work if one need to use #!/usr/bin/env bash -- Regards, Peng
Re: -i option of set missing in man bash
>> The -i option obviously works with set. But it is missing in the man >> page. Should this be added? > > No. It's really only there for completeness, so things like `set $-' > work as expected without error. But if something is in the implementation, it should be also in the documentation, right? > The -s option does the same thing, > for example. > > -i is described in the OPTIONS section of the man page, since invocation > is the only place using it makes sense. Not necessarily. See my other example sent recently on the mailing list of trying to get COLUMNS. One may want to manually set -i option in a bash script for whatever reason. (In this case, it is to check COLUMNS.) -- Regards, Peng
-i option of set missing in man bash
Hi, The -i option obviously works with set. But it is missing in the man page. Should this be added? ~$ echo $- himBH ~$ set +i ~$ echo $- hmBH The following lines are from the man page. set [--abefhkmnptuvxBCEHPT] [-o option-name] [arg ...] set [+abefhkmnptuvxBCEHPT] [+o option-name] [arg ...] -- Regards, Peng
Re: How to deal with errors in <()?
On Mon, Mar 9, 2015 at 2:07 PM, Chet Ramey wrote: > On 3/8/15 6:05 PM, Stephane Chazelas wrote: > >> Are bash questions no longer on topic here? bash-bug used to be >> the place to discuss bash (before help-bash was created). It maps to the >> gnu.bash.bug newsgroup. I don't think help-bash maps to usenet >> (though you can access it over NNTP on gmane). > > I personally don't care which mailing list gets the questions. > >> $ bash --norc >> bash-4.3$ echo <(exit 123) >> /dev/fd/63 >> bash-4.3$ echo "$!" >> 12142 >> bash-4.3$ wait "$!" >> bash: wait: pid 12142 is not a child of this shell >> >> Having the process substitution pid in $! is not very useful if >> you can't wait for it to retrieve the status. > > I agree. I will look at making that work for the next release of bash. Would making an environment array (just as PIPE_STATUS) that will be expanded to the process IDs of the all executed background commands started from the last command be helpful to my original question? (But this should also work with commands connected by pipes in order to solve the problem.) -- Regards, Peng
How to deal with errors in <()?
Hi, http://mywiki.wooledge.org/ProcessSubstitution The above webpage says the following. commandA <(commandB; [commandB's exit code is available here from $?]) [commandB's exit code cannot be obtained from here. $? holds commandA's exit code] But I am wondering if there is a walkaround to deal with errors in <(). The ideal behavior should be that if there is a error in <(), then we should not consider commandA is executed correctly even if its return status is 0. Does anybody have a good solution for this situation? Thanks. -- Regards, Peng
Where is the usage "for x in a b c; { echo $x; }" documented in bash man page?
Hi,
The following code works in bash.
for x in a b c; { echo $x; }
But I only find the following in bash man page. Does anybody know
where the above usage is documented? Thanks.
"for name [ [ in [ word ... ] ] ; ] do list ; done"
--
Regards,
Peng
The correct way to use "for" without polluting the environment
Hi, I use unset to remove x from the environment once the for loop is finished. Is it the best way to do in bash? Thanks. for x in a b c do echo "$x" done unset x -- Regards, Peng
"echo" does not follow the "getopt" convention
Hi, The following shows that echo does not following the "getopt" convection. Is it better to make all bash internal command following the getopt convention? ~$ echo -n ~$ echo -n -n ~$ echo -n -- -n # I think that the output should just be "-n" not "-- -n". -- -n~$ http://software.frodo.looijaard.name/getopt/ -- Regards, Peng Yu, Ph.D. Assistant Professor Dept. Electrical and Computer Engineering & TEES-AgriLife Center for Bioinformatics and Genomic Systems Engineering (CBGSE) Texas A&M University Office: 215F WEB Phone: (979) 320-9822 Email: pe...@tamu.edu http://yubiolab.wordpress.com
cd won't change the prompt if the command is typed in the vi editor (vi mode).
Hi, I have vi mode set. set -o vi. Then I type v to enter the vi editor. In the vi editor, I type 'cd /tmp'. Then, I get the following screen output. Notice that the prompt does not change immediately after the cd command. Could anybody reproduce this behavior? Thanks. ~$ cd /tmp ~$ echo $PWD /tmp /tmp$ -- Regards, Peng
Re: Specify completion without name
> The bash man page already has ~70 pages manual. I don't like it to grow to > ~700 pages (like the ABS Guide) with all the working examples you expected. > :) Do you use search at all? :) If you use search, it doesn't really matter if is a 700 page manual. -- Regards, Peng
Re: Specify completion without name
> I would envision that such a completion function would assemble its list > of possible completions by using your read-from-a-file mechanism and > augment the list using compgen -a/compgen -b/compgen -A function. It > would probably also want to handle glob patterns and expand them to > potentially multiple completions, but that gets tricky. I did not know that it is so simple to get the alias (compgen -a), buildins (compgen -b) and functions (compgen -A function) as you mentioned. Once I know these, I agree with you that bash need not handle these internally, rather user can call these three functions directly. But beware to clearly document these by giving working EXAMPLE code which include these three commands (not just text explanation without working code, by "working code" I mean code snippet is discouraged, a complete completion function should be provided). BTW, as I mentioned several times the bash man favors document maintainer rather readers. For example, the following help doesn't help me much when I want to learn how to use compgen. ~$ help compgen compgen: compgen [-abcdefgjksuv] [-o option] [-A action] [-G globpat] [-W wordlist] [-P prefix] [-S suffix] [-X filterpat] [-F function] [-C command] [word] Display the possible completions depending on the options. Intended to be used from within a shell function generating possible completions. If the optional WORD argument is supplied, matches against WORD are generated. The manpage also use a reference rather than list all the options directly. Readers have to jump to complete to understand how to use compgen. This is also inconvenient to users. Generate possible completion matches for word according to the options, which may be any option accepted by the complete builtin If you consider it repetitive to discuss the same option twice in both compgen and complete, at least, you can expand "help compgen" to describe all the options (merge the current description of compgen and complete in man). Other help messages are so concise that they are not very helpful for learning how to use them. I'd suggest change all of them as well. -- Regards, Peng
Re: Specify completion without name
> Presumably you would also include aliases, shell builtins, and functions > in this file. Yes. I just want to replace executables in PATH by the result from my custom function. I think that aliases, builtins, and functions are all in the memory of bash already, so it doesn't take extra time to search them from bash. Therefore, the ideal behaviors is that bash still search for aliases, builtins and functions, and in addition check the results returned by my custom function. Hence, my custom function should not return anything about them (this should be expect at least by default). -- Regards, Peng
Re: Specify completion without name
> empty lines. There is no programmable completion mechanism to complete > on non-empty command names. I'm wondering if it is worthwhile to add such a feature. I have run into the problem that it is very slow to command complete an incomplete command especially when other programs are accessing the disk (note that I frequently open new terminals, so the caching done within a bash process does not help me much). If I can configure how to complete on non-empty command names, I could just check a file with all the commands in PATH are stored. By checking just a single file, presumable the search can be much faster than searching many directories. -- Regards, Peng -- Regards, Peng
Specify completion without name
Hi, I want to customize the command completion for completing executables, I want to search in a file (which includes all the executables in PATH) rather than the default PATH variable. But I don't see how to do so, as the following help indicates that it can only configure how to complete the arguments of a command. Does anybody know how to configure command completion for the name itself rather than its arguments? Thanks! ~/Downloads$ help complete complete: complete [-abcdefgjksuv] [-pr] [-o option] [-A action] [-G globpat] [-W wordlist] [-P prefix] [-S suffix] [-X filterpat] [-F function] [-C command] [name ...] For each NAME, specify how arguments are to be completed. If the -p option is supplied, or if no options are supplied, existing completion specifications are printed in a way that allows them to be reused as input. The -r option removes a completion specification for each NAME, or, if no NAMEs are supplied, all completion specifications. -- Regards, Peng
Re: Is the description of set -- missing in man bash or at least difficult to find?
> There are shortcomings in _the man documentation format_ and one of them > is that it doesn't work (at least for me...) when the documentation is > longer than one screen or thereabouts. I've pretty much come to the > conclusion that any man page that is over a couple of hundred lines is > a waste of my time and should probably not even exist in the first > place. Hi CJ, You didn't get my point. Please see my reply to DJ Mills, if my first email is not clear enough to you. The shortcomings is not in the man format. Even if it is in the man format, it can be compensated in the way that I mentioned. -- Regards, Peng
Re: Is the description of set -- missing in man bash or at least difficult to find?
> +1 vote on getting the parameters listed with a leading dollar sign. > The individual single character is difficult to search for but the > combination of "$@" and so forth for the others is a useful search > string. I have often wanted the manual to include the "$@" > combination instead of just the "@" name. I agree $@ is better than @. But not to repeat myself again, did you get my major point that I have restated to DJ Mills? -- Regards, Peng
Re: Is the description of set -- missing in man bash or at least difficult to find?
> Second, just search for the 'set' builtin, near the bottom of the man page. Thank for clarifying the usage of set. I looked closely to the document of set. I just find another problem, it says the following. However, the description of -- way down below. It should be the option be described. A rule of thumb is that the order of the description of the options should be the same as the order in the following line. set [--abefhkmnptuvxBCEHPT] [-o option-name] [arg ...] But my main point is if a person only knows the keyword $@, he will not be able to find how to set it in the manual. The description of 'set --' should also be added to the description of $@ to explain how to set $@. Also, "If no arguments follow this option, then the positional parameters are unset. Otherwise, the positional parameters are set to the args, even if some of them begin with a -." should be rewritten as "If no arguments follow this option, then the positional parameters $@ are unset. Otherwise, the positional parameters $@ are set to the args, even if some of them begin with a -." so that $@ is searchable. -- Regards, Peng
Is the description of set -- missing in man bash or at least difficult to find?
Hi,
As I mentioned previously, there are shortcomings in man bash. Here, I
just point another example. And I hope my suggestion will be
addressed.
As a reasonable search strategy to search for how to set $@ is to
search for '$@' in man bash. The literal word '$@' appears at the
following locations.
...performed, with the exception of "$@" as explained below under Special...
...expands to a separate word. That is, "$@" is equivalent to "$1"...
...When there are no positional parameters, "$@" and $@ expand to...
...of "$@" and "${name[@]}" as explained above (see PARAMETERS)...
Search for 'set --' returns nothing. And search for '--' is prohibited
as there are too many of them. If the manual discusses how to 'set
$@', then it is at least hard to find. My suggestion is to add a new
paragraph to the existing discussion of $@, so that by searching $@,
readers can easily see how set $@.
@ Expands to the positional parameters, starting from one. When
the expansion occurs within double quotes, each parameter
expands to a separate word. That is, "$@" is equivalent to "$1"
"$2" ... If the double-quoted expansion occurs within a word,
the expansion of the first parameter is joined with the begin-
ning part of the original word, and the expansion of the last
parameter is joined with the last part of the original word.
When there are no positional parameters, "$@" and $@ expand to
nothing (i.e., they are removed).
As others pointed out LDP/abs is a more readable document, the
following link is a much better document on helping me to find how to
set "$@".
http://tldp.org/LDP/abs/html/internalvariables.html
--
Regards,
Peng
Re: [Help-bash] How to keep only files match the prefix when do command completion?
Hi Greg, > New users do not mess with programmable completion. Given the context, I though that it was clear that "new users" means users new to command completion. If it was not clear, I make it explicit here. -- Regards, Peng
Ill positioned 'until' keyword
Hi, I looks a little wired why 'until' is the way it is now. According to the manual until is before the do-done block. until test-commands; do consequent-commands; done A common design of until in other language is that it allows the loop body be executed at least once and test the condition at the end of the run of the first time. It seems that a better of bash should also follow the practice. If I understand it correctly, the above is exact the same as the following, in which case the do done block can be executed zero time. Because of this, I think that the current 'until' is not necessary, and probably better to change its definition so that it allows the execution of the loop at least once. while ! test-commands; do consequent-commands; done In short, I'd expect the following code echo 9 (not working with the current bash). COUNTER=9 do echo COUNTER $COUNTER let COUNTER-=1 done until [ $COUNTER -lt 10 ]; I'd like to hear for what reason 'until' is designed in the way it is now. Shall we considered to at least allow an option in bash to change it meaning to the one I propose (or adding a different command, like until2, for what I proposed), which give us time to let the orignal until usage dies out. -- Regards, Peng
Re: How to protect > and interpret it later on? (w/o using eval)
> THAT will work. But why are you writing a script to read a shell command > and then execute it? There is already a program that reads shell commands This capability will be useful for debugging bash script. For example, I have a set of commands in a bash script, each of them output some thing to stdout. However, I can not tell the start and end of each output. I could have a command that does what I expect execute.sh to do. cmd1 cmd2 ... I can easily append each command with 'execute.sh', which will not only tell me the boundary of each output but also help me remember what command was executed. execute.sh cmd1 execute.sh cmd2 ... Since what I expect is not possible, the next solution is to use the eval version which requires to put quote around cmd1, cmd2... But this is very annoying, when cmd1, cmd2 themselves have the quotation marks, I have to escape these marks. So neither solution to execute.sh is ideal, although any of them is better than the other in certain situations. -- Regards, Peng
Re: How to protect > and interpret it later on? (w/o using eval)
> WHAT are you trying to DO? I think that you might completely miss my point. I try to explain it better. Let me know if this time it makes more sense to you. I want to execute any command as if the 'execute.sh' does not present, except that I want to print the command so that I know want the command is executed. (This can be used when I call several commands in a script and I know what part of the output associated to what command). E.g. I can run ls > /tmp/tmp.txt When I call, execute.sh ls > /tmp/tmp.txt I want it actually to do echo "ls > /tmp/tmp.txt" ls > /tmp/tmp.txt Note that I could define execute.sh such that execute.sh "ls > /tmp/tmp.txt" means echo "ls > /tmp/tmp.txt" eval "ls > /tmp/tmp.txt" But this interface of execute.sh is not as good the previous one. Note that there could be other symbols that bash normal process, such as '2>&1'. I'm looking for a general solution, Pierre's answer is not as general as I want. The FAQ http://mywiki.wooledge.org/BashFAQ/050 doesn't really answer my question. BTW, where is the help-bash mailing list mentioned (at least not on bash home page)? I have never seen it before. http://tiswww.case.edu/php/chet/bash/bashtop.html -- Regards, Peng
How to protect > and interpret it later on? (w/o using eval)
Hi, ~$ cat ../execute.sh #!/usr/bin/env bash echo "$@" "$@" $ ../execute.sh ls >/tmp/tmp.txt $ cat /tmp/tmp.txt #I don't want "ls" be in the file ls main.sh '>' will not work unless eval is used in execute.sh. $ ../execute.sh ls '>' /tmp/tmp.txt ls > /tmp/tmp.txt ls: cannot access >: No such file or directory /tmp/tmp.txt How to make execute protect > and interpret it later on w/o using eval? -- Regards, Peng
How to directly modify $@?
Hi, I don't see if there is a way to directly modify $@. I know 'shift'. But I'm wondering if there is any other way to modify $@. ~$ 1=x -bash: 1=x: command not found ~$ @=(a b c) -bash: syntax error near unexpected token `a' -- Regards, Peng
Re: converting array to string by quoting each element for eval
> You may safely use getopts (the builtin). Never getopt. If my understanding is correct, 'getopts' doesn't support the long format. Hence, it does not satisfy my need and I shall not use it. -- Regards, Peng
Re: converting array to string by quoting each element for eval
> And that is enough of this nonsense. I have cited three official manuals > for you already. Let's move on. I don't get it. Do you mean both traditional getopt and Debian getopt are broken. To me it seems that Debian getopt is made to address the short coming of transitional getopt. Yet you still think Debian getopt is broken? > Now that we know the goal, it's simple enough: No. That is not my goal. It is just a simplification of my goal. It is really hard to enumerate all the possible use cases, but I will try. In additional to the use cases I stated in the previous email, let's consider some more: ./myscript 'a b' 'c d' -o output.txt -type f -name '*' or ./myscript 'a b' 'c d' -output output.txt -type f -name '*' or ./myscript -output output.txt -type f -name '*' 'a b' 'c d' or ./myscript -type f -name '*' 'a b' -output output.txt 'c d' is equivalent find 'a b' 'a d' -maxdepth 1 -type f -name '*' > output.txt -- Regards, Peng
Re: converting array to string by quoting each element for eval
Hi Greg, > **NEVER** use getopt(1). It is broken. It cannot be made to work > correctly. Its entire design is flawed. I don't see these warnings in my systems (macports and ubuntu) (This is version of getopt on macports and ubuntu is free, I don't see there is a reason that getopt can not be ported to the two systems that you mentioned). All I see that is relevant is the following. I don't think that just because that it has a BUGS section in the manpage, it can be called broken. man bash also has BUGS section, is bash considered as broken? BUGS getopt(3) can parse long options with optional arguments that are given an empty optional argument (but can not do this for short options). This getopt(1) treats optional arguments that are empty as if they were not present. The syntax if you do not want any short option variables at all is not very intuitive (you have to set them explicitely to the empty string). AUTHOR Frodo Looijaard Note that it doesn't mean that I am resistant to getopts. It is just that I don't think that your logic is valid. But I will read more about getopts to see if it is necessary to convert from getopt to getopts. > ./myscript . -type f -name '*' To support your claim, tell me what myscript would be if the commands ./myscript 'a b' 'c d' -type f -name '*" ./myscript 'a b' -type f -name '*" actually do find 'a b' 'a d' -maxdepth 1 -type f -name '*" find 'a b' -maxdepth 1 -type f -name '*" -- Regards, Peng
Re: converting array to string by quoting each element for eval
> Then why don't you post that?
Please take a look and see if you have a better solution than the
following code without using quotearg.sh.
/tmp/tmp$ ll *
c d:
total 0
-rw-r--r-- 1 pengy wheel 0 2011-11-15 20:18:11 xx
a b:
total 0
-rw-r--r-- 1 pengy wheel 0 2011-11-15 20:18:15 yy
/tmp/tmp$ ../convert_args_to_string.sh 'a b' 'c d' -O "-type f -name '*'"
TEMP= -O '-type f -name '\''*'\''' -- 'a b' 'c d'
a b/yy
c d/xx
find 'a b' 'c d' -type f -name '*'
/tmp/tmp$ cat ../convert_args_to_string.sh
#!/bin/bash
TEMP=`getopt -o O: --long options: -n "${script_name}.sh" -- "$@"`
echo TEMP=$TEMP
if [ $? != 0 ] ; then printf "Terminating...\n" >&2 ; exit 1 ; fi
eval set -- "$TEMP"
abspath_script=`readlink -f -e "$0"`
script_absdir=`dirname "$abspath_script"`
while true ; do
case "$1" in
-O|--options)
options="$2"
shift 2
;;
--)
shift
break
;;
*)
echo "Internal error!"
exit 1
;;
esac
done
arg_string=`quotearg.sh "$@"`
cmd="find $arg_string $options"
eval "$cmd"
echo $cmd
--
Regards,
Peng
Re: converting array to string by quoting each element for eval
On Tue, Nov 15, 2011 at 6:43 PM, Chris F.A. Johnson wrote: > On Tue, 15 Nov 2011, Peng Yu wrote: > >>> In any case, combining a command and its arguments in a single >>> string is almost always the wrong way to go about it. >> >> Please compare the two scripts and see if the second one makes more sense. >> >> /tmp$ cat not_convert_args_to_string.sh >> #!/bin/bash >> >> options="$2" >> find $1 $options >> echo find $1 $options > > More sensible would be to have each option a separate argument and > do: > > location=$1 > shift > find "$location" "$@" No. My real example use getopt. If I have each option in a separate argument, I need to know all the possible arguments to find, which is not a viable route. >> /tmp$ cat convert_args_to_string.sh >> #!/bin/bash >> >> options="$2" >> cmd="find $1 $options" >> eval "$cmd" >> echo $cmd > > See above. > >> /tmp$ ./not_convert_args_to_string.sh . "-type f -name '*'" >> find: `./cvcd': Permission denied >> find . -type f -name '*' > > Use 'set -x' to see exactly what your script is doing. How to pass the option "-type f -name '*'" correctly? /tmp$ ./not_convert_args_to_string.sh . "-type f -name '*'" + options='-type f -name '\''*'\''' + find . -type f -name ''\''*'\''' find: `./cvcd': Permission denied + echo find . -type f -name ''\''*'\''' find . -type f -name '*' -- Regards, Peng
Re: converting array to string by quoting each element for eval
> In any case, combining a command and its arguments in a single > string is almost always the wrong way to go about it. Please compare the two scripts and see if the second one makes more sense. /tmp$ cat not_convert_args_to_string.sh #!/bin/bash options="$2" find $1 $options echo find $1 $options /tmp$ cat convert_args_to_string.sh #!/bin/bash options="$2" cmd="find $1 $options" eval "$cmd" echo $cmd /tmp$ ./not_convert_args_to_string.sh . "-type f -name '*'" find: `./cvcd': Permission denied find . -type f -name '*' /tmp$ ./convert_args_to_string.sh . "-type f -name '*'"|head find: `./cvcd': Permission denied ./.main.sh.swp ./0001e4ecf6175 ./alm.log ./amt.log ./convert_args_to_string.sh ./FLEXnet/11.5.0.0 build 56285Macrovision ./FLEXnet/524288-16F7558F-328B-4dc3-BEDF-095C1F14FFF1 ./FLEXnet/524288-34E9EE98-50ED-4c6c-BD0F-F539123FD064 ./FLEXnet/524288-85A0F138-527D-4012-8175-79A3AEA4152E ./FLEXnet/freCqflgCxFrwiBvBiCadibCwg -- Regards, Peng
Re: converting array to string by quoting each element for eval
> Why not use the array instead of making it into a single string?
>
> $cmd "${args[@]}"
>
> Why are you using eval or quotearg.sh? It sounds as if you are
> making the process more complicated than it need be.
For the examples that I gave, probably it is not necessary.
I'm yet to make a concrete complex example to demonstrate its
usefulness in practice. But I think that it is easy to imagine that
there will be some complex commands that are composed by concatenating
multiple strings to be evaled, in which case this may be useful.
--
Regards,
Peng
converting array to string by quoting each element for eval
Hi,
I find that I have to make a program quotearg.sh to convert an array
to a string by quoting each element. So that it be used for eval.
I'm not sure if there is a way that I can do eval in bash without
having to use quotearg.sh. If there is no such a way, should
quotearg.sh be added in bash (if it is not available in bash yet), as
it provides a fundamental functionality?
~/linux/bin/src/bash/quotearg/main$ cat ./main.sh
#!/usr/bin/env bash
../quotearg.sh a b c
../quotearg.sh "'" ' ' '"'
../quotearg.sh 'a' 'a b'
echo
args=('a' 'a b')
cmd="printf 'x%sx\n' ${args[@]}"
eval "$cmd"
echo #the following is what I want, the above is not.
args=('a' 'a b')
arg_string=`../quotearg.sh "${args[@]}"`
cmd="printf 'x%sx\n' $arg_string"
eval "$cmd"
~/linux/bin/src/bash/quotearg/main$ ./main.sh
'a' 'b' 'c'
''\''' ' ' '"'
'a' 'a b'
xax
xax
xbx
xax
xa bx
--
Regards,
Peng
Re: Why complete doesn't print anything if it is called in a bash script?
On Sat, Nov 12, 2011 at 10:18 AM, Chet Ramey wrote: > On 11/12/11 10:41 AM, Peng Yu wrote: >> Hi, >> >> It is strange to me why complete doesn't print anything when it is >> called in a bash script. I must have misunderstood some fundamentals. >> Does anybody know why? Thanks! > > Since complete happily shows completions when run from a shell script, > there must be code in your bashrc that prevents them from being > defined if the shell is not interactive. Thank you for reminding me! That's indeed the case (shown below is from the bashrc file). case $- in *i*) [[ -f /opt/local/etc/bash_completion ]] && . /opt/local/etc/bash_completion ;; esac -- Regards, Peng
Re: Why complete doesn't print anything if it is called in a bash script?
On Sat, Nov 12, 2011 at 10:01 AM, Andreas Schwab wrote: > Peng Yu writes: > >> It is strange to me why complete doesn't print anything when it is >> called in a bash script. I must have misunderstood some fundamentals. >> Does anybody know why? Thanks! > > If complete does not print anything then there are no completions > defined. The question is why. I have source my bashrc, as you can see there are completions defined in the interactive shell. -- Regards, Peng
Why complete doesn't print anything if it is called in a bash script?
Hi, It is strange to me why complete doesn't print anything when it is called in a bash script. I must have misunderstood some fundamentals. Does anybody know why? Thanks! ~$ cat main.sh #!/usr/bin/env bash . ~/.bashrc complete ~$ ./main.sh ~$ complete |head complete -F _kill kill complete -F _renice renice complete -F _smbpasswd smbpasswd complete -F _postconf postconf complete -F _ldapwhoami ldapwhoami complete -F _ldapaddmodify ldapadd complete -F _launchctl launchctl complete -F _java java complete -F _stream stream complete -F _filedir_xspec oodraw -- Regards, Peng
invoke tilde expansion on quoted string
Hi, I know from the document that tilde expansion only works if the string is unquoted (see below) ~$ cd '~/..' -bash: cd: ~/..: No such file or directory ~$ cd ~/.. /Users$ I'm wondering if I already have a string variable, is there a bash native to do tilde expansion on it. var='~/..' cd $var#how to change this line? -- Regards, Peng
Customize the command resolution in bash?
Hi, bash by default searchs in paths specified in the environment variable PATH (separated by ":"). I'm not aware if there is any cache mechanism to save the run time (but even so, different terminals still can not see the same cache, hence each terminal has the overhead to create the cache). When there are many files in PATH, it is going to slow down the performance. One simple remedy is to instead search in a file where the abspaths of all the commands are saved (of course, this database file can be generated by using the command 'find' to search for all the directories in $PATH, which process can be scheduled to run periodically using cron). To make this work, I'm wondering if there is an easy way to customize the way that bash resolve a command. -- Regards, Peng
Re: What is the correct way to set up login environment in crontab?
On Wed, Nov 9, 2011 at 10:41 AM, Greg Wooledge wrote: > On Wed, Nov 09, 2011 at 10:29:52AM -0600, Peng Yu wrote: >> I sourced my ~/.bashrc, which source some other files. It seems the >> environment variables defined in these files are not seen with env. >> Why is so? > > Without seeing the code? Impossible to say. But you're doing it backwards. > ~/.bashrc should be sourced FROM ~/.bash_profile. A login shell reads > ~/.bash_profile only, so it's the responsibility of ~/.bash_profile to > read ~/.bashrc to set up aliases, functions, shopts, and other ephemeral > shell settings that can't be inherited from the environment. Sorry for the confusion. ~/.bash_profile is not the problem here. I have the following line in /path/programtorun.sh . ~/.bashrc In ~/.bashrc, I have ". ~/bash_some". In ~/.bash_some, I have some variable assignment VAR=blah. However, VAR is not seen in env in /path/programtorun.sh (called from cron). -- Regards, Peng
Re: What is the correct way to set up login environment in crontab?
On Wed, Nov 9, 2011 at 7:45 AM, Greg Wooledge wrote: > On Tue, Nov 08, 2011 at 09:46:37PM -0600, Peng Yu wrote: >> I need to use cron to run some job. I know that cron only set up very >> basic environment. I'd like to duplicate my login environment. > > Just source /etc/profile and your ~/.bash_profile or ~/.profile (or > whatever) from the script that your cron job executes. > > Personally I would advise against this. Login environments are for > interactive logins, not cron jobs. I sourced my ~/.bashrc, which source some other files. It seems the environment variables defined in these files are not seen with env. Why is so? -- Regards, Peng
Re: What is the correct way to set up login environment in crontab?
On Wed, Nov 9, 2011 at 7:45 AM, Greg Wooledge wrote: > On Tue, Nov 08, 2011 at 09:46:37PM -0600, Peng Yu wrote: >> I need to use cron to run some job. I know that cron only set up very >> basic environment. I'd like to duplicate my login environment. > > Just source /etc/profile and your ~/.bash_profile or ~/.profile (or > whatever) from the script that your cron job executes. > > Personally I would advise against this. Login environments are for > interactive logins, not cron jobs. So neither -i nor -l is necessary? -- Regards, Peng
What is the correct way to set up login environment in crontab?
Hi, I need to use cron to run some job. I know that cron only set up very basic environment. I'd like to duplicate my login environment. But some environment variables are still not seen when I set the following crontab entry. Does anybody know how to correctly set up the login enviroment? (The manual describes what files are sourced, but I don't want to manually source these files. I'm looking for a simpler way.) * * * * * /bin/bash -i -l -c '/path/programtorun.sh > /tmp/my.log' -- Regards, Peng
Re: What is the best to pass an array with specially characters as command line arguments?
On Mon, Nov 7, 2011 at 8:29 AM, Dennis Williamson
wrote:
> On Mon, Nov 7, 2011 at 7:23 AM, Peng Yu wrote:
>> Hi Clark,
>>
>>> What do you mean by "1 long argument"?
>>>
>>> [bash-4.2.10] # cat foo.sh
>>> v=" a b c ( a'b | "
>>> set -o noglob
>>> a=( $v )
>>> set +o noglob
>>> for i in "${a[@]}"; do
>>> echo "$i"
>>> done
>>> [bash-4.2.10] # bash foo.sh
>>> a
>>> b
>>> c
>>> (
>>> a'b
>>> |
>>> [bash-4.2.10] #
>>
>>
>> I misunderstood the usage of "${args[@]}". I though it returns only
>> one long argument " a b c ( a'b | ", but it actually expanded to 6
>> short arguments "a", "b", "c", "(", "a'b" and "|". Thanks for
>> clarification.
>>
>>
>>
>> --
>> Regards,
>> Peng
>>
>>
>
> If you use "${args[*]}" (with quotes and an asterisk instead of an at
> sign), the result is one long argument. Otherwise, it's split.
Thanks! I didn't notice this difference before.
--
Regards,
Peng
Re: What is the best to pass an array with specially characters as command line arguments?
Hi Clark,
> What do you mean by "1 long argument"?
>
> [bash-4.2.10] # cat foo.sh
> v=" a b c ( a'b | "
> set -o noglob
> a=( $v )
> set +o noglob
> for i in "${a[@]}"; do
> echo "$i"
> done
> [bash-4.2.10] # bash foo.sh
> a
> b
> c
> (
> a'b
> |
> [bash-4.2.10] #
I misunderstood the usage of "${args[@]}". I though it returns only
one long argument " a b c ( a'b | ", but it actually expanded to 6
short arguments "a", "b", "c", "(", "a'b" and "|". Thanks for
clarification.
--
Regards,
Peng
Re: What is the best to pass an array with specially characters as command line arguments?
Hi Clark,
>> > v=" a b c ( a'b | "
>> > a=( $v )
>> > echo "${a[@]}"
>
> There's a @ char here.
I see. It's my mistake.
But I want to pass the 6 short arguments instead of 1 long argument to
echo. (echo is just an example, it can be any command that accepts
multiple arguments.)
~$ cat ./main1.sh
#!/usr/bin/env bash
#set -o noglob
verbatim_string=" a b c ( a'b | "
args=( $verbatim_string )
#set +o noglob
echo "${args[@]}"
~$ ./main1.sh
a b c ( a'b |
--
Regards,
Peng
