Re: [PATCH 0/9] Add library mode to source builtin
> I'm thinking of the theoretical case in which a user exports BASH_SOURCE_PATH > then runs existing scripts - not necessarily of their own design > that weren't counting on this feature to ever exist. The option solution completely avoids that possibility. Existing scripts do not pass any options to the source builtin. No unexpected incompatibility should arise if it ignores the new variable unless the option is passed. Matheus
Re: [PATCH 0/9] Add library mode to source builtin
> It does not. It searches for readable files. Look at > findcmd.c:find_path_file(). That's true, I stand corrected. I was thinking of the find_user_command function because it's what I based the find_user_library function on. They are all implemented in terms of the find_user_command_internal. The find_user_command function passes FS_EXEC_PREFERRED|FS_NODIRS while find_path_file just passes FS_READABLE. I suppose I should also have passed FS_READABLE. Matheus
Re: [PATCH 0/9] Add library mode to source builtin
On 5/7/24 3:40 PM, Kerin Millar wrote: On Tue, 7 May 2024, at 7:14 PM, Chet Ramey wrote: On 5/7/24 1:42 PM, Kerin Millar wrote: On Tue, 7 May 2024, at 3:27 PM, Chet Ramey wrote: On 5/5/24 3:39 PM, Kerin Millar wrote: Such is the extent to which I concur that I find even -l to be irritating. The option character isn't important. Is it useful to have an additional If it were of no importance at all, once might then choose the character entirely at random. That's not usually what happens. The issue is whether one is needed at all, not whether or not one character irritates you. If you meant something to the effect of "in the case that I decide that having an option is worthwhile, the choice of character is hardly the foremost concern in all of this", then by all means. Pretty much "Don't get hung up on the specific character until we decide we need one. Certainly don't let one irritate you." -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [PATCH 0/9] Add library mode to source builtin
On Tue, 7 May 2024, at 7:14 PM, Chet Ramey wrote: > On 5/7/24 1:42 PM, Kerin Millar wrote: >> On Tue, 7 May 2024, at 3:27 PM, Chet Ramey wrote: >>> On 5/5/24 3:39 PM, Kerin Millar wrote: >>> Such is the extent to which I concur that I find even -l to be irritating. >>> >>> The option character isn't important. Is it useful to have an additional >> >> If it were of no importance at all, once might then choose the character >> entirely at random. That's not usually what happens. > > The issue is whether one is needed at all, not whether or not one character > irritates you. In the course of quoting me, you stated that the character isn't important then proceeded to pose an open question as to whether having an option is worthwhile, so I engaged on both counts. I don't require for anyone's agreement to think that naming matters. I don't believe that anyone that writes software genuinely believes it doesn't. I didn't say what the character ought to be. I simply didn't like the one that was presented and still don't. If you meant something to the effect of "in the case that I decide that having an option is worthwhile, the choice of character is hardly the foremost concern in all of this", then by all means. >>> option to `source' that forces it to use $BASH_SOURCE_PATH, or should that >>> behave like other builtins ($CDPATH, $BASH_LOADABLES_PATH)? >> >> If BASH_SOURCE_PATH is to be able to take its value from the environment, it >> might be useful. > > That's the standard behavior, with only a few exceptions. > > >> That is, to have a means of opting in may be sensible from a backward- >> compatibility standpoint, whether it be in the form of an option character, >> a >> shell option or something else. Otherwise, probably not. I'm thinking of the >> theoretical case in which a user exports BASH_SOURCE_PATH then runs existing >> scripts - not necessarily of their own design - that weren't counting on >> this >> feature to ever exist. > > You can export CDPATH and get the same effect with `cd', or export PATH and > modify command search order. Yes (albeit standard behaviours). I find myself mildly inclined towards the position of not adding more options unless it is considered truly necessary. I have nothing further to add, so will take my leave of this thread. -- Kerin Millar
Re: [PATCH 0/9] Add library mode to source builtin
On 5/7/24 1:42 PM, Kerin Millar wrote: On Tue, 7 May 2024, at 3:27 PM, Chet Ramey wrote: On 5/5/24 3:39 PM, Kerin Millar wrote: Such is the extent to which I concur that I find even -l to be irritating. The option character isn't important. Is it useful to have an additional If it were of no importance at all, once might then choose the character entirely at random. That's not usually what happens. The issue is whether one is needed at all, not whether or not one character irritates you. option to `source' that forces it to use $BASH_SOURCE_PATH, or should that behave like other builtins ($CDPATH, $BASH_LOADABLES_PATH)? If BASH_SOURCE_PATH is to be able to take its value from the environment, it might be useful. That's the standard behavior, with only a few exceptions. That is, to have a means of opting in may be sensible from a backward- compatibility standpoint, whether it be in the form of an option character, a shell option or something else. Otherwise, probably not. I'm thinking of the theoretical case in which a user exports BASH_SOURCE_PATH then runs existing scripts - not necessarily of their own design - that weren't counting on this feature to ever exist. You can export CDPATH and get the same effect with `cd', or export PATH and modify command search order. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [PATCH 0/9] Add library mode to source builtin
On Tue, 7 May 2024, at 3:27 PM, Chet Ramey wrote: > On 5/5/24 3:39 PM, Kerin Millar wrote: > >> Such is the extent to which I concur that I find even -l to be irritating. > > The option character isn't important. Is it useful to have an additional If it were of no importance at all, once might then choose the character entirely at random. That's not usually what happens. > option to `source' that forces it to use $BASH_SOURCE_PATH, or should that > behave like other builtins ($CDPATH, $BASH_LOADABLES_PATH)? If BASH_SOURCE_PATH is to be able to take its value from the environment, it might be useful. That is, to have a means of opting in may be sensible from a backward-compatibility standpoint, whether it be in the form of an option character, a shell option or something else. Otherwise, probably not. I'm thinking of the theoretical case in which a user exports BASH_SOURCE_PATH then runs existing scripts - not necessarily of their own design - that weren't counting on this feature to ever exist. On the other hand, bash already has a cornucopia of options and I'm uncertain as to how realistic this concern is. There are various ways for the environment to influence the behaviour of a bash script as it stands. I would be interested to know what others think. -- Kerin Millar
Re: [PATCH 0/9] Add library mode to source builtin
On 5/5/24 11:55 PM, Matheus Afonso Martins Moreira wrote: Such is the extent to which I concur that I find even -l to be irritating. Well, I don't concur at all. I think it's just the opposite. I think it's actually the source builtin which incorrectly implies that all sourced scripts are executables when they might actually be libraries and modules. It implies this because the source code for source prefers files with the executable bit set It does not. It searches for readable files. Look at findcmd.c:find_path_file(). -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: [PATCH 0/9] Add library mode to source builtin
On 5/5/24 3:39 PM, Kerin Millar wrote: Such is the extent to which I concur that I find even -l to be irritating. The option character isn't important. Is it useful to have an additional option to `source' that forces it to use $BASH_SOURCE_PATH, or should that behave like other builtins ($CDPATH, $BASH_LOADABLES_PATH)? -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Re: Re: [PATCH 0/9] Add library mode to source builtin
Honestly, I share the same feeling as the others about the name `library'. The use of the ``library'' feels something wrong in this context. I still think the variable name should be BASH_SOURCE_PATH. I've suggested `-l' as an *example*, but that's just because I couldn't come up with another letter. It could probably be `-p' following the similar option of `command -p'. The reasons that using "library" in this context feels strange are as follows: * The suggested feature just changes how the script file would be searched. This doesn't define what would be the library in Bash and what would not. The suggested feature can be used to load a library, but the reverse is not true; a library doesn't need to be sourced through the suggested feature. Even if a script file that contains a set of utilities is sourced with an absolute/relative path with a slash, I'd still call it a library. Even if it is found in PATH, I'd still call it a library. Even if it is found in the current working directory, I'd still call it a library. The suggested command-line flag forces the search to be in BASH_LIBRARIES_PATH and disables the search of libraries in PATH and the current working directory. You seem to try to justify the naming by explaining how the script files loaded by the suggested feature can be considered libraries, but to justify the naming you would need to explain also how the script files with utilities located in PATH or the current working directory are NOT considered libraries. If you would try that, I will contest against it. * We haven't been using the term `library' in the Bash codebase to mean the general scripts that contain a set of functions and are intended to be sourced. These have been just called script files without a specific name. When we introduce a new term, I'd expect a clear difference from the existing one. If the new mechanism is incompatible with the conventional script files, it would suffice as a difference. Or if the suggested feature would introduce new mechanisms dedicated to libraries (such as namespacing, dependency resolution, etc.) and those mechanisms are not going to be provided/available with the conventional `source' builtin, it would also suffice as a difference. As far as the new term means essentially a similar thing as existing script files, I don't see a reason to introduce a new term. * I feel not every script file that is intended to be sourced and usefully placed in BASH_LIBRARIES_PATH would be actually a library. It might define aliases and shell functions that the user can interactively call from the command line. It can also be an initialization code for the environment. It can also be just personal configurations. For example, would you call a single line file « eval "$(starship init bash)" » a library? Maybe you call it a library, but I wouldn't.
Re: Re: Re: [PATCH 0/9] Add library mode to source builtin
> Libraries are 'compiled collection of object module', can be static/dynamic, > dynamic imply a runtime load. By this definition, dynamic languages and scripting languages don't have libraries. They clearly do have libraries. Sometimes they're called modules but it's the same thing. > This is valid for interpreters with JIT compiler that can produce librairies. It's also valid for code that's not compiled at all. > Bash do have libraries by means of loadable .so files, .so files can > legitimatly called libraries. Those are called shared objects, as implied by the .so extension. Bash documentation also calls them shared objects, not libraries. https://www.gnu.org/software/bash/manual/html_node/Bash-Builtins.html#index-enable > The -f option means to load the new builtin command name from shared object > filename, > on systems that support dynamic loading. Bash will use the value of the > BASH_LOADABLES_PATH > variable as a colon-separated list of directories in which to search for > filename. Shared objects can also be executables. The dynamic linker is an example. /usr/lib/ld-linux-x86-64.so.2 --help Usage: /usr/lib/ld-linux-x86-64.so.2 [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...] You have invoked 'ld.so', the program interpreter for dynamically-linked ELF programs. Usually, the program interpreter is invoked automatically when a dynamically-linked executable is started. You may invoke the program interpreter program directly from the command line to load and run an ELF executable file; this is like executing that file itself, but always uses the program interpreter you invoked, instead of the program interpreter specified in the executable file you run. Invoking the program interpreter directly provides access to additional diagnostics, and changing the dynamic linker behavior without setting environment variables (which would be inherited by subprocesses). Note how I had to use the full path to the linker in order to invoke it. It's a library. It's not in the PATH. And yet it's an executable. You can have perfectly normal executables whose functions you can link against and use as though they were themselves libraries. With GCC, you make them by passing -fPIC -pie -rdynamic to create a position independent executable which includes all defined symbols in the ELF table, not just the used ones. The result of that is an executable whose symbols can be looked up, linked and relocated. In other words, a library that can also run. As you can see, the lines can be quite blurry indeed. When it comes down to it, the only difference between an executable and a library is whether it does something useful when you run it directly. The presence or absence of an ELF entry point for the program. The existence of a main function. In scripting languages, the main function is implicit. The code is usually interpreted line by line by a virtual machine. This interpretation essentially is the main function. Functions are defined dynamically as the interpreter moves through the code. So the only way to distinguish libraries from executables is to look at the what the code does and interpret its purpose. Does it just define functions and exits? That's a library. Or does it actually do something with those functions? That's an executable. > A 'script' is a bash source file with the +x bit set (executable) https://en.wikipedia.org/wiki/Scripting_language > In computing, a script is a relatively short and simple set of instructions that typically automate an otherwise manual process. What are scripts called when they automate no tasks and contain nothing but function and variable definitions? I call them libraries.
Re: Re: [PATCH 0/9] Add library mode to source builtin
On Mon, May 6, 2024 at 5:43 AM Matheus Afonso Martins Moreira < math...@matheusmoreira.com> wrote: > > I think every single use of the term "library" in this whole endeavor > > is misleading and misguided. > > I don't think so. A library is just a logical collection of code and data > that you can use to implement software. This is a mechanism meant > to load those libraries separately from executables. > Not in my parlance. Libraries are 'compiled collection of object module', can be static/dynamic, dynamic imply a runtime load. This is valid for interpreters with JIT compiler that can produce librairies. Bash do have libraries by means of loadable .so files, .so files can legitimatly called libraries. Those libraries do have their own PATH thing (i.e LD_LIBRARY_PATH) A 'script' is a bash source file with the +x bit set (executable), and as such don't have extension like 'foo'. A non executable bash file, better with a .sh extension, is a bash source. The source builtin just load 'file', albeit .sh file, not library.
Re: Re: [PATCH 0/9] Add library mode to source builtin
> Such is the extent to which I concur that I find even -l to be irritating. Well, I don't concur at all. I think it's just the opposite. I think it's actually the source builtin which incorrectly implies that all sourced scripts are executables when they might actually be libraries and modules. It implies this because the source code for source prefers files with the executable bit set, and also because it searches for the files in a list of paths meant for executables/commands. The implication is obviously false. This patch allows them to be avoided. It simply adds an option to source. I really don't understand how it could possibly be irritating to anyone. -- Matheus
Re: Re: [PATCH 0/9] Add library mode to source builtin
> I think every single use of the term "library" in this whole endeavor
> is misleading and misguided.
I don't think so. A library is just a logical collection of code and data
that you can use to implement software. This is a mechanism meant
to load those libraries separately from executables.
The same reasons which justify having separate /usr/{bin,lib}
also justify this feature. The library directories are not placed
in the PATH even though shared objects might be executable.
The binary directories on the other hand are placed in the PATH
even though executables might also be libraries at the same time.
It's all about how the developer intends the software to be used.
This loading mechanism makes it easy to use in the intended way.
That's all.
> It implies something unique about the files being sourced
I don't think it necessarily implies anything about the scripts
other than the purpose for which they were written.
Usually, executables perform concrete tasks
while libraries merely provide the building blocks
necessary for accomplishing said tasks.
That's the only general implication that may be drawn.
It does not necessarily need to be that way but it usually is.
> when in fact there is nothing special about them at all
> they can still run arbitrary code, modify the global namespace,
> etc. etc. etc. etc.
This is also true in essentially every other programming language.
There is nothing special about libraries, it's just code and data.
They usually provide the building blocks for building an executable
but that's not actually mandated.
Dynamic languages have libraries which are literally just scripts
that are evaluated when imported. Essentially the same thing
that's being proposed here. Their library systems consist of an
in-language loading mechanism, which this patch implements,
and also usually an external package manager and repository
which is out of scope here and not actually necessary.
Even static compiled shared objects are not actually static
collections of functions and variables, they can and do have
things such as init/fini sections which run arbitrary code
implicitly when they are loaded and unloaded. They had to
implement topological sorting in the dynamic linker for this.
The dynamic linker actually does dependency resolution.
I think that's actually kind of insane but somebody needed it
and they made it happen.
Some languages have isolated modules, others don't.
Some natively support namespacing, others don't.
The most common complication among them is
the load-once mechanism: usually only a single copy
of a library is ever loaded by the interpreter, no matter
how many times it's referenced.
I am of course open to discussing and implementing all of
these additional semantics. In the previous email thread,
I said I was even hoping to have this discussion.
However, the lack of these semantics does not preclude
bash from having a library system. It's already functional.
I think load-once is important and should be implemented
but opinions appeared to diverge in the comments to my first
patch set so I would rather wait for consensus before doing it.
-- Matheus
Re: [PATCH 0/9] Add library mode to source builtin
On Sun, 5 May 2024, at 8:11 PM, Lawrence Velázquez wrote: > On Sun, May 5, 2024, at 5:54 AM, Matheus Afonso Martins Moreira wrote: >> This patch set adds a special operating mode to the existing source >> builtin to make it behave in the desired way. When source is passed >> the options --library or -l, it will search for files in the >> directories given by the BASH_LIBRARIES_PATH environment variable, > > I think every single use of the term "library" in this whole endeavor > is misleading and misguided. It implies something unique about the > files being sourced, when in fact there is nothing special about > them at all -- they can still run arbitrary code, modify the global > namespace, etc. etc. etc. etc. Such is the extent to which I concur that I find even -l to be irritating. -- Kerin Millar
Re: [PATCH 0/9] Add library mode to source builtin
On Sun, May 5, 2024, at 5:54 AM, Matheus Afonso Martins Moreira wrote: > This patch set adds a special operating mode to the existing source > builtin to make it behave in the desired way. When source is passed > the options --library or -l, it will search for files in the > directories given by the BASH_LIBRARIES_PATH environment variable, I think every single use of the term "library" in this whole endeavor is misleading and misguided. It implies something unique about the files being sourced, when in fact there is nothing special about them at all -- they can still run arbitrary code, modify the global namespace, etc. etc. etc. etc. -- vq
