Re: Bash-5.2-rc3 available
On 8/26/22 9:29 PM, Kerin Millar wrote: Hi Chet, On Fri, 26 Aug 2022 14:28:00 -0400 Chet Ramey wrote: 1. Changes to Bash a. Added a compatibility mode feature that causes the parser to parse command substitutions as if extglob were enabled. If it is enabled before execution, parse at execution will succeed. If not, the subsequent execution parse will fail. I harbour some concerns as to both the utility and robustness of this. Thanks for the report. This was an issue with both constructs using the same mechanism for parser error recovery and colliding, an easy fix. For extglob to be arbitrarily enabled in either context is unexpected (and undesirable). The conditional command behavior is compatible with ksh93, and has been this way for nearly twenty years. It's documented to behave that way. One would think people are used to it by now. The parser change to enable this for backwards compatibility is controlled by the compatibility mode, and requires an explicit action to enable. It's a choice to enable compatibility mode, and choices have consequences; compatibility mode is intended to provide an opportunity to update scripts, not provide perfect emulation of a previous version. I understand from Sam James that Gentoo globally forces the compatibility mode to 5.0. That, too, is a choice. Paradoxically, this breaks one of the QA tests implemented by portage. It's certainly ironic. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: How should 'vi-yank-pop' be used?
On 8/27/22 2:38 PM, Matteo Paolini wrote: Bash Version: 5.1 Patch Level: 16 Release Status: release Description: I recently discovered Readline's 'yank-pop' and 'vi-yank-pop' commands. From what I understand, 'yank-pop' is supposed to cycle through Bash's internal kill-ring after yanking the most recent killed text. The command works as expected in Emacs mode, but trying to use its counterpart in vi mode seems to just return an error (the bell sound is played). Thanks for the report. It looks like this behavior (which happens only at the beginning of a line) has been around since the original code was contributed in 2011. I'll look at it after bash-5.2/readline-8.2 is released. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: How should 'vi-yank-pop' be used?
Thanks for your reply. > > Thanks for the report. It looks like this behavior (which happens only at > the beginning of a line) has been around since the original code was > contributed in 2011. > That's strange, for me 'vi-yank-pop' issues an error regardless of the cursor's position on the line. Matteo
Re: Light weight support for JSON
On 8/28/22 5:06 PM, Saint Michael wrote: He has a point, though. To have some of the functionality of jq inside Bash may be very useful. If he can supply a patch, why not? Because then it becomes a support and maintenance issue, and a piece of technical debt. A well-encapsulated loadable builtin addresses some of these issues. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Light weight support for JSON
On 8/28/22 8:47 PM, Dale R. Worley wrote: The "obvious" way to support Json in Bash would be a utility that parses Json and produces e.g. a Bash associative array, and conversely a utility that reads a Bash associative array and produces Json. The real limitation is that it's difficult to have a subprocess set Bash's variables. As far as I know, there's no good idiom for that. That's why a loadable builtin is the preferred mechanism. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Light weight support for JSON
On 8/28/22 5:50 PM, Yair Lenga wrote: First, thanks for taking the time to read and provide your thoughts. This is the real value of the discussion/ Second: I'm NOT trying to argue that there isn't valid use for combining bash/curl/jq, Nor do I suggest adding JSON as first class object to bash (Python/node/Perl/Groovy are way ahead ...). I hope to get feedback from other readers for the news group that may find this approach useful. I'll take the very common use case of using AWS CLI, which does produces JSON response for most calls. Processing the response, while possible with JQ, it is challenging to many junior (and intermediate) developers. In many cases, they fall into the traps that I mentioned above - performance (excessive forking or fork/exec), or code that is hard to read (I've seen really bad code - combining pipes of JQ/awk/sed). I'm trying to address those cases. Almost always they fail to properly handle objects with while space, new-lines, etc. To be practical, I'll try to follow the loadable extension path, and see how much I can get thru that path. This is the path I generally recommend. You will have to address the data representation issues through some kind of convention(s), and encapsulating those in a loadable builtin will minimize changes elsewhere. Possibly will make sense to continue discussion with a concrete implementation. I believe the necessary commands are: These can all be options to a single `json' loadable builtin. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Light weight support for JSON
On 8/28/22 2:11 PM, Lawrence Velázquez wrote: On Sun, Aug 28, 2022, at 9:24 AM, Yair Lenga wrote: Wanted to get feedback about the following "extensions" to bash that will make it easier to work with simple JSON object. To emphasize, the goal is NOT to "compete" with Python/Javascript (and other full scale language) - just to make it easier to build bash scripts that cover the very common use case of submitting REST requests with curl (checking results, etc), and to perform simple processing of JSON files. I do not think bash needs to sprout functionality to support every data-exchange format of the month. A loadable module might be okay, I guess. Loadable builtins are the way to do this; you can encapsulate all the functionality you need there without changes to bash itself. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Bash Coding style - Adopting C99 declarations
On 8/28/22 10:47 AM, Yair Lenga wrote: Hi, I've noticed Bash code uses "old-style" C89 declarations: * Parameters are separated from the prototype Changing to C99-style function declarations is something that's on the list to do. * Variables declared only at the beginning of the function * No mixed declaration/statements These are indeed style issues that aren't of general concern. * No block local variables I use these in some cases where I think they make sense. Curious as to the motivation of sticking to this standard for new development/features. Specifically, is there a requirement to keep bash compatible with C89 ? For loadable builtins, there are no restrictions. For the mainline bash code, I'd like to keep it consistent and change it all at once. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Bash-5.2-rc3 available
On Mon, 29 Aug 2022 09:53:39 -0400
Chet Ramey wrote:
> On 8/26/22 9:29 PM, Kerin Millar wrote:
> > Hi Chet,
> >
> > On Fri, 26 Aug 2022 14:28:00 -0400
> > Chet Ramey wrote:
> >
> >> 1. Changes to Bash
> >>
> >> a. Added a compatibility mode feature that causes the parser to parse
> >> command
> >> substitutions as if extglob were enabled. If it is enabled before
> >> execution,
> >> parse at execution will succeed. If not, the subsequent execution
> >> parse will
> >> fail.
> >
> > I harbour some concerns as to both the utility and robustness of this.
>
> Thanks for the report. This was an issue with both constructs using the
> same mechanism for parser error recovery and colliding, an easy fix.
Good! That's certainly a relief.
>
>
> > For extglob to be arbitrarily enabled in either context is unexpected (and
> > undesirable).
>
> The conditional command behavior is compatible with ksh93, and has been
> this way for nearly twenty years. It's documented to behave that way. One
> would think people are used to it by now.
I am aware of how the == and != operators work, and that the pattern on the
right hand side is always treated as an extglob in the shell performing the
test [*]. I don't see what bearing that has on my report or its wording. Might
I enquire as to what it is that I am supposed to be used to on account of the
passage of those twenty years? Certainly not for a command substitution on the
right hand-side of the == operator to implicitly perform the equivalent of
shopt -s extglob within the implied subshell without so much as a by-your-leave
on the part of the programmer, much less doing so in the shell that specified
the command substitution. To make that observation was neither contentious, nor
indicative of a need to study the manual or the CHANGES file more astutely.
For the benefit of anyone else reading, the following amounts to an error in
bash 5.1.16, as it should.
$ shopt extglob
extglob off
$ [[ '' == $( : !() ) ]]
bash: command substitution: line 3: syntax error near unexpected token `('
bash: command substitution: line 3: ` : !() )'
>
> The parser change to enable this for backwards compatibility is controlled
> by the compatibility mode, and requires an explicit action to enable. It's
> a choice to enable compatibility mode, and choices have consequences;
> compatibility mode is intended to provide an opportunity to update scripts,
> not provide perfect emulation of a previous version.
I understand that you are striving to do right by a majority of your users at
all times. The reported issue was a consequence of a regression in a late-stage
release candidate introduced by a new backward compatibility feature that I had
not elected to enable. I found this regression somewhat startling. I hope that
you can understand where my concern stems from, at least.
>
> I understand from Sam James that Gentoo globally forces the compatibility
> mode to 5.0. That, too, is a choice.
Yes, that's right - subject to the "EAPI" level defined by a given ebuild. I
would aver that all of this amounted to a storm in a teacup. In only one single
instance was a pathname expansion found that was incorrectly attempting to
enable the extglob option after parsing has occurred, as was learned in the
course of Sam and I proceeding to work together on this issue. I must emphasise
that it's not that I don't think that backward compatibility is important. Far
from it. Yet, I can't shake the feeling I had from the outset that, in this
particular instance, the compromise may simply not be worth it. I considered
the behaviour of the prior release candidates to be a step in the right
direction. In any case, I have already expressed as much so I'll leave it at
that and shall look forward to the next release candidate, should there be one.
>
> > Paradoxically, this breaks one of the QA tests implemented by portage.
>
> It's certainly ironic.
In case you were wondering, it's a test that entails stashing the output of
shopt then later comparing, so as to determine whether an ebuild/eclass changed
any options without restoring their prior values. Coercing extglob on results
in a constant stream of false positives.
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
>``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
>
[*] Notwithstanding the change introduced by 4.1, whereby it was no longer
necessary to either enable extglob first or store the pattern in a variable
that was subsequently expanded on the right hand side of the operator.
--
Kerin Millar
Re: Light weight support for JSON
On 2022-08-29 11:43, Chet Ramey wrote: On 8/28/22 2:11 PM, Lawrence Velázquez wrote: On Sun, Aug 28, 2022, at 9:24 AM, Yair Lenga wrote: Wanted to get feedback about the following "extensions" to bash that will make it easier to work with simple JSON object. (...) just to make it easier to build bash scripts that cover the very common use case of submitting REST requests with curl (checking results, etc), and to perform simple processing of JSON files. I do not think bash needs to sprout functionality to support every data-exchange format of the month. Loadable builtins are the way to do this In that case could I suggest providing a simplified, stable programming interface for loadable builtins? (I understand this is not a trivial thing, of course.) As I understand it (having made a few loadable built-ins myself, but possibly a bit ignorant/hazy on the details) - the process involves a fair bit of hands-on with the nitty-gritty of the version of Bash you're building against. A loadable needs to deal directly with implementation details of how Bash handles variables, open file handles, etc., and resulting code (source and binary) is very sensitive to changes in Bash. For me personally I was struggling with questions like, how do I make a loadable module that sets a variable? Sounds simple but it seems to require a fair bit of attention to implementation details of the shell to get it right. What if it's a local variable? What if it's an array or hash element? And so on. I understand the hesitancy to support "format of the month" data formats directly in the shell. Though I think JSON has proven itself to be rather worth supporting, personally: to call it a "format of the month" is almost a bit comical. It's been quite relevant for the last 15 years or so at least. In principle, "extending" the shell to support some new data structure should be a task we can delegate to an external program. (That is, basically, the ideal of the Unix shell) In practice, though, if we use an external program to parse a hierarchical format like JSON, it can't effectively communicate the result back to the shell. To do so, it would need to encode the data, and the encoded data would have roughly the same complexity as JSON. For the external tool to actually simplify the process, the shell and the tool would have to share a common language. (JSON could serve as such a language.) It would also help greatly if the shell could internally handle hierarchical data in variables. (That is a pretty basic problem facing any potential implementation of a JSON parser for Bash: When the data is parsed, what does the parser do with it? It can't store the result in a variable because variables aren't hierarchical.)
Re: bug-bash Digest, Vol 237, Issue 30
On 8/28/22 1:17 PM, Yair Lenga wrote: Yes, you are correct - (most/all of) of those examples "K&R". However, given bash's important role in modern computing - isn't it time to take advantage of new language features ? this can make code more readable, efficient and reliable. There's no actual evidence for this assertion. I doubt that many users are trying to install a new bash in a system that was built/configured 15 years ago. You might be surprised. I have corresponded with folks who maintain and distribute 4.3 BSD systems and want to use bash on them. (There are many bigger problems with that than using prototypes in source code, without a doubt.) The oldest I've ever personally built a `modern' bash version on is Openstep 4.2, and the most recent version I've built there is 5.0.2. That's not nearly as big a headache as something like 4.3 BSD. Many Java/python/C++ projects that want to move forward do it as part of the "major" release, in which they indicate Java 7 (or java java 8) support will be phased out. Same for C++ and python. Bash has a different set of dependencies. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: How should 'vi-yank-pop' be used?
On 8/29/22 11:20 AM, Matteo Paolini wrote: Thanks for your reply. Thanks for the report. It looks like this behavior (which happens only at the beginning of a line) has been around since the original code was contributed in 2011. That's strange, for me 'vi-yank-pop' issues an error regardless of the cursor's position on the line. After further testing, you're right, I'll fix that in readline-8.3, too. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
