Re: can you parse and "tail" at once? (and if you can't why not?)
Thank you very much, Greg!
Since ".description" is constant (an extension used by youtube) I chose to go:
ydx=".description"
ydxL=${#ydx}
...
yut=${yl:-${ydxL}}
...
where yl is the line read in in the way you suggested.
lbrtchx
Re: can you parse and "tail" at once? (and if you can't why not?)
Hello, On Mon, Oct 23, 2023 at 01:32:08AM +, Albretch Mueller wrote: > On 10/22/23, Andy Smith wrote: > > So you might consider telling us what you will do next with the > > suffix of each line. > > Now you have gone into mind reading mode. I've gone into "avoid the need for mind reading" mode. Because I could see that from your very limited provision of context, you were going to get a lot of different solutions from different people, and that you were likely to disregard most of those due to context you've not supplied. So I've suggested you supply more contact so that people DON'T have to read your mind. > On 10/22/23, Andy Smith wrote: > > Most of the points Greg makes to you are matters of correctness, not > > matters of taste. > > that you called "matters of correctness" may be "visual things" to > other people. I think it's important to you understand that often only the experienced person can tell the difference. > from where do you take the time to keep talking about any of it? Okay, message received and understood. I won't be troubling you again. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Shell-script variable name case [WAS: Re: can you parse and "tail" at once? (and if you can't why not?)]
On 23/10/2023 10:56, David wrote: Hi, for your info, this convention is specified by POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html Which says: Environment variable names used by the utilities in the Shell and Utilities volume of POSIX.1-2017 consist solely of uppercase letters, digits, and the ( '_' ) from the characters defined in Portable Character Set and do not begin with a digit. And: The name space of environment variable names containing lowercase letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behavior of the standard utilities. That's good to know, and pretty much invalidates everything I said. -- BOFH excuse #239: CPU needs bearings repacked Eduardo M KALINOWSKI edua...@kalinowski.com.br
Shell-script variable name case [WAS: Re: can you parse and "tail" at once? (and if you can't why not?)]
On Mon, 23 Oct 2023 at 13:25, Eduardo M KALINOWSKI wrote: > On 22/10/2023 23:13, Greg Wooledge wrote: > > 2) All-caps variable name IFL. All-caps variable names are reserved, > >by convention, for environment variables (e.g. PATH) and special > >shell variables (e.g. IFS). > While I don't disagree with the suggestion of using lower case for > variables (and have even started doing so myself), it seems that this > "convention" is far from universal. > I did a quick search on the bash manual page and found no suggestion on > how to name user variables, or that uppercase names are reserved (but it > was a very quick search - I might have skipped something). > Even an internet search shows that people seem to be divided: Hi, for your info, this convention is specified by POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html Which says: Environment variable names used by the utilities in the Shell and Utilities volume of POSIX.1-2017 consist solely of uppercase letters, digits, and the ( '_' ) from the characters defined in Portable Character Set and do not begin with a digit. And: The name space of environment variable names containing lowercase letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behavior of the standard utilities.
Re: can you parse and "tail" at once? (and if you can't why not?)
On 22/10/2023 23:13, Greg Wooledge wrote: 2) All-caps variable name IFL. All-caps variable names are reserved, by convention, for environment variables (e.g. PATH) and special shell variables (e.g. IFS). While I don't disagree with the suggestion of using lower case for variables (and have even started doing so myself), it seems that this "convention" is far from universal. I did a quick search on the bash manual page and found no suggestion on how to name user variables, or that uppercase names are reserved (but it was a very quick search - I might have skipped something). Even an internet search shows that people seem to be divided: this style guide[0], for example, suggests lower case variable names (for the same reason), but suggests that if the variable is actually a constant, it should be upper case. (This seems to be influenced by Java conventions.) [0] https://google.github.io/styleguide/shellguide.html#s7-naming-conventions This guy[1] even says that all variables should be uppercase. [1] https://www.linkedin.com/pulse/bash-scripting-conventions-engin-polat It's certainly very common (even if unfortunate) to use all uppercase variables, as you certainly know. So while I agree that it's a good idea to use lowercase variable names, the way you put it seems a bit too strong. I'd call it a preference, or even a matter of style, but convention to me implies something that's more widely agreed upon or more widespread, which it's not (unfortunately). -- Every word is like an unnecessary stain on silence and nothingness. -- Beckett Eduardo M KALINOWSKI edua...@kalinowski.com.br
Re: can you parse and "tail" at once? (and if you can't why not?)
On Mon, Oct 23, 2023 at 01:21:43PM +0530, Pankaj Jangid wrote:
> Another way to do it is,
>
> cat "${IFL}"| cut -d "/" -f7-|sed 's/.description//'
>
> because you already know the prefix, you can count the fields. So "-f7-"
> i.e. 7 onwards. Then use sed to remove the extension.
Whoops! I completely missed the requirement of removing ".description"
in my original answer.
Here's a corrected version:
#!/bin/bash
inputfile=${1-/whatever}
prefix='[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/'
while IFS= read -r line; do
line=${line#"$prefix"}
printf '%s\n' "${line%.description}"
done < <(grep -F -- "$prefix" "$inputfile")
Pankaj's answer is not wrong, but it dropped the "grep" which filters
the lines containing the desired prefix. I'd also add an $ anchor to
the sed regex, and escape the . with backslash so that it becomes
literal. This answer also requires counting the fields by hand each
time the prefix changes, which may or may not be a concern.
Re: can you parse and "tail" at once? (and if you can't why not?)
Albretch Mueller writes:
> After generating a file with lines of text, I have to:
> 1) parse some of those lines based on a pattern I know (no regex
> necessary, just a FS path)
> 2) once parsed from those lines I need the last n characters only
> I am trying to use a one liner like:
> cat "${IFL}" | grep "${DESC_DIR}" | tail -c+$_DESC_DIR
> but this one liner repeats the output and the tail
>
> Say the lines of text in IFL are:
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Uppsala_Algebra/45_16.3_45MEB5h5H9Y.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Uppsala_Algebra/46_17.1_LE_gl5RcDnY.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/17_9jKzwoBGFs8.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/18_plOUIOo91lI.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/19_TJ7yAiq8gEw.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/20_UMBEgb14uqw.description
>
> I know the prefix:
> [info] 123XYZ: /media/user/1234qwer/Algebraic_Geometry04/20231022040239/
>
> I need as output:
>
> Uppsala_Algebra/45_16.3_45MEB5h5H9Y
> Uppsala_Algebra/46_17.1_LE_gl5RcDnY
> Göttsche/17_9jKzwoBGFs8
> Göttsche/18_plOUIOo91lI
> Göttsche/19_TJ7yAiq8gEw
> Göttsche/20_UMBEgb14uqw
> ~
> lbrtchx
Another way to do it is,
cat "${IFL}"| cut -d "/" -f7-|sed 's/.description//'
because you already know the prefix, you can count the fields. So "-f7-"
i.e. 7 onwards. Then use sed to remove the extension.
Re: can you parse and "tail" at once? (and if you can't why not?)
On Sun, Oct 22, 2023 at 10:13:26PM -0400, Greg Wooledge wrote: [...] Good points all over -- I envy your clarity. To drive home Greg's point about naming conventions a bit more: bash (in general, shells) have very little protection for mis-naming variables (you don' have to declare them, they have dynamic, not lexical bindings, etc.). That's why naming conventions are more important there. In shells, there's more: shell variables and environment variables (those you "export") are quite different beasts, so making them visually differentiable is the more important. See, a programming language is a communication tool. You talk to other people with it. Observing conventions is part of a language. Cheers -- t signature.asc Description: PGP signature
Re: can you parse and "tail" at once? (and if you can't why not?)
On Sun, Oct 22, 2023, 10:39 PM Albretch Mueller wrote: > OK, Greg's suggestion once again "made my day". > I know at some point I will have to code everything in some > programming language, but for now I will just get things done as > quickly as possible. > Also, Greg, please, I would like for you to understand that it is not > my intention to upset you about what you find visually upsetting. We > have talked about that before. > I am sure you could certainly understand that there are reasons why > some people build certain habits and that not everyone gets upset > about the same kinds of "secondary accidents" (as Aristotle would put > it). Some people may have a hard time even noticing what upsets you. > lbrtchx > > Context? Totally lost. No need to reply, I haven't been following anyway.
Re: can you parse and "tail" at once? (and if you can't why not?)
On Mon, Oct 23, 2023 at 01:32:08AM +, Albretch Mueller wrote:
> On 10/22/23, Andy Smith wrote:
> > Most of the points Greg makes to you are matters of correctness, not
> > matters of taste.
>
> that you called "matters of correctness" may be "visual things" to
> other people. this is something I see on a daily basis I don't try to
> "correct" my students ways, but help them learn, see things for
> themselves.
Was I not sufficiently clear in my explanations? Sorry.
Your original post contained this line of code:
cat "${IFL}" | grep "${DESC_DIR}" | tail -c+$_DESC_DIR
In the interests of learning, let's analyze this line *in full*. From
left to right:
1) Useless use of cat. In this particular example, the result is still
correct. The code is simply longer and slower than it needs to be.
2) All-caps variable name IFL. All-caps variable names are reserved, by
convention, for environment variables (e.g. PATH) and special shell
variables (e.g. IFS). I assume your variable IFL stands for "input
file". But note how close it is to IFS. I can easily imagine that
at some point in the future, if you continue using all-caps variable
names inappropriately, you might accidentally use one that collides
with an environment variable or a special shell variable.
Accidentally overwriting IFS is going to break your scripts faster
than you can imagine.
3) grep "${DESC_DIR}". Another all-caps variable name, but I won't repeat
that point. $DESC_DIR is being passed to a command that expects a
regular expression, but in your example, it's a literal string.
Therefore you should use grep -F. Or use a regular expression in
DESC_DIR, but that wouldn't be my preferred choice.
4) The tail command is not correct here. This is the primary error.
The tail needs to be replaced to make the script work. Tail operates
on whole files, not on each line of input, regardless of what options
you use.
5) You used the _DESC_DIR variable and the DESC_DIR variable together in
the same command. I can only imagine one of them was a typo for the
other.
6) The _DESC_DIR variable expansion is not quoted. Assuming that it was
meant to be DESC_DIR which contains a string with spaces and glob
characters in it, it *must* be quoted, or else it gets passed as an
unguessable number of separate arguments.
Those are the actual issues I spotted. I omitted a few in my previous
post, because I wasn't trying to be fully pedantic. I am now.
Beyond that, I pointed out (by inference) that the code you wrote was
too *limited*. If the errors are corrected, it will print the output
that you asked for. But it can't be extended easily to do anything *more*
than that.
Assuming that you actually wanted to *do* something with each line's
suffix, it makes more sense to include some sort of loop, in which you
can add arbitrary commands to do whatever task is needed.
That's why I replaced the tail with a "while read" loop instead of a
single self-contained command, such as sed, which could have done the
printing task more concisely.
Finally, one might wonder why I used this form:
while IFS= read -r line; do
done < <(grep ...)
instead of the simpler form:
grep ... |
while IFS= read -r line; do
...
done
This was another conscious choice to offer more flexibility for extending
the script. In the simpler form, the while loop is executed in a subshell
(because of the pipeline). When the loop terminates, any changes made in
the subshell are discarded. This makes it impossible to set persistent
variables inside the loop, and then use them afterward.
The form that I used, with the process substition <(...), runs the loop
in the main shell process, not in a subshell. With this form, it's
possible to set variables within the loop, and refer to them afterward.
If you have additional questions, feel free to ask them.
Re: can you parse and "tail" at once? (and if you can't why not?)
On 10/22/23, Andy Smith wrote: > So you might consider telling us what you will do next with the > suffix of each line. Now you have gone into mind reading mode. On 10/22/23, Andy Smith wrote: > Most of the points Greg makes to you are matters of correctness, not > matters of taste. that you called "matters of correctness" may be "visual things" to other people. this is something I see on a daily basis I don't try to "correct" my students ways, but help them learn, see things for themselves. > I don't know how many times you will have the opportunity to > continue talking with knowledgable people who try to help you, if > you dismiss their points as them just being "upset" over aesthetics. and I don't understand your adhominen cr@p. For one, Greg understood right away my problem which I explicitly explained, so from where do you take the time to keep talking about any of it? On 10/22/23, gene heskett wrote: > Very well said Andy. > "There are four boxes to be used in defense of liberty: > soap, ballot, jury, and ammo. Please use in that order." > -Ed Howdershelt (Author, 1940) > If we desire respect for the law, we must first make the law respectable. > - Louis D. Brandeis Nor do I understand what support from "the international community" (tm) has to do with some Linux bash script. lbrtchx
Re: can you parse and "tail" at once? (and if you can't why not?)
On 10/22/23 17:11, Andy Smith wrote: Hello, On Sun, Oct 22, 2023 at 08:50:55PM +, Albretch Mueller wrote: Greg, please, I would like for you to understand that it is not my intention to upset you about what you find visually upsetting. We have talked about that before. Most of the points Greg makes to you are matters of correctness, not matters of taste. I don't know how many times you will have the opportunity to continue talking with knowledgable people who try to help you, if you dismiss their points as them just being "upset" over aesthetics. I am sure you could certainly understand that there are reasons why some people build certain habits and that not everyone gets upset about the same kinds of "secondary accidents" (as Aristotle would put it). Some people may have a hard time even noticing what upsets you. Most of the time when asking for help from people with a lot more knowledge than me in a given subject, I tend to assume that if I find something they say to be oddly picky or unnecessarily strict then it's most likely because of a gap in my own knowledge, rather than them trying to make me slaves to their own personal style. Thanks, Andy Very well said Andy. Cheers, Gene Heskett. -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author, 1940) If we desire respect for the law, we must first make the law respectable. - Louis D. Brandeis
Re: can you parse and "tail" at once? (and if you can't why not?)
Hello, On Sun, Oct 22, 2023 at 08:50:55PM +, Albretch Mueller wrote: > Greg, please, I would like for you to understand that it is not my > intention to upset you about what you find visually upsetting. We > have talked about that before. Most of the points Greg makes to you are matters of correctness, not matters of taste. I don't know how many times you will have the opportunity to continue talking with knowledgable people who try to help you, if you dismiss their points as them just being "upset" over aesthetics. > I am sure you could certainly understand that there are reasons why > some people build certain habits and that not everyone gets upset > about the same kinds of "secondary accidents" (as Aristotle would put > it). Some people may have a hard time even noticing what upsets you. Most of the time when asking for help from people with a lot more knowledge than me in a given subject, I tend to assume that if I find something they say to be oddly picky or unnecessarily strict then it's most likely because of a gap in my own knowledge, rather than them trying to make me slaves to their own personal style. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: can you parse and "tail" at once? (and if you can't why not?)
OK, Greg's suggestion once again "made my day". I know at some point I will have to code everything in some programming language, but for now I will just get things done as quickly as possible. Also, Greg, please, I would like for you to understand that it is not my intention to upset you about what you find visually upsetting. We have talked about that before. I am sure you could certainly understand that there are reasons why some people build certain habits and that not everyone gets upset about the same kinds of "secondary accidents" (as Aristotle would put it). Some people may have a hard time even noticing what upsets you. lbrtchx
Re: can you parse and "tail" at once? (and if you can't why not?)
Hello, On Sun, Oct 22, 2023 at 04:33:12PM +, Albretch Mueller wrote: > After generating a file with lines of text, I have to: > 1) parse some of those lines based on a pattern I know (no regex > necessary, just a FS path) > 2) once parsed from those lines I need the last n characters only There's a large number of ways to achieve what you want, but it's hard to believe that simply getting the suffix is all that you want. I think it's likely that you'll reject most of the otherwise correct answers because they don't suit what you're about to do next. So you might consider telling us what you will do next with the suffix of each line. Thanks, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: can you parse and "tail" at once? (and if you can't why not?)
On Sun, Oct 22, 2023 at 04:33:12PM +, Albretch Mueller wrote:
> After generating a file with lines of text, I have to:
> 1) parse some of those lines based on a pattern I know (no regex
> necessary, just a FS path)
> 2) once parsed from those lines I need the last n characters only
> I am trying to use a one liner like:
> cat "${IFL}" | grep "${DESC_DIR}" | tail -c+$_DESC_DIR
> but this one liner repeats the output and the tail
>
The way that -c operates for tail is based on 'each file' provided as
input:
-c, --bytes=[+]NUM
output the last NUM bytes; or use -c +NUM to output starting with
byte NUM of
each file
However, when piping input in the way you are above, the entire output
of grep is taken as one file and fed into the input of tail. You could
instead loop over the lines coming out of grep, but that feels somewhat
ugly.
Essentially, your pipeline probably needs to be something more like
this:
sed -r -e "s+.*${DESC_DIR}++" "${IFL}" | tail
Or drop the tail entirely if what you really want is the last part of
each line, regardless of how many there are.
Regards,
-Roberto
--
Roberto C. Sánchez
Re: can you parse and "tail" at once? (and if you can't why not?)
On Sun, Oct 22, 2023 at 04:33:12PM +, Albretch Mueller wrote:
> I am trying to use a one liner like:
> cat "${IFL}" | grep "${DESC_DIR}" | tail -c+$_DESC_DIR
> but this one liner repeats the output and the tail
Brevity is not the measure of goodness. When you write a shell script,
it's *OK* to use more than one line.
Also, please stop using ALL_CAPS variable names for internal script
variables. ALL_CAPS names are by convention reserved for the environment.
Names with a leading underscore are allowed, but should be used only
when you need something special. There's nothing special in the code
above to merit a variable name like _DESC_DIR.
Also also, useless use of cat .
Finally, mixing up _DESC_DIR and DESC_DIR is probably not helping you.
> Say the lines of text in IFL are:
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Uppsala_Algebra/45_16.3_45MEB5h5H9Y.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Uppsala_Algebra/46_17.1_LE_gl5RcDnY.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/17_9jKzwoBGFs8.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/18_plOUIOo91lI.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/19_TJ7yAiq8gEw.description
> [info] 123XYZ:
> /media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/20_UMBEgb14uqw.description
>
> I know the prefix:
> [info] 123XYZ: /media/user/1234qwer/Algebraic_Geometry04/20231022040239/
>
> I need as output:
>
> Uppsala_Algebra/45_16.3_45MEB5h5H9Y
> Uppsala_Algebra/46_17.1_LE_gl5RcDnY
> Göttsche/17_9jKzwoBGFs8
> Göttsche/18_plOUIOo91lI
> Göttsche/19_TJ7yAiq8gEw
> Göttsche/20_UMBEgb14uqw
Yay! Sample input! That makes it nice and easy.
#!/bin/bash
inputfile=${1-/whatever}
prefix='[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/'
while IFS= read -r line; do
printf '%s\n' "${line#"$prefix"}"
done < <(grep -F -- "$prefix" "$inputfile")
Yes, I could make it shorter. I could even use that functional approach
where you pipe grep and sed together, which some people think is the only
way to write shell code. It could actually be done in a single sed
command (but you'd have to convert the prefix string to a regular
expression first).
I chose not to do any of those, for two reasons:
1) To make it easier to read and understand.
2) To make it possible to do more than simply *printing* the suffixes.
If your script wants to do something with those suffixes, more than
simply printing them on the screen, then the script I wrote is a much
better starting point. You can stick additional commands inside the
loop body to do whatever you need.
On Sun, Oct 22, 2023 at 11:07:46AM -0600, Charles Curley wrote:
> Bash has built in operators to extract this sort of thing. Greg
> Wooledge will know what they are called. Something like:
>
> fspec="/exp/home1/abc.txt"
> filename="${fspec##*/}" # get filename
> dirname="${fspec%/*}" # get directory/path name
>
> That doesn't do exactly what you want, but comes close.
Those are called Parameter Expansions. For a beginner-friendly intro,
please see .
Re: can you parse and "tail" at once? (and if you can't why not?)
On Sun, 22 Oct 2023 16:33:12 +
Albretch Mueller wrote:
> After generating a file with lines of text, I have to:
> 1) parse some of those lines based on a pattern I know (no regex
> necessary, just a FS path)
> 2) once parsed from those lines I need the last n characters only
> I am trying to use a one liner like:
> cat "${IFL}" | grep "${DESC_DIR}" | tail -c+$_DESC_DIR
> but this one liner repeats the output and the tail
Bash has built in operators to extract this sort of thing. Greg
Wooledge will know what they are called. Something like:
fspec="/exp/home1/abc.txt"
filename="${fspec##*/}" # get filename
dirname="${fspec%/*}" # get directory/path name
That doesn't do exactly what you want, but comes close.
--
Does anybody read signatures any more?
https://charlescurley.com
https://charlescurley.com/blog/
can you parse and "tail" at once? (and if you can't why not?)
After generating a file with lines of text, I have to:
1) parse some of those lines based on a pattern I know (no regex
necessary, just a FS path)
2) once parsed from those lines I need the last n characters only
I am trying to use a one liner like:
cat "${IFL}" | grep "${DESC_DIR}" | tail -c+$_DESC_DIR
but this one liner repeats the output and the tail
Say the lines of text in IFL are:
[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/Uppsala_Algebra/45_16.3_45MEB5h5H9Y.description
[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/Uppsala_Algebra/46_17.1_LE_gl5RcDnY.description
[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/17_9jKzwoBGFs8.description
[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/18_plOUIOo91lI.description
[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/19_TJ7yAiq8gEw.description
[info] 123XYZ:
/media/user/1234qwer/Algebraic_Geometry04/20231022040239/Göttsche/20_UMBEgb14uqw.description
I know the prefix:
[info] 123XYZ: /media/user/1234qwer/Algebraic_Geometry04/20231022040239/
I need as output:
Uppsala_Algebra/45_16.3_45MEB5h5H9Y
Uppsala_Algebra/46_17.1_LE_gl5RcDnY
Göttsche/17_9jKzwoBGFs8
Göttsche/18_plOUIOo91lI
Göttsche/19_TJ7yAiq8gEw
Göttsche/20_UMBEgb14uqw
~
lbrtchx
