bash using unknown tmp - library based? static link on linux? (was Re: bash not using pipes or /tmp @ boot?)
Chet Ramey wrote: On 10/6/14, 6:03 PM, Linda Walsh wrote: Not sure how but this went off into space, sorta... Greg Wooledge wrote: On Mon, Oct 06, 2014 at 12:14:57PM -0700, Linda Walsh wrote: done <<<"$(get_net_IFnames_hwaddrs)" Where am I using a HERE doc? <<< and << both create temporary files. Yeah... where? Wherever $TMPDIR says to. If $TMPDIR doesn't name a writable directory, bash looks for various system definitions (P_tmpdir), finally defaulting to /tmp or /var/tmp. P_tmpdir? in ENV? I don't have TMP or TMPDIR in my env during normal runtime and tmp's were put in /tmp... I don't know where they were going @ boot time, as /tmp and /var/tmp were writeable, but still TMP/TMPDIR were unset. (anyway -- went back to the process substitution) since that actually worked -- when trying to debug it earlier, it was thought that might be the problem... but it *seemed* to be a local redefinition of a global -- which actually doesn't make complete sense, as the script worked interactively. Hmmm...oh well... even if /var/tmp and /tmp were not mounted, bash running as root could still create files in it so not sure what the problem was. Maybe if bash didn't create tmp files in non-standard locations, this never would have been an issue (or just use pipes).. Please. Your assumptions about what is going on make your conclusions shaky at best. Um...assumptions were that tmp files weren't getting created in standard locations of /tmp or /var/tmp for unknown reasons. Interactively and @ run time, it works fine... it's probably some library bash is linking with... speaking of which ... does anyone have static link working on linux these days, or is the static link break a suse-special? I wanted a safe shell to put on root, since the way things are going, I'm expecting bash to be the next piece of software that has to be merged with systemd...
Re: "export -p" output not suitable for input when env vars contain non-identifiers
On 10/7/14, 12:28 PM, Stephane Chazelas wrote: > Since those environment variables cannot be mapped to shell > variables, they should not be included in the output of "export > -p". Thanks. This will be fixed in the next release of bash. > I can see there are a number of bugs at savannah > (https://savannah.gnu.org/support/?group=bash), but not much > activity on those bugs. Is that in use? Is it worth me adding an > entry there? I don't look there very often. This mailing list is the best way to report bugs. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Bash-4.3 Official Patch 30
On 10/7/14, 6:31 PM, Ángel González wrote: >>> Changing it to return 0 instead of '\0' would probably be more clear. >>> No need to return a pointer to a static empty string. >> >> It depends on how you want the function to work. It is nice to >> differentiate between the cases where there is no shell input line >> at all, where the index is just wrong, and the actual current input >> pointer. A "" indicates the second case better than a 0, though it's >> not perfect. >> >> Chet > > > Note it **is** returning NULL, not "" as it seemed implied by earlier > messages. Only verified gcc (4.9.1) behavior, but that's also what I > expected from the C code: '\0' (char) promoted to 0 (int), then to > NULL (char*) Yes, that's right. The question is whether that case of the index out of bounds should continue to return NULL, but in a clearer way, or return "" to differentiate it from the case where shell_input_line is NULL. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash uses tmp files for inter-process communication instead of pipes?
On 10/7/14, 5:35 PM, Linda Walsh wrote: > > > Chet Ramey wrote: >> On 10/7/14, 1:45 PM, Linda Walsh wrote: >> >>> There is no fundamental reason why, say, process substitution needs to >>> use /dev/fd or /proc/anything -- and couldn't operate exactly like piped >>> processes do now. On my first implementation of multiple IPC programs, >>> I've used semaphores, message queues, named pipes, and shared memory. >>> With the exception of using shared memory for mutiple megabytes of >>> memory for each of several children, pipes provided equal or better >>> performance. >> >> I think you have a fundamental misunderstanding of the purpose of >> process substitution. It is a way to turn a pipe to another process >> into a file name, so that you can open what looks like a file, > > You mean "< <(process)" is the part you are saying looks like a file? > > If it wasn't for the shell mangling eoln's why wouldn't <$(process) be the > same? OK. Let's take a step back. <(process) expands to a file name like /dev/fd/26. When you open that file for reading, you can read the output of the process. If you open it for writing, using the >(process) syntax, you can write to the process's stdin. This can be asynchronous, not synchronous like pipes, and, since it can be combined with `exec' and a redirection, you can save the file descriptor for later use. In other words, you can treat dynamically-generated data like a file. That filename isn't private to the shell; it can be used as an argument to another program. $(process) expands directly to the output of `process'. Instead of being able to read and process that data, it appears on the command line as an expanded word. You could make `process' output a filename that the redirection opens, but that's not the same thing as dynamically generating the data. > Or are you saying that the shell not mangling eoln's is a way of making it > look like a filei that you can use the normal set of !!!WRITE!!!... > calls I can write to such a process? This doesn't make any sense. > > But I never got that < <(process) was anything more than a one way read > from the > right-hand process into the left -- and was not any different than > (process)| where left hand process feeds into the right hand process. > > I.e. both looked like unidirectional communication over a pipe. They are superficially simimlar. The < redirection is what makes it a `one-way read'. That's not the only thing you can do with a process substitution. > Both being cases where you can write to a process through a pipe. I've > used FIFO's but other than making it a bit easier to parse lines, they > weren't much different than pipes -- which get parsed into lines all the time > over the net et al. Pipes are synchronous and anonymous. Process substitution creates a usable filename that you can use to communicate asynchronously with a process. They can be made to do the same thing, but are different. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Bash-4.3 Official Patch 30
Chet Ramey wrote:
> On 10/6/14, 6:16 PM, John E. Malmberg wrote:
>
> >>> Do you mean return ""; ?
> >>
> >> Yes, good catch. It doesn't make a difference: clang and gcc both accept
> >> it as written and it behaves as desired. However, I'll change it for the
> >> next version.
> >
> > Changing it to return 0 instead of '\0' would probably be more clear.
> > No need to return a pointer to a static empty string.
>
> It depends on how you want the function to work. It is nice to
> differentiate between the cases where there is no shell input line
> at all, where the index is just wrong, and the actual current input
> pointer. A "" indicates the second case better than a 0, though it's
> not perfect.
>
> Chet
Note it **is** returning NULL, not "" as it seemed implied by earlier
messages. Only verified gcc (4.9.1) behavior, but that's also what I
expected from the C code: '\0' (char) promoted to 0 (int), then to
NULL (char*)
Regards
> #include
>
> char *
> foo()
> {
> return '\0';
> }
>
> int main()
> {
> printf("The value is: %p\n", foo());
> return 0;
> }
Output:
> (nil)
Re: bash uses tmp files for inter-process communication instead of pipes?
Pierre Gaston wrote:
That's where you are wrong, there is no reason for *your* use case, but the
basic idea behind process substitution is to be able to use a pipe in a
place where you normally need a file name.
Well, that's not what I needed it for. I needed to read from a child
process that
changed directories.
How can you say there is no reason for my use case...
It's not thoroughly polished yet, but it does work. It can rename and
reorder your
network interfaces on boot so you can keep them with consistent names and
ordering -- something systemd doesn't seem to support. Furthermore, it
doesn't depend on any suse library code -- which they've been putting in
hooks and traps for those not using systemd. Supposedly people not using
systemd won't be able to boot their systems.
This is one of my side projects to keep my system booting and to make
available
to anyone who might want to use it. But you call that no reason?
Sigh.
IF you want to see what I was doing in context, its attached.
It's a stand alone script to tell you if your interfaces are in your
desired order, reorder them if you wish, and act as a startup/shutdown
script
to fix things each boot. I haven't spent any time in cleanup, but it's
still
not bad code. See if the code is not readable -- even for shell!
#!/bin/bash
#boot.assign_network_names
### BEGIN INIT INFO
# Provides: net-devices
# Required-Start: boot.udev boot.device-mapper boot.localfs
# Required-Stop: $null
# Default-Start: B
# Default-Stop:
# Short-Description: reorder and rename net devices
# Description: reorder and rename net devs if needed
### END INIT INFO
#
# assign network names as rc-script
# L A Walsh, (free to use/modify/distribute to nice people) (c) 2013-2014
#
#include standard template:
_prgpth=${0:?}; _prgpth=${_prgpth#-} _prg=${_prgpth##*/};
_prgdr=${_prgpth%/$_prg}
[[ -z $_prgdr || $_prg == $_prgdr ]] && _prgdr="$PWD"
#if ! typeset -f include >&/dev/null ;then
# source ${_LOCAL_DIR:=/etc/local}/bash_env.sh;
#fi
export PATH="/etc/local/bin:/etc/local/lib:$PATH"
export
PS4='>>${BASH_SOURCE:+${BASH_SOURCE[0]}}#${LINENO}${FUNCNAME:+(${FUNCNAME[0]})}>
'
#include stdalias (needed entries included "inline", below)
shopt -s extglob expand_aliases
alias dcl=declare sub=function
alias int=dcl\ -i map=dcl\ -A
hash=dcl\ -Aarray=dcl\ -a
alias lower=dcl\ -l upper=dcl\ -u
string=dcl my=dcl
alias map2int=dcl\ -Ai intArray=dcl\ -ia
cfg_fn="/etc/sysconfig/assign_netif_names"
[[ -f $cfg_fn ]] || {
echo "Cannot find required config file: $cfg_fn"; exit 1; }
# expected variables to be read in
array needed_drivers
array Linknames
array Default
map if2hw
. $cfg_fn || { echo "Error in reading config file $cfg_fn"; exit 2; }
DfltRE="^+(${Default_good[@]})$"
DfltRE=${DfltRE// /|}
declare -a down_ln=()
for ln in "${Linknames[@]}"; do
[[ $DfltRE =~ $ln ]] && continue;
down_ln+=($ln)
done
map hw2if
for intf in "${!if2hw[@]}" ; do
addr=${if2hw[$intf]}
hw2if[$addr]="$intf"
done
# end read config ##
#include rc.status -- essential funcs included below:
int rc_status=0
sub rc_reset { rc_status=0; }
sub rc_status {
rc_status=$?;
if ((rc_status)) && { (($#)) && [[ $1 = -v ]] ; }; then
echo "Abnormal rc_status was $rc_status."
elif (($#)) && [[ $1 = -v ]] ; then
echo "rc_status: ok"
fi
}
sub rc_exit {
rc_status=$?;
rc_status
exit $rc_status
}
sub warn () { local msg="Warning: ${1:-"general"}"
printf "%s" >&2 "$msg"
}
sub die () { int stat=$?; local msg="Error. ${1:-"unknown"}"
echo "$msg (errno=$stat)" >&2
(exit $stat);
#sets $? (doesn't exit inside ()
rc_status -v
rc_exit
exit $stat
# exit unless die called in
subshell
}
sysfs=/sys
sysnet=$sysfs/class/net
[[ -d $sysfs && -d $sysnet ]] ||
sysnet=$sysfs/class/net
sys_modules=$sysfs/module
if [[ -z $(type -P modprobe) ]]; then # verify find of modprobe
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:$PATH
fi
if [[ -n $(type -P modprobe) ]]; then # if found, alias it
alias modprobe="$(type -P modprobe)"
else
# only throw
error if needed
alias modprobe="die 'cannot load required modules'"
fi
if [[ -z $(type -P ip ) ]]; then# ip is ne
Re: bash uses tmp files for inter-process communication instead of pipes?
Chet Ramey wrote: On 10/7/14, 1:45 PM, Linda Walsh wrote: There is no fundamental reason why, say, process substitution needs to use /dev/fd or /proc/anything -- and couldn't operate exactly like piped processes do now. On my first implementation of multiple IPC programs, I've used semaphores, message queues, named pipes, and shared memory. With the exception of using shared memory for mutiple megabytes of memory for each of several children, pipes provided equal or better performance. I think you have a fundamental misunderstanding of the purpose of process substitution. It is a way to turn a pipe to another process into a file name, so that you can open what looks like a file, You mean "< <(process)" is the part you are saying looks like a file? If it wasn't for the shell mangling eoln's why wouldn't <$(process) be the same? Or are you saying that the shell not mangling eoln's is a way of making it look like a filei that you can use the normal set of !!!WRITE!!!... calls I can write to such a process? I've never seen this documented where did I miss this... if that's the case, then I'll rethink my position. But I never got that < <(process) was anything more than a one way read from the right-hand process into the left -- and was not any different than (process)| where left hand process feeds into the right hand process. I.e. both looked like unidirectional communication over a pipe. But if < <(process) has bidirectional communication, do you have an example of this? Ahhh... you are referring to > >(process) which is like |(process) Both being cases where you can write to a process through a pipe. I've used FIFO's but other than making it a bit easier to parse lines, they weren't much different than pipes -- which get parsed into lines all the time over the net et al. What read/write calls can you do with < <() > >() that you cannot with "|"' that don't involve a "which process is on top" paradigm (which, as you mention, is switchable with "lastpipe" anyway)? Gosh, you mean I could write a stdio library that would have to be portable over the whole set of systems on which bash currently runs and support it forever, abandoning all the work vendors have done? It sounds too good to be true! I can let you use mine but it's specific to my use case and partly in c++ which I'm learning as I work on something. But the specific to use case is also the point. You don't need the whole set to buffer IPC in memory. I generally prefer my own code and utils because their interfaces are relatively stable compared to my needs -- BUT MOST of all -- because if something is broken, since I know the code, I have a reasonable chance of fixing it. That's harder to say picking up other people's stuff -- IF that is even an option. FWIW -- I went back to using process substitution because I don't know where the tmpfiles are opening (TMP and TMPFILE are not set, and /tmp and /var/tmp are/were both writeable). Now it is working. None of the problems had to do with any of the IPC methods but with what appeared to be a secondary declaration of the global variables in a sub... hiding the globals. I couldn't see what I was doing that would have caused problems since both dev and proc were mounted and tmp is writeable but that's what tmp error message seemed to think. Even though it's a startup script, it's not starting up the kernel -- its renaming and reordering network interfaces to suit the user's choice. Responses on other stuff later...
Re: bash uses tmp files for inter-process communication instead of pipes?
On 10/7/14, 1:45 PM, Linda Walsh wrote: > There is no fundamental reason why, say, process substitution needs to > use /dev/fd or /proc/anything -- and couldn't operate exactly like piped > processes do now. On my first implementation of multiple IPC programs, > I've used semaphores, message queues, named pipes, and shared memory. > With the exception of using shared memory for mutiple megabytes of > memory for each of several children, pipes provided equal or better > performance. I think you have a fundamental misunderstanding of the purpose of process substitution. It is a way to turn a pipe to another process into a file name, so that you can open what looks like a file, get a file descriptor, use the normal set of read and write calls on it, and all the while talk to another process. There are only a few ways to do that, and /dev/fd and named pipes are really the only viable ones. > It was certainly fun to try the various methods and compare them, but > having an implementation that only depends on the LCD (pipes in this > case), would make the code more portable and faster. For HEREDOC's > there is no reason why chet couldn't use his own STDIO lib for the needs > of bash that keep everything in memory or spill on extra large sizes. Gosh, you mean I could write a stdio library that would have to be portable over the whole set of systems on which bash currently runs and support it forever, abandoning all the work vendors have done? It sounds too good to be true! > Design choice doesn't have to become feature constraint -- or if it > does, then it's time to think about a different design choice. Then all > of those features that currently use linux-only or linux-like extensions > like proc and /dev/fd can be ported to platforms that don't support such > a wide feature set. /dev/fd (which I prefer to /proc) is not Linux specific. Named pipes are standard. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash uses tmp files for inter-process communication instead of pipes?
On Tue, Oct 7, 2014 at 8:45 PM, Linda Walsh wrote:
>
>
> Greg Wooledge wrote:
>
>> OK, then use a function to give you an escapable block:
>>
>> declare -A ...
>> create_maps() {
>> cd "$sysnet" || return
>> for ifname in ...; do
>> hwaddr=$(<"$ifname"/address)
>> act_hw2if[$hwaddr]="$ifname"
>> act_hw2if[$ifname]="$hwaddr"
>> done
>> }
>> create_maps
>>
>> Either way, they code as you have suggested won't work
>>> without overcoming another set of side effects.
>>>
>>
>> There are ways to NOT use subshells. I have given you two of those
>> ways now.
>>
>
>
> I appreciate the options, but the option I want is the parent
> staying
> "put", and only sending children out to change dir's. Say some new
> version of init wants to isolate init processes by putting them in their
> own dir and then deleting the dir. As long as they don't cd out of the
> dir, they are fine, but if they do, they can't get back to it.
>
> Why can't I spawn a child that cd's into a dir and reports back
> what
> it finds in the dir? I do this in perl or C w/no problem.
>
> I have an iomonitor program, that monitors multiple data inputs. For
> each, I spawn a child that sits in that directory, holding the FD open
> and rewinding it for updated status. There's no constant
> opening/reopening of files -- no walking paths. Each time you open
> a path, the kernel has to walk the path -- if cached, happens very
> quickly -- but still work as it also has to check access on each
> traversal (it can change). If your data-gathering routines sit where
> the data is and don't move, there is no waisted I/O or CPU accessing the
> data and the parent gets updates via pipes.
>
> There is no fundamental reason why, say, process substitution needs to
> use /dev/fd or /proc/anything -- and couldn't operate exactly like piped
> processes do now. On my first implementation of multiple IPC programs,
> I've used semaphores, message queues, named pipes, and shared memory.
>
That's where you are wrong, there is no reason for *your* use case, but the
basic idea behind process substitution is to be able to use a pipe in a
place where you normally need a file name.
Re: bash uses tmp files for inter-process communication instead of pipes?
On Tue, Oct 07, 2014 at 10:45:53AM -0700, Linda Walsh wrote: > I appreciate the options, but the option I want is the parent staying > "put", and only sending children out to change dir's. Say some new > version of init wants to isolate init processes by putting them in their > own dir and then deleting the dir. As long as they don't cd out of the > dir, they are fine, but if they do, they can't get back to it. > > Why can't I spawn a child that cd's into a dir and reports back what > it finds in the dir? I do this in perl or C w/no problem. You are adding layers of non-negligible complexity to work around an imaginary problem.
Re: bash uses tmp files for inter-process communication instead of pipes?
Greg Wooledge wrote:
OK, then use a function to give you an escapable block:
declare -A ...
create_maps() {
cd "$sysnet" || return
for ifname in ...; do
hwaddr=$(<"$ifname"/address)
act_hw2if[$hwaddr]="$ifname"
act_hw2if[$ifname]="$hwaddr"
done
}
create_maps
Either way, they code as you have suggested won't work
without overcoming another set of side effects.
There are ways to NOT use subshells. I have given you two of those
ways now.
I appreciate the options, but the option I want is the parent staying
"put", and only sending children out to change dir's. Say some new
version of init wants to isolate init processes by putting them in their
own dir and then deleting the dir. As long as they don't cd out of the
dir, they are fine, but if they do, they can't get back to it.
Why can't I spawn a child that cd's into a dir and reports back what
it finds in the dir? I do this in perl or C w/no problem.
I have an iomonitor program, that monitors multiple data inputs. For
each, I spawn a child that sits in that directory, holding the FD open
and rewinding it for updated status. There's no constant
opening/reopening of files -- no walking paths. Each time you open
a path, the kernel has to walk the path -- if cached, happens very
quickly -- but still work as it also has to check access on each
traversal (it can change). If your data-gathering routines sit where
the data is and don't move, there is no waisted I/O or CPU accessing the
data and the parent gets updates via pipes.
There is no fundamental reason why, say, process substitution needs to
use /dev/fd or /proc/anything -- and couldn't operate exactly like piped
processes do now. On my first implementation of multiple IPC programs,
I've used semaphores, message queues, named pipes, and shared memory.
With the exception of using shared memory for mutiple megabytes of
memory for each of several children, pipes provided equal or better
performance.
It was certainly fun to try the various methods and compare them, but
having an implementation that only depends on the LCD (pipes in this
case), would make the code more portable and faster. For HEREDOC's
there is no reason why chet couldn't use his own STDIO lib for the needs
of bash that keep everything in memory or spill on extra large sizes.
Design choice doesn't have to become feature constraint -- or if it
does, then it's time to think about a different design choice. Then all
of those features that currently use linux-only or linux-like extensions
like proc and /dev/fd can be ported to platforms that don't support such
a wide feature set.
Re: bash uses tmp files for inter-process communication instead of pipes?
On Tue, Oct 07, 2014 at 09:54:21AM -0700, Linda Walsh wrote:
> Chet Ramey wrote:
> >It's syslog. Some vendors integrated patches that log all shell commands
> >to syslog.
> >
> No... that wasn't syslog, it was strace in another terminal where I attached
> the bacsh that was doing the various types of data transfer (pipes, process
> subst)
> assignment... etc)
Chet is saying that your vendor's version of Bash may have been patched
to make client-side calls to syslog(2), and this might be what you are
seeing in your strace.
If it turns out this is what's causing your problem, then you'll have
some tough choices to make.
> Um... it used a socket.. to transfer *i^Ht^H*SOMETHING,
> then it uses a tmp file on top of that:
>
> rt_sigaction(SIGINT, {0x4320b1, [], SA_RESTORER|SA_RESTART, 0x30020358d0},
> {0x4320b1, [], SA_RESTORER|SA_RESTART, 0x30020358d0}, 8) = 0
> open("/tmp/sh-thd-110678907923", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3
> write(3, "one two three four", 18) = 18
> write(3, "\n", 1) = 1
> open("/tmp/sh-thd-110678907923", O_RDONLY) = 4
> close(3)= 0
> unlink("/tmp/sh-thd-110678907923") = 0
The opens & writes & close & unlink are straight out of redir.c function
here_document_to_fd. So that's bash opening the temp file for the
here document. Any socket stuff before that could be from a vendor
patch, or a child process, or something else other than vanilla bash.
> I have neither TMP nor TMPDIR set at this point. I don't
> think TMP or TMPDIR is set in the runscript at boot, but the script could
> be inheriting
> a value from the bootup-infrastructure environment.
You could always add some debugging lines to the script to find out.
> the script in question is "assign_netif_names" run fairly late.
> But if I Break in on 'B', I get control before S01boot. If I run the
> command
> manually at that point, **it works**. Only when I'm letting it boot
> normally, does
> it hit this problem.
Ouch.
> >>>/etc/init.d/boot.assign_netif_names#193(get_net_IFnames_hwaddrs)> echo
> >>>eth5 a0:36:9f:15:c9:c2
> /etc/init.d/boot.assign_netif_names: line 204: cannot create temp file for
> here-document: No such file or directory
This error message definitely indicates the failure of bash internal
function here_document_to_fd (in redir.c). The question is exactly
what failed, and why.
I've given you a workaround for the immediate problem you're seeing,
so you could use that until you are able to diagnose the underlying
problem and find out why this particular feature is failing.
To reiterate (and expanding a bit, because I omitted a few details
in the previous message):
declare -A map1 map2 # etc.
here=$PWD
create_maps() {
cd /some/place || return
for ifname in ...; do
hwaddr=$(
Re: bash uses tmp files for inter-process communication instead of pipes?
Chet Ramey wrote:
On 10/7/14, 8:55 AM, Dan Douglas wrote:
Um... it used a socket.. to transfer it, then it uses a tmp file on top
of that?! :
That's the wrong process. AF_NETLINK is the interface used by iproute2 for IPC
with the kernel. Bash doesn't use netlink sockets.
It's syslog. Some vendors integrated patches that log all shell commands
to syslog.
Chet
No... that wasn't syslog, it was strace in another terminal where I attached
the bacsh that was doing the various types of data transfer (pipes, process
subst)
assignment... etc)
strace -p 48785 -ff
Process 48785 attached
read(0, "\r", 1)= 1
write(2, "\n", 1) = 1
socket(PF_NETLINK, SOCK_RAW, 9) = 3
sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=},
msg_iov(2)=[{"*\0\0\0d\4\1\0\0\0\0\0\0\0\0\0", 16}, {"read a b c d
<<<${arr[@]}\0", 26}], msg_controllen=0, msg_flags=0}, 0) = 42
close(3)= 0
-
Um... it used a socket.. to transfer *i^Ht^H*SOMETHING,
then it uses a tmp file on top of that:
rt_sigaction(SIGINT, {0x4320b1, [], SA_RESTORER|SA_RESTART, 0x30020358d0},
{0x4320b1, [], SA_RESTORER|SA_RESTART, 0x30020358d0}, 8) = 0
open("/tmp/sh-thd-110678907923", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3
write(3, "one two three four", 18) = 18
write(3, "\n", 1) = 1
open("/tmp/sh-thd-110678907923", O_RDONLY) = 4
close(3)= 0
unlink("/tmp/sh-thd-110678907923") = 0
fcntl(0, F_GETFD) = 0
fcntl(0, F_DUPFD, 10) = 10
fcntl(0, F_GETFD) = 0
fcntl(10, F_SETFD, FD_CLOEXEC) = 0
dup2(4, 0) = 0
close(4)= 0
Bash may not use netlink sockets directly, but perhaps that was a weird looking
call
to 'nss' to do some name or group lookup... but the presence of the read text as
part of one of the messages seems a bit peculiar.
Of NOTE:
straces I ran with the system up in normal RL3 -- that is where I see it writing
to tmp. I have neither TMP nor TMPDIR set at this point. I don't
think TMP or TMPDIR is set in the runscript at boot, but the script could be
inheriting
a value from the bootup-infrastructure environment. I would have expected them
to write into /tmp -- which was why in one of the listings I did the "du" on
both
/tmp and /var/tmp to show they were mounted and did a test "write" (touch) on
a file in /tmp and verified I was able to do so and see it with "ls".
If TMP{DIR} are set to a value, it isn't "/tmp or /var/tmp" since both are up
and
available at that point.
The problem is that if I manually break into the boot process (this system goes
through a 'B' state to do these things:
/etc/rc.d/boot.d/S01boot.sysctl@
/etc/rc.d/boot.d/S01boot.usr-mount@
/etc/rc.d/boot.d/S02boot.udev@
/etc/rc.d/boot.d/S04boot.lvm@
/etc/rc.d/boot.d/S05boot.clock@
/etc/rc.d/boot.d/S06boot.localnet@
/etc/rc.d/boot.d/S07boot.cgroup@
/etc/rc.d/boot.d/S11boot.localfs@
/etc/rc.d/boot.d/S13boot.klog@
/etc/rc.d/boot.d/S13boot.lvm_monitor@
/etc/rc.d/boot.d/S13boot.swap@
/etc/rc.d/boot.d/S13setserial@
/etc/rc.d/boot.d/S14boot.isapnp@
/etc/rc.d/boot.d/S14boot.ldconfig@
/etc/rc.d/boot.d/S50boot.assign_netif_names@
/etc/rc.d/boot.d/S50boot.sys_settings@
/etc/rc.d/boot.d/S50boot.sysstat@
the script in question is "assign_netif_names" run fairly late.
But if I Break in on 'B', I get control before S01boot. If I run the command
manually at that point, **it works**. Only when I'm letting it boot normally,
does
it hit this problem.
During a normal boot some of these are happening in parallel, however, that was
why I tested the writability of /tmp (verified it to be so). I'll print out
TMP and TMPDIR -- and see if they contain unexpected values, but my assumption
has
been they are empty then as they are after boot, with /tmp taking the tmp file
(as
was the case in the strace I ran w/system up).
The messages I see, record and transmit are the boot logs -- not trace logs, and
show the kernel boot followed by a switchover (when klog is called) from the
kernel format to a more user-ish format i.e:
[4.058222] rtc_cmos 00:01: RTC can wake from S4
[4.058567] rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0
[4.058607] rtc_cmos 00:01: alarms up to one day, y3k, 242 bytes nvram, hpet
irqs
[4.058772] device-mapper: uevent: version 1.0.3
[4.059034] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised...
.root fs is next:
(> mount |grep sdc1
/dev/sdc1 on / type xfs (rw,nodiratime,relatime,attr2,inode64,noquota))
[4.356688] XFS (sdc1): Mounting V4 Filesystem
[4.423530] usb 1-3.1: new low-speed USB device number 3 using ehci-pci
[4.447749] XFS (sdc1): Ending clean mount
[4.451993] VFS: Mounted root (xfs filesystem) on device 8:33.
[4.461451] devtmpfs: mounted
... last kernel USB probe.. then a 15-second ga
"export -p" output not suitable for input when env vars contain non-identifiers
2014-09-30 17:06:22 +0100, Stephane Chazelas: [...] > $ env -i $'a\necho test\na=b' bash -c 'export -p' > declare -x OLDPWD > declare -x PWD="/home/stephane" > declare -x SHLVL="1" > declare -x a > echo test > a [...] Just reiterating in case that had gone unnoticed earlier: $ env 'a;echo OOPS;: =' bash -c 'export -p' | bash OOPS That's because bash outputs: declare -x a;echo OOPS;: Since those environment variables cannot be mapped to shell variables, they should not be included in the output of "export -p". Same applies for env variables called "1" or "#" or "*" for instance. I had already reported that 5 years ago actually: http://thread.gmane.org/gmane.comp.shells.bash.bugs/11306 and it doesn't look like it was addressed at the time. I can see there are a number of bugs at savannah (https://savannah.gnu.org/support/?group=bash), but not much activity on those bugs. Is that in use? Is it worth me adding an entry there? -- Stephane
Re: bash uses tmp files for inter-process communication instead of pipes?
On 10/7/14, 8:55 AM, Dan Douglas wrote: >> Um... it used a socket.. to transfer it, then it uses a tmp file on top >> of that?! : > > That's the wrong process. AF_NETLINK is the interface used by iproute2 for > IPC > with the kernel. Bash doesn't use netlink sockets. It's syslog. Some vendors integrated patches that log all shell commands to syslog. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Bash-4.3 Official Patch 30
On 10/6/14, 6:16 PM, John E. Malmberg wrote: >>> Do you mean return ""; ? >> >> Yes, good catch. It doesn't make a difference: clang and gcc both accept >> it as written and it behaves as desired. However, I'll change it for the >> next version. > > Changing it to return 0 instead of '\0' would probably be more clear. > No need to return a pointer to a static empty string. It depends on how you want the function to work. It is nice to differentiate between the cases where there is no shell input line at all, where the index is just wrong, and the actual current input pointer. A "" indicates the second case better than a 0, though it's not perfect. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash uses tmp files for inter-process communication instead of pipes?
On 10/7/14, 2:07 AM, Linda Walsh wrote: > I thought I was getting rid of this bogus problem (which shouldn't > be a problem anyway -- since it's just another pipe but with parent receiving > the data instead of child receiving it) by storing it in a variable > transfer form > <<<($VAR)... cuz was told that didn't involve a voodoo process-substitution. I suppose it's voodoo because you didn't understand the underlying OS resource process substitution uses and therefore didn't realize when that resource is and is not available. People using process substitution using /dev/fd on a system like FreeBSD also encounter similar issues, since /dev/fd is a separate file system that must be mounted. > > But it involves a heredoc that even though /tmp and /tmp/var are writeable, > BASH can't use. Either it's writing to //network or trying to open a > tmp file in the current (/sys) directory... eitherway is buggy. You misunderstand what is going on, but now that you know here documents and here strings use temporary files, and that, as documented, bash uses $TMPDIR as the directory where it creates these temporary files, you can modify your script to set TMPDIR appropriately. > > But a heredoc is just reading from an in-memory buffer. That bash is > going the inefficient route and putting the buffer in "tmp", first, is > the 2nd problem you are pointing out -- why do people who want a heredoc > need a fork OR a tmpfile? The output I generated is < 512 bytes. Would it > be too silly to simply put any heredoc output write it to any file? shopt heredocsize=XXX[KMG]... and overflow to tmp > if needed. Maybe. At some point, however, you have to turn it into a file descriptor, because that is what the redirections produce. That constrains your set of choices. You could invent new syntax, but it would not be a here-document or here-string any more. > Having several that would work woudl be nice. Um... having 1 that would > work even... how do you turn around parent/child I/O and get the vars on the > other side without heredoc or procsub? Why no pipes? There are several mechanisms. You simply have to use them correctly. Since you're running a non-interactive shell without job control enabled, you could even use pipes and the `lastpipe' shell option, but that only first appeared in bash-4.2. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Fwd: bash not using pipes or /tmp @ boot?
On 10/6/14, 6:03 PM, Linda Walsh wrote: > Not sure how but this went off into space, sorta... > > Greg Wooledge wrote: >> On Mon, Oct 06, 2014 at 12:14:57PM -0700, Linda Walsh wrote: >>>done <<<"$(get_net_IFnames_hwaddrs)" >> >>> Where am I using a HERE doc? >> >> <<< and << both create temporary files. > > > Yeah... where? Wherever $TMPDIR says to. If $TMPDIR doesn't name a writable directory, bash looks for various system definitions (P_tmpdir), finally defaulting to /tmp or /var/tmp. > Maybe if bash didn't create tmp files in non-standard locations, > this never would have been an issue (or just use pipes).. Please. Your assumptions about what is going on make your conclusions shaky at best. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash uses tmp files for inter-process communication instead of pipes?
On 10/6/14, 5:00 PM, Linda Walsh wrote:
>> You are working in a severely constrained environment.
> That isn't the problem: the assignment using a tmp file is:
>> strace -p 48785 -ff
> Process 48785 attached
> read(0, "\r", 1)= 1
> write(2, "\n", 1) = 1
> socket(PF_NETLINK, SOCK_RAW, 9) = 3
> sendmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=},
> msg_iov(2)=[{"*\0\0\0d\4\1\0\0\0\0\0\0\0\0\0", 16}, {"read a b c d
> <<<${arr[@]}\0", 26}], msg_controllen=0, msg_flags=0}, 0) = 42
> close(3)= 0
> -
> Um... it used a socket.. to transfer it, then it uses a tmp file on top
> of that?! :
Slow down and read the trace, espcially the data being transferred. This
is syslog saving the command.
> rt_sigaction(SIGINT, {0x4320b1, [], SA_RESTORER|SA_RESTART, 0x30020358d0},
> {0x4320b1, [], SA_RESTORER|SA_RESTART, 0x30020358d0}, 8) = 0
> open("/tmp/sh-thd-110678907923", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3
> write(3, "one two three four", 18) = 18
> write(3, "\n", 1) = 1
> open("/tmp/sh-thd-110678907923", O_RDONLY) = 4
> close(3)= 0
> unlink("/tmp/sh-thd-110678907923") = 0
> fcntl(0, F_GETFD) = 0
> fcntl(0, F_DUPFD, 10) = 10
> fcntl(0, F_GETFD) = 0
> fcntl(10, F_SETFD, FD_CLOEXEC) = 0
> dup2(4, 0) = 0
> close(4)= 0
> ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS,
> 0x7fff85627820) = -1 ENOTTY (Inappropriate ioctl for device)
> lseek(0, 0, SEEK_CUR) = 0
> read(0, "one two three four\n", 128)= 19
> dup2(10, 0) = 0
> fcntl(10, F_GETFD) = 0x1 (flags FD_CLOEXEC)
> close(10) = 0
> -
>
> Why in gods name would it use a socket (still of arguable overhead, when
> it could be done in a local lib), but THEN it duplicates the i/o in a file?
Read the trace. There's no duplication going on. The shell uses temp
files to implement here documents -- an implementation choice you may
disagree with under your peculiar circumstances but one that is simple
and general.
>> Maybe the best solution here is to move your script to a different part
>> of the boot sequence. If you run it after all the local file systems
>> have been mounted, then you should be able to create temporary files,
>> which in turns means << and <<< become available, should you need them.
>
> Theoretically, they ARE mounted What I think may be happening
> is that $TMP is not set so it is trying to open the tmp dir in:
>
> "//sh-thd-183928392381" -- a network address.
This is not a network address. The pathname you've shown is consistent
with TMPDIR being set to a single slash (TMPDIR=/), which is a directory
writable by the current user.
You might try setting TMPDIR=/tmp in your startup script.
> ---
> Correctness before elegance. 1), use memory before OS services.
> 2) use in-memory services before file services, 3) Don't use uninitialized
> variables (TMP) -- verify that they are sane values before usage.
> 4) don't use network for tmp when /tmp or /var/tmp would have worked just
> fine.
This is basically word salad.
> This type of redirection instructs the shell to read input from the
>current source until a line containing only delimiter (with no trailing
>blanks) is seen. All of the lines read up to that point are then used
>as the standard input for a command.
>
> "The current source" -- can be anything that input can be
> redirected from -- including memory.
In your case, the current source is the script, where the data appears.
> How could you use a seek in the middle of a HERE doc
> read?
Think about your use cases. Many programs check whether or not their
input is a seekable device so they can read more than one character at
a time.
> Not only that, it's just wrong./dev/stdin most often comes from
> a tty which is not random access.
Again, it depends on your use cases. There are a number of programs
whose primary use is in a filter environment rather than directly by
a user at a terminal.
>>> Creating a tmp file to do an assignment, I assert is a bug.
You're still mixing a general capability with your use of that capability.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash uses tmp files for inter-process communication instead of pipes?
On 10/6/14, 3:38 PM, Linda Walsh wrote:
> Greg Wooledge wrote:
>> On Mon, Oct 06, 2014 at 12:14:57PM -0700, Linda Walsh wrote:
>>>done <<<"$(get_net_IFnames_hwaddrs)"
>>
>>> Where am I using a HERE doc?
>>
>> <<< and << both create temporary files.
>
>
> According to Chet , only way to do a multi-var assignment in bash is
>
> read a b c d <<<$(echo ${array[@]})
>
> Forcing a simple assignment into using a tmp file seems Machiavellian --
> as it does exactly the thing the user is trying to avoid through
> unexpected means.
Don't mix a capability -- shorthand to provide arbitrary data on stdin
to the shell or a different process -- with your use of it. You've
chosen to use this mechanism for assignment rather than something less
baroque, but that doesn't mean the capability is any less general.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash uses tmp files for inter-process communication instead of pipes?
On 10/6/14, 3:14 PM, Linda Walsh wrote:
> In running a startup script, I am endeavoring not to use tmp files
> where possible, As part of this, I sent the output of a command
> to stdout where I read it using the "variable read" syntax:
>
>while read ifname hwaddr; do
>printf "ifname=%s, hwaddr=%s\n" "$ifname" "$hwaddr"
>act_hw2if[$hwaddr]="$ifname"
>act_if2hw[$ifname]="$hwaddr"
>printf "act_hw2if[%s]=%s, act_if2hw[%s]=%s\n"
> "${act_hw2if[$hwaddr]}" "${act_if2hw[$ifname]}"
>done <<<"$(get_net_IFnames_hwaddrs)"
>
> Note, I used the <<<"$()" form to avoid process substitution -- as I was told
> on this list that the <<< form didn't use process substitution.
Correct, it does not.
>
> So I now get:
>> >>/etc/init.d/boot.assign_netif_names#192(get_net_IFnames_hwaddrs)>
> echo eth5 a0:36:9f:15:c9:c2
> /etc/init.d/boot.assign_netif_names: line 203: cannot create temp file for
> here-document: No such file or directory
>
> Where am I using a HERE doc?
You're using a here-string, which is a variant of a here-document. They
differ very little, mostly in syntax. The internal implementation is
identical.
> More importantly, at this point, where is it trying to write?/(Read).
>
> Simple Q. Why isn't it using some small fraction of the 90+G of memory
> that is free for use at this point?
Because the shell uses temp files for this. It's a simple, general
solution to the problem of providing arbitrary data on stdin.
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: bash uses tmp files for inter-process communication instead of pipes?
On Monday, October 06, 2014 02:00:47 PM Linda Walsh wrote:
>
> Greg Wooledge wrote:
> > On Mon, Oct 06, 2014 at 12:38:21PM -0700, Linda Walsh wrote:
> >> According to Chet , only way to do a multi-var assignment in bash is
> >>
> >> read a b c d <<<$(echo ${array[@]})
It's best to avoid turning your nice data structure into a stream only to read
(and mangle) it back again.
If you must, create a function.
function unpack {
${1+:} return 1
typeset -n _arrRef=$1 _ref
typeset -a _keys=("${!_arrRef[@]}")
typeset -i n=0
shift
for _ref; do
${_keys[n]+:} return
_ref=${_arrRef[${_keys[n++]}]}
done
}
# $ a=(1 2 3) e=hi; unpack a b c d e; typeset -p {a..e}
# declare -a a='([0]="1" [1]="2" [2]="3")'
# declare -- b="1"
# declare -- c="2"
# declare -- d="3"
# declare -- e="hi"
> Um... it used a socket.. to transfer it, then it uses a tmp file on top
> of that?! :
That's the wrong process. AF_NETLINK is the interface used by iproute2 for IPC
with the kernel. Bash doesn't use netlink sockets.
> >> So why would someone use a tmp file to do an assignment.
> >
> > It has nothing to do with assignments. Temp files are how here documents
> > (and their cousin here strings) are implemented. A here document is a
> > redirection of standard input. A redirection *from a file*
>
> Nope:
> This type of redirection instructs the shell to read input from the
> current source until a line containing only delimiter (with no
trailing
> blanks) is seen. All of the lines read up to that point are then
used
> as the standard input for a command.
>
> "The current source" -- can be anything that input can be
> redirected from -- including memory.
>
>
> > in fact,
> > albeit a file that is created on the fly for you by the shell.
>
> Gratuitous expectation of slow resources... Non-conservation
> of resources, not for the user, but for itself.
>
> Note:
> > a=$( > echo "$a"
> one
> two
> three
> ---
> no tmp files used, but it does a file read on foo.
>
> b=$a
> -- the above uses no tmp files.
>
> b=$(echo "$a")
> ---
> THAT uses no tmp files.
>
> But
> b=<<<$a
>
> uses a tmp file.
>
> That's ridiculously lame.
That isn't working code the way you think (except in mksh). In most shells its
just a null command that sets b to empty.
Just perform the assignment directly. There's no need for any special
expansion or redirect.
> >> So why would a temp file be used?
> >
> > Historical reasons, and because many processes will expect standard input
> > to have random access capabilities (at least the ability to rewind via
> > lseek). Since bash has no idea what program is going to be reading the
> > here document, why take chances?
>
> How could you use a seek in the middle of a HERE doc
> read?
>
> Not only that, it's just wrong./dev/stdin most often comes from
> a tty which is not random access.
There is no requirement for how heredocs are implemented. Temp files are
perfectly reasonable and most shells use them. The shell reads the heredoc,
but it is the command that would be doing any seeking, or possibly the shell
itself via nonstandard features like the <#(()) redirect.
This can be an optimization. Consider:
for file in file1 ... filen; do
ed -s "$file"
Re: bash uses tmp files for inter-process communication instead of pipes?
On Mon, Oct 06, 2014 at 04:04:25PM -0700, Linda Walsh wrote:
> Greg Wooledge wrote:
> >netdev_pat=... # (and other variable assignments)
> >(cd "$sysnet" &&
> >for ifname in ...; do
> >hwaddr=$(<"$ifname"/address)
> >act_hw2if[$hwaddr]="$ifname"
> >act_if2hw[$ifname]="$hwaddr"
> >done)
>
> Except that either act_hw2if + pair were just assigned to
> in the sub process that was meant to isolate the change
> of directory from the rest of the program,
> OR
> we aren't in the right directory when we do the reads.
OK, then use a function to give you an escapable block:
declare -A ...
create_maps() {
cd "$sysnet" || return
for ifname in ...; do
hwaddr=$(<"$ifname"/address)
act_hw2if[$hwaddr]="$ifname"
act_hw2if[$ifname]="$hwaddr"
done
}
create_maps
> Either way, they code as you have suggested won't work
> without overcoming another set of side effects.
There are ways to NOT use subshells. I have given you two of those
ways now.
