Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Chet Ramey wrote: but it seemed to work and not be at fault upon further exploration. Now it's one of 2 Associative arrays (often called "'map's" in the code where they are used as such) that is failing due to illegal subscript messages. The fact that one of the maps works and the other does not seems odd. They are both initialized the same way. Have you considered printing the value you're trying to use as a subscript before the failing line is executed? It's been a while, but I seem to remember doing that and setting -x at the beginning of the script and picking through that post-boot. I try it again... might easily get a different result with how things are going for me lately...
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
> but it seemed to work and not be at fault upon further > exploration. Now it's one of 2 Associative arrays (often > called "'map's" in the code where they are used as such) > that is failing due to illegal subscript messages. > > The fact that one of the maps works and the other does not > seems odd. They are both initialized the same way. Have you considered printing the value you're trying to use as a subscript before the failing line is executed? > Is it really the case that now that the problem is better > defined, that it really is just some problem in bash > that can only reproduced on a booting system? Unlikely. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
I got various suggestions for getting my script to work at boot time, and any of them might have been applicable before I'd gotten to the current problem... In particular the <<<"$VAR" construct where VAR was holding output from a program.. was something on my "iffy" list, but it seemed to work and not be at fault upon further exploration. Now it's one of 2 Associative arrays (often called "'map's" in the code where they are used as such) that is failing due to illegal subscript messages. The fact that one of the maps works and the other does not seems odd. They are both initialized the same way. I've also tried resetting the environment... (env -i bash --norc --noprofile; bash-4.2$ ...command works...) Is it really the case that now that the problem is better defined, that it really is just some problem in bash that can only reproduced on a booting system? Note -- if you invoke the script with with no parms, it display normal options "ifmap" (shows current mapping) and "remap" which does a verify on the names and does renaming if needed (based on it's internal table -- i.e. don't use this unless you've adapted it to your system). undocumented: if you invoke it with 'start' it does the "remap" function. Other than a few left-over functions from earlier debug attempts (tests removed, but some of the functions remain so I can maybe move them to a library later (e.g. - isarray & varflags), it is fairly lean for design of non-throwaway code.
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Greg Wooledge wrote: > The 'read' example will not work as you've written it. The change to > the shell variable 'out' will be lost when the pipeline terminates. > (But you can get a very recent bash release and set the "lastpipe" > shopt to work around this.) > > If the while loop also tries to set shell variables, then it will have > the same problem (and the same possible workaround). You are right. Turns out I didn't reset the variable between tests. Or you can put the inside-code into brackets :)
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Greg Wooledge wrote: read out < <(declare -p "$var") That code actually isn't called. It's used by "isarray" to tell me whether or not a var is an array. The code that doesn't work is the line with the comment "line 233". (which is now line 235 from the latest error message. service boot.clock start assign_netif_names=/etc/init.d/boot.assign_netif_names start ifname=eth0, hwaddr=00:15:17:bf:be:b2 ifname=eth1, hwaddr=00:15:17:bf:be:b3 ifname=eth2, hwaddr=00:26:b9:48:71:e2 ifname=eth3, hwaddr=00:26:b9:48:71:e4k ifname=eth4, hwaddr=a0:36:9f:15:c9:c0 ifname=eth5, hwaddr=a0:36:9f:15:c9:c2 /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript Note that according to the "printf in "read_actuals", the values are correctly being read in from my "here var". Note, you can't use an assignment there. It's reading names+hwaddrs of the net devs @ boot in "get_net_IFnames_hwaddrs". That requires line and field splitting. Now that I'm writing about it -- I'm thinking that it doesn't like me using the hwaddr's @ boot time as hash subscripts. That's what I don't understand. It works "normally", but why would "hashes" fail at boot? I couldn't think of how they'd be implemented to use some non-existent service, so I don't know which resource lack is causing the fail. I won't even try to guess why she can't just do out=$var ... there is probably some extremely silly reason that will just make me want to slam my head into my desk --- Yeah, like multiple lines and fields in "$var"...
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
On Tue, Jul 29, 2014 at 10:21:36PM +0200, Angel wrote: > What surprises me is that the only use of here-docs (actually > here-strings) in your script are process substitutions: > > read out <<<$(declare -p "$var" ) > > while ... done <<<"$(get_net_IFnames_hwaddrs)" > > When it looks simpler to write the non-here version: > > declare -p "$var" | read out > > get_net_IFnames_hwaddrs | while ... The 'read' example will not work as you've written it. The change to the shell variable 'out' will be lost when the pipeline terminates. (But you can get a very recent bash release and set the "lastpipe" shopt to work around this.) If the while loop also tries to set shell variables, then it will have the same problem (and the same possible workaround). It looks like Linda's original script was doing something like this: read out < <(declare -p "$var") and she discovered that she couldn't use process substitutions in this boot script (presumably because Linux's /dev/fd/ file system wasn't mounted yet), so she tried to replace it with a here-document, and now has learned that /tmp is not mounted (or is mounted read-only) at that point. I won't even try to guess why she can't just do out=$var ... there is probably some extremely silly reason that will just make me want to slam my head into my desk
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Linda Walsh wrote > > Andreas Schwab wrote: > > Linda Walsh writes: > > > >>Except that in-line HERE docs don't need to be implemented > >> through a tmp file unless you want to slow things down. > >>They should be read out of memory and NOT transfered to > >> to non-existent, external storage. > > > > You need a file descriptor for your memory storage. > --- > Why? Because that's what the input command expects: your HERE-string *when it reads its standard input* (fd 0) You could implement it with pipes, but as you would need a thread to write the partially-read string, just using a file is simpler. And as you can write it into a deleted file, the effect on storage should be minimal. You should however be able to translate “foo << read out <<<$(declare -p "$var" ) > while ... done <<<"$(get_net_IFnames_hwaddrs)" When it looks simpler to write the non-here version: > declare -p "$var" | read out > get_net_IFnames_hwaddrs | while ... Regards
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
On Tue, Jul 29, 2014 at 12:06:41PM -0700, Linda Walsh wrote: > Andreas Schwab wrote: > >You need a file descriptor for your memory storage. > --- > Why? A here-document is a redirection. All it does it change where stdin comes from. There has to be a place for file descriptor 0 to point to. This can be a real file on disk, or one end of a pipe(). I'm not aware of any shells that use a multi-process pipe() implementation for here-documents. Maybe this is just historical inertia, or maybe it's because a temporary file can be lseek()ed, which gives the most flexibility for the processes that are reading them. Or maybe it's because a temp file is considered less of a resource waste than an extra process. I really do advise you to simplify your boot scripts. Boot scripts run in minimalist environments, where advanced features are not always available. Boot scripts also typically are not run by bash; they're run by /bin/sh (or even /sbin/sh), which is more likely to be some stripped-down POSIX shell than it is to be bash. If you've got some big static block of text that you need to feed to a daemon at boot time, maybe it would be appropriate to stick this text in a config file (/etc/default/yourdaemon or /etc/yourdaemon/startup.text or something like that). Then you can redirect from that file instead of using a here-document buried in a boot script. From a system administration perspective, this separation of code from data would be a lot cleaner.
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Andreas Schwab wrote: Linda Walsh writes: Except that in-line HERE docs don't need to be implemented through a tmp file unless you want to slow things down. They should be read out of memory and NOT transfered to to non-existent, external storage. You need a file descriptor for your memory storage. --- Why? If it is the case that you can't rely on OS services, you are saying you can't access memory or create a virtual descriptor that reads out of a large malloc'ed buffer? Both C++ and Perl have routines to do I/O on strings directly that don't go through system file descriptors to do it. Last I checked, 'C' was a common denominator for both as well as Bash. So why should it be the case that Bash can't do it when it is the primary system shell available at boot. I wrote my own line-splitting, word splitting and buffering code to avoid libc buffering to talk to files in "proc", repeatedly w/o closing & reopening. It's mostly in C and was relatively simple compared to the work of writing a simple scheduler. There's no reason for me to expect that a system shell wouldn't have such available.
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Linda Walsh writes: > Except that in-line HERE docs don't need to be implemented > through a tmp file unless you want to slow things down. > They should be read out of memory and NOT transfered to > to non-existent, external storage. You need a file descriptor for your memory storage. Andreas. -- Andreas Schwab, SUSE Labs, sch...@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Greg Wooledge wrote: On Fri, Jul 25, 2014 at 04:44:27PM -0700, Linda Walsh wrote: FWIW, this script DOES work interactively in normal operation. Just when the system is in pre-single-user state and not much in the way of resources is available that it blows. Then you are attempting to use operating system features that haven't been initialized yet. Stop doing that. Except that in-line HERE docs don't need to be implemented through a tmp file unless you want to slow things down. They should be read out of memory and NOT transfered to to non-existent, external storage. Maybe POSIX says differently, but I doubt it differentiates when you are allowed to use features based on when you run the shell. Here documents create temporary files, possibly in /tmp, or possibly in whatever directory $TMPDIR points to. So, /tmp (or /var/tmp or whatever) must be mounted and writable. Is that part of a standard somewhere? The point is to similate an input stream. Wouldn't it be more portable to do that from memory? Process substitutions **ON LINUX** use /dev/fd/* entries, which are in a special file system that must be mounted. Process substitutions on other operating systems may use that, or named pipes, or even temporary files. I'll allow those as they are too new to really be relying on, but HERE docs are basic, and shouldn't need external run-time support for the type of stuff I'm doing. In any case, the problem is clearly that your script is trying to use features that aren't available yet, at that point in the boot process. --- Like reading another process's output so I can alter variables that are in the main thread. The other methods for reading in data reverse the reader/writer and put the reader into a child such that it can't modify variables in the running script. Rewrite the script to avoid using those features, or run it at a different point in the boot process. I have rewritten it to avoid the process substitutions and fell back to HERE docs for that. There is no standard or documentation about what parts of a POSIX shell are not available at boot time or how to work around the deficiencies. Also it isn't easy to run things elsewhere in the boot script, as I'll likely have to run systemd at soem point and it will just take over and I'll have no control of when things run, so I'm doing some early initialization so I can then turn over the control thread to something else. For example, I also have to mount "/usr" and "/usr/share" before I can "boot", or systemd won't be happy. Too large. Trim it down to the smallest possible case that still exhibits the problem. But even then, I'm sure you already know what the problem is. That's not easy, since by definition, it only works when there is no environment to support trimming it. (P.S. I despise that "let me figure out the order for you" crap that Debian has started doing. What a disaster. How in the hell is a sysadmin supposed to add a boot script now, and make sure it runs at the right point) You aren't supposed to be adding boot scripts... Wait till they get the walled garden up around PC's w/secure boot and only allowing signed apps. G.
Re: Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
On Fri, Jul 25, 2014 at 04:44:27PM -0700, Linda Walsh wrote: > FWIW, this script DOES work interactively in normal operation. > > Just when the system is in pre-single-user state and not > much in the way of resources is available that it blows. Then you are attempting to use operating system features that haven't been initialized yet. Stop doing that. Here documents create temporary files, possibly in /tmp, or possibly in whatever directory $TMPDIR points to. So, /tmp (or /var/tmp or whatever) must be mounted and writable. Process substitutions **ON LINUX** use /dev/fd/* entries, which are in a special file system that must be mounted. Process substitutions on other operating systems may use that, or named pipes, or even temporary files. In any case, the problem is clearly that your script is trying to use features that aren't available yet, at that point in the boot process. Rewrite the script to avoid using those features, or run it at a different point in the boot process. > #!/bin/bash > > ### BEGIN INIT INFO > # Provides: net-devices Too large. Trim it down to the smallest possible case that still exhibits the problem. But even then, I'm sure you already know what the problem is. (P.S. I despise that "let me figure out the order for you" crap that Debian has started doing. What a disaster. How in the hell is a sysadmin supposed to add a boot script now, and make sure it runs at the right point)
Code for :"Re: HERE document failed && Re: /dev/fd/62: No such file or directory"
Chet Ramey wrote:
Maybe you should post your script so readers can take a look. It's
unlikely that there is a bug in here-documents, but it's possible.
If you ask for it... I "inlined" the needed library functions,
so should run standalone.
FWIW, this script DOES work interactively in normal operation.
Just when the system is in pre-single-user state and not
much in the way of resources is available that it blows.
Right now only have 'start' implemented, as there is little call
for deconstructing network configs when the system is going down.
'ifmap' by itself should show current eth-dev mappings.
Note, buried in this code are absolute numbers (MAC addrs),
so to test on another machine they would need to be changed.
Have intended to put "site-local" info in a config file, but
as it doesn't workhaven't spent alot of time on expanding it.
--- ifmap -
#!/bin/bash
### BEGIN INIT INFO
# Provides: net-devices
# Required-Start: boot.udev boot.device-mapper boot.localfs
# Required-Stop: $null
# Default-Start: B
# Default-Stop:
# Short-Description: order net devices
# Description: order net devs if needed
### END INIT INFO
#
# assign network names as rc-script
# L A Walsh, (free to use/modify/distribute to nice people) (c) 2013-2014
#
#include standard template:
# gvim=:SetNumberAndWidth
echo "assign_netif_names=$0 $@"
_prgpth=${0:?}; _prgpth=${_prgpth#-} _prg=${_prgpth##*/};
_prgdr=${_prgpth%/$_prg}
[[ -z $_prgdr || $_prg == $_prgdr ]] && _prgdr="$PWD"
#if ! typeset -f include >&/dev/null ;then
# source ${_LOCAL_DIR:=/etc/local}/bash_env.sh;
#fi
export PATH="/etc/local/bin:/etc/local/lib:$PATH"
export
PS4='>>${BASH_SOURCE:+${BASH_SOURCE[0]}}#${LINENO}${FUNCNAME:+(${FUNCNAME[0]})}> '
#include stdalias (needed entries included below)
shopt -s extglob expand_aliases
alias dcl=declare sub=function
alias int=dcl\ -i map=dcl\ -A hash=dcl\ -Aarray=dcl\ -a
alias lower=dcl\ -l upper=dcl\ -u string=dcl my=dcl
alias map2int=dcl\ -AiintArray=dcl\ -ia
#include rc.status -- essential funcs included below:
int rc_status=0
sub rc_reset { rc_status=0; }
sub rc_status {
rc_status=$?;
if ((rc_status)) && { (($#)) && [[ $1 = -v ]] ; }; then
echo "Abnormal rc_status was $rc_status)."
elif (($#)) && [[ $1 = -v ]] ; then
echo "rc_status: ok"
fi
}
sub rc_exit {
rc_status=$?;
rc_status
exit $rc_status
}
# need to list commands here:
# modprobe
sub warn () { local msg="Warning: ${1:-"general"}"
echo "$msg" >&2
}
sub die () { int stat=$?; local msg="Error. ${1:-"unknown"}"
echo "$msg (errno=$stat)" >&2
(exit $stat);
rc_status -v
rc_exit
exit $stat
}
if [[ -z $(type -P modprobe) ]]; then
export PATH=/bin:/sbin:/usr/bin:/usr/sbin:$PATH
fi
if [[ -n $(type -P modprobe) ]]; then
alias modprobe="$(type -P modprobe)"
else
#delay failure until use
alias modprobe="die 'cannot load required modules'"
fi
if [[ -z $(type -P ip ) ]]; then
die "Cannot find 'ip' util -- needed for network setup"
fi
alias ip="$(type -P ip)"
sub varflags() {
my var="${1:-""}"
read out <<<$(declare -p "$var" )
[[ $out =~ /^declare.*=.*$/ ]] || die "no such variable"
out="${out%% +([^-])=*}"
out="${out#declare }"
[[ ${out:0:1} == - ]] || { echo ""; return 0 ; }
echo "${out#-}"
}
sub isarray() {
my name="${1:-""}"
flags=$(varflags $name)
[[ $flags =~ a ]] && return 0
return 1
}
sub ipcmd () {
my ipcmd="${1:?}"; shift; array tmpbuff
my outbuff="${2:-tmpbuff}"
}
sysfs=/sys
sysnet=$sysfs/class/net
sys_modules=$sysfs/module
sub rev () {
(($#==0)) && { echo ""; return 0 ;}
my element=${1:?}; shift;
(($#==0)) && { echo "$element"; return 0;}
echo "$(rev "$@") $element"
}
sub rename_if () {
my old_name=${1:?} new_name=${2:?}
echo ip link set name "$new_name" dev "$old_name"
}
sub down_if () {
my if_name=${1:?}
echo ip link set down dev "$if_name"
}
sub set_links_down() {# can't operate on up links
down_if eth2 down
down_if eth3 down
down_if eth4 down
down_if eth5 down
}
map act_hw2if=() #actual values (to be read in)
map act_if2hw=()
map XIF=()#tmp array to hold exchanged IF's
##inline data (should be in external file)
map hw2if=( [00:15:17:bf:be:b2]=eth0 [00:15:17:bf:be:b3]=eth1
[00:26:b9:48:71:e2]=eth2 [00:26:b9:48:71:e4]=eth3
[a0:36:9f:15:c9:c0]=eth4 [a0:36:9f:15:c9:c2]=eth5 )
map if2hw=( [eth0]=00:15:17:bf:be:b2 [eth1]=00:15:17:bf:be:b3
[eth2]=00:26:b9:48:71:e2 [eth3]=00:26:b9:48:71:e4
[eth4]=a0:36:9f:15:c9:c0 [eth5]=a0:36:9f:15:c9:c2 )
#
#needed_drivers=(e1000e bnx2 ixgbe bonding)
needed_drivers=(e1000e bnx2 ixgbe)
sub vrfy_drivers () {
int errors=0;
for i in ${needed_drivers[@]} ; do
if [[ ! -d $sys_modules/$i ]]; then
modprobe "$i" || {
warn "Module $i is not in kernel and can't be loaded"
errors+=1
