Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
but it seemed to work and not be at fault upon further exploration. Now it's one of 2 Associative arrays (often called 'map's in the code where they are used as such) that is failing due to illegal subscript messages. The fact that one of the maps works and the other does not seems odd. They are both initialized the same way. Have you considered printing the value you're trying to use as a subscript before the failing line is executed? Is it really the case that now that the problem is better defined, that it really is just some problem in bash that can only reproduced on a booting system? Unlikely. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
Chet Ramey wrote: but it seemed to work and not be at fault upon further exploration. Now it's one of 2 Associative arrays (often called 'map's in the code where they are used as such) that is failing due to illegal subscript messages. The fact that one of the maps works and the other does not seems odd. They are both initialized the same way. Have you considered printing the value you're trying to use as a subscript before the failing line is executed? It's been a while, but I seem to remember doing that and setting -x at the beginning of the script and picking through that post-boot. I try it again... might easily get a different result with how things are going for me lately...
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
I got various suggestions for getting my script to work at boot time, and any of them might have been applicable before I'd gotten to the current problem... In particular the $VAR construct where VAR was holding output from a program.. was something on my iffy list, but it seemed to work and not be at fault upon further exploration. Now it's one of 2 Associative arrays (often called 'map's in the code where they are used as such) that is failing due to illegal subscript messages. The fact that one of the maps works and the other does not seems odd. They are both initialized the same way. I've also tried resetting the environment... (env -i bash --norc --noprofile; bash-4.2$ ...command works...) Is it really the case that now that the problem is better defined, that it really is just some problem in bash that can only reproduced on a booting system? Note -- if you invoke the script with with no parms, it display normal options ifmap (shows current mapping) and remap which does a verify on the names and does renaming if needed (based on it's internal table -- i.e. don't use this unless you've adapted it to your system). undocumented: if you invoke it with 'start' it does the remap function. Other than a few left-over functions from earlier debug attempts (tests removed, but some of the functions remain so I can maybe move them to a library later (e.g. - isarray varflags), it is fairly lean for design of non-throwaway code.
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
Linda Walsh b...@tlinx.org writes: Except that in-line HERE docs don't need to be implemented through a tmp file unless you want to slow things down. They should be read out of memory and NOT transfered to to non-existent, external storage. You need a file descriptor for your memory storage. Andreas. -- Andreas Schwab, SUSE Labs, sch...@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 And now for something completely different.
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
Andreas Schwab wrote: Linda Walsh b...@tlinx.org writes: Except that in-line HERE docs don't need to be implemented through a tmp file unless you want to slow things down. They should be read out of memory and NOT transfered to to non-existent, external storage. You need a file descriptor for your memory storage. --- Why? If it is the case that you can't rely on OS services, you are saying you can't access memory or create a virtual descriptor that reads out of a large malloc'ed buffer? Both C++ and Perl have routines to do I/O on strings directly that don't go through system file descriptors to do it. Last I checked, 'C' was a common denominator for both as well as Bash. So why should it be the case that Bash can't do it when it is the primary system shell available at boot. I wrote my own line-splitting, word splitting and buffering code to avoid libc buffering to talk to files in proc, repeatedly w/o closing reopening. It's mostly in C and was relatively simple compared to the work of writing a simple scheduler. There's no reason for me to expect that a system shell wouldn't have such available.
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
On Tue, Jul 29, 2014 at 12:06:41PM -0700, Linda Walsh wrote: Andreas Schwab wrote: You need a file descriptor for your memory storage. --- Why? A here-document is a redirection. All it does it change where stdin comes from. There has to be a place for file descriptor 0 to point to. This can be a real file on disk, or one end of a pipe(). I'm not aware of any shells that use a multi-process pipe() implementation for here-documents. Maybe this is just historical inertia, or maybe it's because a temporary file can be lseek()ed, which gives the most flexibility for the processes that are reading them. Or maybe it's because a temp file is considered less of a resource waste than an extra process. I really do advise you to simplify your boot scripts. Boot scripts run in minimalist environments, where advanced features are not always available. Boot scripts also typically are not run by bash; they're run by /bin/sh (or even /sbin/sh), which is more likely to be some stripped-down POSIX shell than it is to be bash. If you've got some big static block of text that you need to feed to a daemon at boot time, maybe it would be appropriate to stick this text in a config file (/etc/default/yourdaemon or /etc/yourdaemon/startup.text or something like that). Then you can redirect from that file instead of using a here-document buried in a boot script. From a system administration perspective, this separation of code from data would be a lot cleaner.
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
Linda Walsh wrote Andreas Schwab wrote: Linda Walsh b...@tlinx.org writes: Except that in-line HERE docs don't need to be implemented through a tmp file unless you want to slow things down. They should be read out of memory and NOT transfered to to non-existent, external storage. You need a file descriptor for your memory storage. --- Why? Because that's what the input command expects: your HERE-string *when it reads its standard input* (fd 0) You could implement it with pipes, but as you would need a thread to write the partially-read string, just using a file is simpler. And as you can write it into a deleted file, the effect on storage should be minimal. You should however be able to translate “foo SOMETHING” into “echo SOMETHING | foo” What surprises me is that the only use of here-docs (actually here-strings) in your script are process substitutions: read out $(declare -p $var ) while ... done $(get_net_IFnames_hwaddrs) When it looks simpler to write the non-here version: declare -p $var | read out get_net_IFnames_hwaddrs | while ... Regards
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
On Tue, Jul 29, 2014 at 10:21:36PM +0200, Angel wrote: What surprises me is that the only use of here-docs (actually here-strings) in your script are process substitutions: read out $(declare -p $var ) while ... done $(get_net_IFnames_hwaddrs) When it looks simpler to write the non-here version: declare -p $var | read out get_net_IFnames_hwaddrs | while ... The 'read' example will not work as you've written it. The change to the shell variable 'out' will be lost when the pipeline terminates. (But you can get a very recent bash release and set the lastpipe shopt to work around this.) If the while loop also tries to set shell variables, then it will have the same problem (and the same possible workaround). It looks like Linda's original script was doing something like this: read out (declare -p $var) and she discovered that she couldn't use process substitutions in this boot script (presumably because Linux's /dev/fd/ file system wasn't mounted yet), so she tried to replace it with a here-document, and now has learned that /tmp is not mounted (or is mounted read-only) at that point. I won't even try to guess why she can't just do out=$var ... there is probably some extremely silly reason that will just make me want to slam my head into my desk
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
Greg Wooledge wrote: read out (declare -p $var) That code actually isn't called. It's used by isarray to tell me whether or not a var is an array. The code that doesn't work is the line with the comment line 233. (which is now line 235 from the latest error message. notice -- Jul 22 19:02:49.388388000 service boot.clock start assign_netif_names=/etc/init.d/boot.assign_netif_names start ifname=eth0, hwaddr=00:15:17:bf:be:b2 ifname=eth1, hwaddr=00:15:17:bf:be:b3 ifname=eth2, hwaddr=00:26:b9:48:71:e2 ifname=eth3, hwaddr=00:26:b9:48:71:e4k ifname=eth4, hwaddr=a0:36:9f:15:c9:c0 ifname=eth5, hwaddr=a0:36:9f:15:c9:c2 /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript /etc/init.d/boot.assign_netif_names: line 235: act_hw2if: bad array subscript Note that according to the printf in read_actuals, the values are correctly being read in from my here var. Note, you can't use an assignment there. It's reading names+hwaddrs of the net devs @ boot in get_net_IFnames_hwaddrs. That requires line and field splitting. Now that I'm writing about it -- I'm thinking that it doesn't like me using the hwaddr's @ boot time as hash subscripts. That's what I don't understand. It works normally, but why would hashes fail at boot? I couldn't think of how they'd be implemented to use some non-existent service, so I don't know which resource lack is causing the fail. I won't even try to guess why she can't just do out=$var ... there is probably some extremely silly reason that will just make me want to slam my head into my desk --- Yeah, like multiple lines and fields in $var...
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
Greg Wooledge wrote: The 'read' example will not work as you've written it. The change to the shell variable 'out' will be lost when the pipeline terminates. (But you can get a very recent bash release and set the lastpipe shopt to work around this.) If the while loop also tries to set shell variables, then it will have the same problem (and the same possible workaround). You are right. Turns out I didn't reset the variable between tests. Or you can put the inside-code into brackets :)
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
On Fri, Jul 25, 2014 at 04:44:27PM -0700, Linda Walsh wrote: FWIW, this script DOES work interactively in normal operation. Just when the system is in pre-single-user state and not much in the way of resources is available that it blows. Then you are attempting to use operating system features that haven't been initialized yet. Stop doing that. Here documents create temporary files, possibly in /tmp, or possibly in whatever directory $TMPDIR points to. So, /tmp (or /var/tmp or whatever) must be mounted and writable. Process substitutions **ON LINUX** use /dev/fd/* entries, which are in a special file system that must be mounted. Process substitutions on other operating systems may use that, or named pipes, or even temporary files. In any case, the problem is clearly that your script is trying to use features that aren't available yet, at that point in the boot process. Rewrite the script to avoid using those features, or run it at a different point in the boot process. #!/bin/bash ### BEGIN INIT INFO # Provides: net-devices Too large. Trim it down to the smallest possible case that still exhibits the problem. But even then, I'm sure you already know what the problem is. (P.S. I despise that let me figure out the order for you crap that Debian has started doing. What a disaster. How in the hell is a sysadmin supposed to add a boot script now, and make sure it runs at the right point)
Re: Code for :Re: HERE document failed Re: /dev/fd/62: No such file or directory
Greg Wooledge wrote: On Fri, Jul 25, 2014 at 04:44:27PM -0700, Linda Walsh wrote: FWIW, this script DOES work interactively in normal operation. Just when the system is in pre-single-user state and not much in the way of resources is available that it blows. Then you are attempting to use operating system features that haven't been initialized yet. Stop doing that. Except that in-line HERE docs don't need to be implemented through a tmp file unless you want to slow things down. They should be read out of memory and NOT transfered to to non-existent, external storage. Maybe POSIX says differently, but I doubt it differentiates when you are allowed to use features based on when you run the shell. Here documents create temporary files, possibly in /tmp, or possibly in whatever directory $TMPDIR points to. So, /tmp (or /var/tmp or whatever) must be mounted and writable. Is that part of a standard somewhere? The point is to similate an input stream. Wouldn't it be more portable to do that from memory? Process substitutions **ON LINUX** use /dev/fd/* entries, which are in a special file system that must be mounted. Process substitutions on other operating systems may use that, or named pipes, or even temporary files. I'll allow those as they are too new to really be relying on, but HERE docs are basic, and shouldn't need external run-time support for the type of stuff I'm doing. In any case, the problem is clearly that your script is trying to use features that aren't available yet, at that point in the boot process. --- Like reading another process's output so I can alter variables that are in the main thread. The other methods for reading in data reverse the reader/writer and put the reader into a child such that it can't modify variables in the running script. Rewrite the script to avoid using those features, or run it at a different point in the boot process. I have rewritten it to avoid the process substitutions and fell back to HERE docs for that. There is no standard or documentation about what parts of a POSIX shell are not available at boot time or how to work around the deficiencies. Also it isn't easy to run things elsewhere in the boot script, as I'll likely have to run systemd at soem point and it will just take over and I'll have no control of when things run, so I'm doing some early initialization so I can then turn over the control thread to something else. For example, I also have to mount /usr and /usr/share before I can boot, or systemd won't be happy. Too large. Trim it down to the smallest possible case that still exhibits the problem. But even then, I'm sure you already know what the problem is. That's not easy, since by definition, it only works when there is no environment to support trimming it. (P.S. I despise that let me figure out the order for you crap that Debian has started doing. What a disaster. How in the hell is a sysadmin supposed to add a boot script now, and make sure it runs at the right point) You aren't supposed to be adding boot scripts... Wait till they get the walled garden up around PC's w/secure boot and only allowing signed apps. G.
Re: HERE document failed Re: /dev/fd/62: No such file or directory
On 7/24/14, 10:46 PM, Linda Walsh wrote: Chet Ramey wrote: Only you can answer questions about where your scripts, and your `system scripts', use process substitution. I tried to switch to in-line HERE documents instead of /dev/fd/62... but those don't work either. I thought HERE docs were pretty basic? Are HERE documents also an optional feature that are not included in POSIX? Maybe you should post your script so readers can take a look. It's unlikely that there is a bug in here-documents, but it's possible. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, ITS, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/
Re: HERE document failed Re: /dev/fd/62: No such file or directory
Chet Ramey wrote: Only you can answer questions about where your scripts, and your `system scripts', use process substitution. I tried to switch to in-line HERE documents instead of /dev/fd/62... but those don't work either. I thought HERE docs were pretty basic? Are HERE documents also an optional feature that are not included in POSIX? Thanks...
