Re: bash sockets: printf \x0a does TCP fragmentation
On 9/23/18 8:26 PM, Chet Ramey wrote: > On 9/22/18 4:22 PM, Bob Proulx wrote: > >> Note that I *did* provide you with a way to do what you wanted to do. :-) >> >> It was also noted in another message that the external standalone >> printf command line utility did buffer as you desired. That seems >> another very good solution too. Simply use "command printf ..." to >> force using the external version. > > This won't work the way you want. The `command' builtin only inhibits > execution of shell functions. It still executes builtins. You want to > either get the full pathname of a printf utility using `type -ap printf' > and use that, or use the env or exec variants I recommended in my last > message. FYI: I ended up checking with type before whether an external printf exists and set a variable for this and then just call this variable. env or exec: never thought about it (thanks!) but as both are external commands, that would mean upon every call one additional external program. (yes, I know that there is such thing as a fs buffer). Subshells also costs resources. As this is a core function I am happy for every homeopathic dose of time I safe here :-) Cheers, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 12:30 PM, Ilkka Virta wrote: > The coreutils printf seems to output 'foo\nbar\n' as a single write, though > (unless > it goes to the terminal, so the usual stdio buffering), so you might be able > to use > that. thx. Might be not that portable but we'll see. > In any case, if a TCP endpoint cares about getting full data packets within a > single > segment, I'd say it's broken. fully agree. But unfortunately it just comforts us :-) Keep in mind that the purpose of the tool is testing and if due to a bug it can't do that, were the ones being blamed or we need to do really strange workarounds to avoid '\x0a' in the first 8 bytes. Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 12:38 PM, Ilkka Virta wrote: > On 22.9. 02:34, Chet Ramey wrote: >> Newline? It's probably that stdout is line-buffered and the newline causes >> a flush, which results in a write(2). > > Mostly out of curiosity, what kind of buffering logic does Bash (or the > builtin > printf in particular) use? It doesn't seem to be the usual stdio logic where > you get > line-buffering if printing to a terminal and block buffering otherwise. I get > a > distinct write per line even if the stdout of Bash itself is redirected to say > /dev/null or a pipe: > > $ strace -etrace=write bash -c 'printf "foo\nbar\n"' > /dev/null > write(1, "foo\n", 4) = 4 > write(1, "bar\n", 4) = 4 > +++ exited with 0 +++ Oh. But thanks anyway! coreutils in fact does it in one shot as you indicated. Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 7:30 AM, Bob Proulx wrote: > dirk+b...@testssl.sh wrote: >> we discovered a strange phenomenon in the project testssl.sh: > > You are doing something that is quite unusual. You are using a shell > script direction on a TCP socket. That isn't very common. Do you think there should be a paragraph NOT COMMON where bash sockets should rather belong to? > More > typically one would use a C program instead. So it isn't surprising > that you are finding interactions that are not well known. Bob, my intention was not to discuss program languages and what is typical with you or anybody else here. >> printf -- "$data" >&5 2>/dev/null > > Why is stderr discarded? That is almost always bad because it > discards any errors that might occur. You probably shouldn't do this.> > What happens if $data contains % format strings? What happens if the > format contains a sequence such as \c? This looks problematic. This > is not a safe programming proctice. I doubt you can judge on this by just looking at a single line of code -- the project has > 18k LoC in bash. Github is the place to discuss and do PRs for our project. >> If there's a workaround, please let me know. (tried to add "%b" with no >> effect). Otherwise I believe it's a bug. > > You can re-block the output stream using other tools such as 'cat' or > 'dd'. Since you are concerned about block size then perhaps dd is the > better of the two. > > | cat cat has a problem with binary chars, right? And: see below. > Or probably better: > > | dd status=none bs=1M > > Or use whatever block size you wish. The 'dd' program will read the > input into its buffer and then output that block of data all in one > write(2). That seems to be what you are wanting. We actually use dd to read from the socket. Of course we could use writing to it as well -- at a certain point of time. Still, a prerequisite would be that printf is the culprit and not how bash + libs do sockets. > P.S. You can possibly use the 'stdbuf' command to control the output > buffering depending upon the program. > > info stdbuf That could be an option, thanks. Need to check though whether a) it doesn't fragment then -- not sure while reading it b) it's per default available on every platform supported by testssl.sh. Cheers, Dirk
Re: bash sockets: printf \x0a does TCP fragmentation
On 9/22/18 1:34 AM, Chet Ramey wrote: > On 9/21/18 4:13 PM, dirk+b...@testssl.sh wrote: >> >> Hello there, >> >> we discovered a strange phenomenon in the project testssl.sh: >> >> After opening a TCP socket with a fd (here: 5), when writing to it, >> it seems that >> >> printf -- "$data" >&5 2>/dev/null >> >> does not do what it is intended. "$data" is a ClientHello like >> >> '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...' >> >> Each \x0a like the last one causes a new TCP fragment to begin which can be >> easily >> spotted when using wireshark while running e.g. > > Newline? It's probably that stdout is line-buffered and the newline causes > a flush, which results in a write(2). Anything one can do on the level of bash or non-syscall land? What about ulimit -b ? >> If there's a workaround, please let me know. (tried to add "%b" with no >> effect). Otherwise I believe it's a bug. > > How? Does the emitted output not correspond to what's passed to printf > in some way? "\x0a" is a legitimate byte which is send from time to time over the socket. It happens if the record layer is e.g. 522 bytes (\x02\x0a), if a standard cipher is included in the handshake like (\xc0\x0a) or DES-CBC3-SHA (\x00\x0a) ECDHE-ECDSA-AES256-SHA or at any other occasion. Everything works as expected and like a charm for years now -- the only thing isthe underlying TCP fragmentation which is caused as a side effect by sending \x0a. As indicated a few servers under certain condition can't cope with it asif the TCP first segment is too small they don't treat this as ClientHello and just drop the packet, see thread @ https://github.com/drwetter/testssl.sh/pull/1113, specifically the hint wrt https://support.f5.com/csp/article/K53322151 . My stance is simply if I use in the internal feature of bash for TCP socket programming I didn't expect to have side effects like this. Thx, Dirk
bash sockets: printf \x0a does TCP fragmentation
Hello there, we discovered a strange phenomenon in the project testssl.sh: After opening a TCP socket with a fd (here: 5), when writing to it, it seems that printf -- "$data" >&5 2>/dev/null does not do what it is intended. "$data" is a ClientHello like '\x16\x03\x01\x2\x00\x01\x00\x1\xfc\x03\x03\x54\x51\x1e\x7a\xde\xad\xbe\xef\x31\x33\x07\x00\x00\x00\x00\x00\xcf\xbd\x39\x04\xcc\x16\x0a\...' Each \x0a like the last one causes a new TCP fragment to begin which can be easily spotted when using wireshark while running e.g. testssl.sh --assume-http -p testssl.sh Starting from the SSLv3 ClientHello the first reassembled packet ends with 0a. See also discussion @ https://github.com/drwetter/testssl.sh/pull/1113. One would assume that a bash socket connection cannot influence the TCP fragmentation but obviously it does. This behavior has a performance penalty and other strange effects, e.g. if the first segment is really small, some devices reject the ClientHello. If there's a workaround, please let me know. (tried to add "%b" with no effect). Otherwise I believe it's a bug. Cheers, Dirk PS: Would ulimit -b help?
