Re: conf/128030: [request] Isn't it time to enable IPsec in GENERIC?
On Saturday 18 October 2008 19:05:26 Sam Leffler wrote: [EMAIL PROTECTED] wrote: Synopsis: [request] Isn't it time to enable IPsec in GENERIC? Responsible-Changed-From-To: freebsd-bugs-freebsd-net Responsible-Changed-By: gavin Responsible-Changed-When: Sat Oct 18 16:55:14 UTC 2008 Responsible-Changed-Why: Over to maintainer(s) for consideration http://www.freebsd.org/cgi/query-pr.cgi?pr=128030 Last I checked IPSEC added noticeable overhead. Before anyone does this you need to measure the cost of having it enabled but not used. It should be possible to turn IPSEC into a module - maybe only loadable on boot to avoid locking issues. This would reduce the overhead to a handful of function pointer checks that should not impact performance (thanks to modern branch prediction and cache sizes). This would have to be measured as well, of course. Maybe this should go to the project page? It's a good junior kernel hacker project, I believe. -- /\ Best regards, | [EMAIL PROTECTED] \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | [EMAIL PROTECTED] / \ ASCII Ribbon Campaign | Against HTML Mail and News ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: conf/128030: [request] Isn't it time to enable IPsec in GENERIC?
Max Laier wrote: On Saturday 18 October 2008 19:05:26 Sam Leffler wrote: [EMAIL PROTECTED] wrote: Synopsis: [request] Isn't it time to enable IPsec in GENERIC? Responsible-Changed-From-To: freebsd-bugs-freebsd-net Responsible-Changed-By: gavin Responsible-Changed-When: Sat Oct 18 16:55:14 UTC 2008 Responsible-Changed-Why: Over to maintainer(s) for consideration http://www.freebsd.org/cgi/query-pr.cgi?pr=128030 Last I checked IPSEC added noticeable overhead. Before anyone does this you need to measure the cost of having it enabled but not used. It should be possible to turn IPSEC into a module - maybe only loadable on boot to avoid locking issues. This would reduce the overhead to a handful of function pointer checks that should not impact performance (thanks to modern branch prediction and cache sizes). This would have to be measured as well, of course. Maybe this should go to the project page? It's a good junior kernel hacker project, I believe. I believe the most important issue are the SADB checks in the tx path. It used to be possible to do them cheaply by checking a single ptr value but now it's much more expensive. My memory is hazy as it's been a while. Sam ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Pipes, cat buffer size
Hi,
I'm working on a program that's intended to be used as a filter, as in
something | myprogram file. I'm trying it with cat and I'm seeing my
read()s return small blocks, 64 kB in size. I suppose this is because
cat writes in 64 kB blocks. So:
a) Is there a way to programatically, per-process, set the pipe buffer
size? The program in question is a compressor and it's particularly
inefficient when given small blocks and I'm wondering if the system can
buffer enough data for it.
b) Is there any objection to the following patch to cat:
Index: cat.c
===
--- cat.c (revision 184033)
+++ cat.c (working copy)
@@ -247,7 +247,16 @@
if (buf == NULL) {
if (fstat(wfd, sbuf))
err(1, %s, filename);
- bsize = MAX(sbuf.st_blksize, 1024);
+ if (S_ISREG(sbuf.st_mode)) {
+ /* If there's plenty of RAM, use a 1 MB
+* copy buffer, else use a 128 kB buffer */
+ if (sysconf(_SC_PHYS_PAGES) 32768)
+ bsize = 1*1024*1024;
+ else
+ bsize = 128*1024;
+ } else
+ bsize = MAX(sbuf.st_blksize,
+ (blksize_t)sysconf(_SC_PAGESIZE));
if ((buf = malloc(bsize)) == NULL)
err(1, buffer);
}
?
signature.asc
Description: OpenPGP digital signature
Re: Pipes, cat buffer size
In the last episode (Oct 18), Ivan Voras said: I'm working on a program that's intended to be used as a filter, as in something | myprogram file. I'm trying it with cat and I'm seeing my read()s return small blocks, 64 kB in size. I suppose this is because cat writes in 64 kB blocks. So: a) Is there a way to programatically, per-process, set the pipe buffer size? The program in question is a compressor and it's particularly inefficient when given small blocks and I'm wondering if the system can buffer enough data for it. Why not keep reading until you reach your desired compression block size? Bzip2's default blocksize is 900k, for example. b) Is there any objection to the following patch to cat: It might be simpler to just use dd if=myfile obs=1m instead of patching cat. -- Dan Nelson [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Pipes, cat buffer size
Dan Nelson wrote: In the last episode (Oct 18), Ivan Voras said: I'm working on a program that's intended to be used as a filter, as in something | myprogram file. I'm trying it with cat and I'm seeing my read()s return small blocks, 64 kB in size. I suppose this is because cat writes in 64 kB blocks. So: a) Is there a way to programatically, per-process, set the pipe buffer size? The program in question is a compressor and it's particularly inefficient when given small blocks and I'm wondering if the system can buffer enough data for it. Why not keep reading until you reach your desired compression block size? Bzip2's default blocksize is 900k, for example. Of course. But that's not the point :) From what I see (didn't look at the code), Linux for example does some kind of internal buffering that decouples how the reader and the writer interact. I think that with FreeBSD's current behaviour the writer could write 1-byte buffers and the reader will be forced to read each byte individually. I don't know if there's some ulterior reason for this. b) Is there any objection to the following patch to cat: It might be simpler to just use dd if=myfile obs=1m instead of patching cat. I believe patching cat to bring its block size into the century of the fruitbat has its own benefits. signature.asc Description: OpenPGP digital signature
Re: Pipes, cat buffer size
In the last episode (Oct 19), Ivan Voras said: Dan Nelson wrote: In the last episode (Oct 18), Ivan Voras said: I'm working on a program that's intended to be used as a filter, as in something | myprogram file. I'm trying it with cat and I'm seeing my read()s return small blocks, 64 kB in size. I suppose this is because cat writes in 64 kB blocks. So: a) Is there a way to programatically, per-process, set the pipe buffer size? The program in question is a compressor and it's particularly inefficient when given small blocks and I'm wondering if the system can buffer enough data for it. Why not keep reading until you reach your desired compression block size? Bzip2's default blocksize is 900k, for example. Of course. But that's not the point :) From what I see (didn't look at the code), Linux for example does some kind of internal buffering that decouples how the reader and the writer interact. I think that with FreeBSD's current behaviour the writer could write 1-byte buffers and the reader will be forced to read each byte individually. I don't know if there's some ulterior reason for this. No; take a look at /sys/kern/sys_pipe.c . Depending on how much data is in the pipe, it switches between async in-kernel buffering (8192 bytes), and direct page wiring between sender and receiver (basically zero-copy). b) Is there any objection to the following patch to cat: It might be simpler to just use dd if=myfile obs=1m instead of patching cat. I believe patching cat to bring its block size into the century of the fruitbat has its own benefits. -- Dan Nelson [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
gprocessor Error
Hi: I have been trying to upgrade to the latest gnash on my W/S running: FreeBSD 6.3-RELEASE-p5 #3. First, I upgraded the kernel and base to the latest patch level, then I executed a portupgrade -R gnash, and after all the pre-requisite packages were upgraded successfully gnash itself began to compile. Then this: === ../libcore/.libs/libgnashcore.so: undefined reference to `std::basic_ostreamchar, std::char_traitschar std::basic_ostreamchar, std::char_traitschar ::_M_insertbool(bool)' /usr/ports/graphics/gnash/work/gnash-0.8.4/libbase/.libs/libgnashbase.so: undefined reference to `std::basic_istreamchar, std::char_traitschar std::basic_istreamchar, std::char_traitschar ::_M_extractunsigned int(unsigned int)' collect2: ld returned 1 exit status gmake[2]: *** [gprocessor] Error 1 gmake[2]: Leaving directory `/usr/ports/graphics/gnash/work/gnash-0.8.4/utilities' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/usr/ports/graphics/gnash/work/gnash-0.8.4' gmake: *** [all] Error 2 *** Error code 2 Stop in /usr/ports/graphics/gnash. *** Error code 1 Stop in /usr/ports/graphics/gnash. ** Command failed [exit code 1]: /usr/bin/script -qa /tmp/portupgrade.43253.1 env UPGRADE_TOOL=portupgrade UPGRADE_PORT=gnash-0.8.2_2 UPGRADE_PORT_VER=0.8.2_2 make ** Fix the problem and try again. ** Listing the failed packages (-:ignored / *:skipped / !:failed) ! graphics/gnash (gnash-0.8.2_2)(new compiler error) === What do I need to do to get this accomplished? I am struggling with limited flash support, and have been doing fairly well with gnash/firefox. I would like to upgrade my gnash-0.8.2_2 to the latest version gnash-0.8.4. and possibly get a bit more functionality. Oh, one more possible clue. The first time I attempted this, I issued a portupgrade gnash without recursion. At about the same point of failure, the system powered down. I mean I saw an LD error, and then POOF, laptop went power-down! I didn't believe that was the cause, so after cleaning the file systems, I tried again, and again it powered down. So I pretty much upgraded everything and after the -R attempt, gnash simply failed. TIA Bob ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Pipes, cat buffer size
2008/10/19 Dan Nelson [EMAIL PROTECTED]:
In the last episode (Oct 19), Ivan Voras said:
Of course. But that's not the point :) From what I see (didn't look at
the code), Linux for example does some kind of internal buffering that
decouples how the reader and the writer interact. I think that with
FreeBSD's current behaviour the writer could write 1-byte buffers and
the reader will be forced to read each byte individually. I don't know
if there's some ulterior reason for this.
No; take a look at /sys/kern/sys_pipe.c . Depending on how much data
is in the pipe, it switches between async in-kernel buffering (8192
bytes), and direct page wiring between sender and receiver (basically
zero-copy).
Ok, maybe it's just not behaving as I thought it should. See this test program:
#include sys/fcntl.h
#include stdlib.h
#include stdio.h
#define BSIZE (1024*1024)
void main() {
int r;
char buf[BSIZE];
while (1) {
r = read(0, buf, BSIZE);
fprintf(stderr, read %d bytes\n, r);
if (r = 0)
break;
}
}
and this command line:
dd bs=1 if=/dev/zero| ./reader
The output of this on RELENG_7 is:
read 8764 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
read 1 bytes
...
The first value puzzles me - so it actually is doing some kind of
buffering. Linux isn't actually much better, but the intention is
there:
$ dd if=/dev/zero bs=1 | ./bla
read 1 bytes
read 38 bytes
read 8 bytes
read 2 bytes
read 2 bytes
read 2 bytes
read 2 bytes
read 4 bytes
read 3 bytes
read 2 bytes
read 2 bytes
read 2 bytes
read 2 bytes
read 2 bytes
read 2 bytes
read 3 bytes
read 3 bytes
read 112 bytes
read 2 bytes
read 2 bytes
...
Maybe FreeBSD switches between the writer and the reader too soon so
the buffer doesn't get filled?
Using cat (which started all this), FreeBSD consistently processes
4096 byte buffers, while Linux's sizes are all over the place - from 4
kB to 1 MB, randomly fluctuating. My goal would be (if it's possible -
it might not be) to maximize coalescing in an environment where the
reader does something with the data (e.g. compression) so there should
be a reasonable amount of backlogged input data.
But if it works in general, it may simply be that it isn't really
applicable to my purpose (and I should modify the reader to read
multiple blocks).
Though it won't help me, I still think that modifying cat is worth it :)
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Pipes, cat buffer size
In the last episode (Oct 19), Ivan Voras said: 2008/10/19 Dan Nelson [EMAIL PROTECTED]: In the last episode (Oct 19), Ivan Voras said: Of course. But that's not the point :) From what I see (didn't look at the code), Linux for example does some kind of internal buffering that decouples how the reader and the writer interact. I think that with FreeBSD's current behaviour the writer could write 1-byte buffers and the reader will be forced to read each byte individually. I don't know if there's some ulterior reason for this. No; take a look at /sys/kern/sys_pipe.c . Depending on how much data is in the pipe, it switches between async in-kernel buffering (8192 bytes), and direct page wiring between sender and receiver (basically zero-copy). Ok, maybe it's just not behaving as I thought it should. See this test program: [ program that prints the amount of data in each read() ] and this command line: dd bs=1 if=/dev/zero| ./reader The output of this on RELENG_7 is: read 8764 bytes read 1 bytes [..] read 1 bytes read 1 bytes ... The first value puzzles me - so it actually is doing some kind of buffering. Linux isn't actually much better, but the intention is there: $ dd if=/dev/zero bs=1 | ./bla read 1 bytes read 38 bytes read 8 bytes read 2 bytes [..] read 2 bytes read 3 bytes read 3 bytes read 112 bytes read 2 bytes read 2 bytes ... Maybe FreeBSD switches between the writer and the reader too soon so the buffer doesn't get filled? If your reader isn't doing any real work between reads, it is always reading, so the pipe will never fill up. The delay in FreeBSD was probably due to the shell spawning the writer first, so it buffered up 8k of data before the reader was ready. After that, the reader was able to pull data as fast as the writer pushed. Using cat (which started all this), FreeBSD consistently processes 4096 byte buffers, while Linux's sizes are all over the place - from 4 kB to 1 MB, randomly fluctuating. My goal would be (if it's possible - it might not be) to maximize coalescing in an environment where the reader does something with the data (e.g. compression) so there should be a reasonable amount of backlogged input data. Remember that increasing coelescing also increases latency and decreases the parallelism between reader and writer (since if you coalesce you cause the reader to wait for data that's already been writen, in the hopes that the writer will write again soon). But if it works in general, it may simply be that it isn't really applicable to my purpose (and I should modify the reader to read multiple blocks). That's my suggestion, yes. That way your program would also work when passed data from an internet socket (where you will get varying read() sizes too). It wouldn't add more than 10 lines to wrap your read in a loop that exits when your preferred size has been reached. Though it won't help me, I still think that modifying cat is worth it :) -- Dan Nelson [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Pipes, cat buffer size
... the writer could write 1-byte buffers and the reader will be forced to read each byte individually. No; take a look at /sys/kern/sys_pipe.c . Depending on how much data is in the pipe, it switches between async in-kernel buffering (8192 bytes), and direct page wiring between sender and receiver (basically zero-copy). Ok, maybe it's just not behaving as I thought it should. See this test program: However, when I add sleep(1) to your test program, I see the following output: $ dd bs=1 if=/dev/zero | ./reader read 2282 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes read 65536 bytes In your original example, because your sample reader does nothing, it sees each write separately. If the reader was actually doing some work then the pipe would buffer up data while your reader was busy. This looks like exactly the right behavior: The reader will only block if there is no data in the pipe at all; the writer will only block if it gets too far ahead of the reader. Except in those cases, each program gets to do I/O as fast as it can. If your program needs larger blocks, it should keep reading until it gets enough data. (The -B option of GNU tar is an example of this sort of behavior.) Tim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
