Re: Using shell commands versus C equivalents
Alexander Leidinger wrote:
Quoting Garrett Cooper [EMAIL PROTECTED] (from Tue, 12 Jun
2007 22:55:18 -0700):
Another simple question (I hope):
Is there any reason why shell commands should be used in place of a
C command (in this case chmod via vsystem instead of the chmod(2)
function)? It seems like the fork / exec would be more expensive with
the shell command, but any area where code could be optimized is more
than welcome I would think.
If it is just one file, I don't see the reason to use the shell
command, but if you need to reinvent chmod -R, I don't see a reason
to forbid the use of the shell command (pragmatic programming).
Bye,
Alexander.
Exactly my thinking (ch{own,mod} for one file, not reinventing the
-R 'wheel'). After looking over style(7) I don't see anything about not
doing that.
Thanks for the reply :).
-Garrett
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
building images for running from flash
Just been looking at rebuilding my router. I understand building the image and etc, but the issue I am running into is what to do about ports. To solve this, I am looking at building the image and using mdconfig, unionfs, and jail to solve building ports. Also looking at the possibility of qemu. Any thoughts or suggestions? Another annoying issue is the muckery involved with /etc/make.conf... The issue I am running into that is the CPUTYPE? option is different for the build machine and router that the flash will be used on. Any nice way to deal with this? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Reason for doing malloc / bzero over calloc (performance)?
Garrett Cooper wrote: Title says it all -- is there a particular reason why malloc/bzero should be used instead of calloc? -Garrett As someone just brought to my attention, I should do some Googling. Initial results brought up this: http://boredzo.org/blog/archives/2006-11-26/calloc-vs-malloc. I would like to provide results for CURRENT, but I don't know offhand what C interface right supports nanoseconds or microseconds precision timing in FreeBSD (apart from just doing nanosleeps, which isn't such a great idea and can drift I would think due to clock skew). The original author's solution is for Mac OSX only :(.. I think it's decided though -- calloc for now wins over malloc / bzero, so I'm going to change that alloc/bzero to reflect the change. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Reason for doing malloc / bzero over calloc (performance)?
* Garrett Cooper [EMAIL PROTECTED] wrote:
Garrett Cooper wrote:
Title says it all -- is there a particular reason why malloc/bzero
should be used instead of calloc?
-Garrett
As someone just brought to my attention, I should do some Googling.
Initial results brought up this:
http://boredzo.org/blog/archives/2006-11-26/calloc-vs-malloc.
To be more precise; I took a look at the source code of calloc on my
FreeBSD 6 box:
| void *
| calloc(num, size)
| size_t num;
| size_t size;
| {
| void *p;
|
| if (size != 0 SIZE_T_MAX / size num) {
| errno = ENOMEM;
| return (NULL);
| }
|
| size *= num;
| if ( (p = malloc(size)) )
| bzero(p, size);
| return(p);
| }
This means that the results on that website would be quite different
than the the ones that the FreeBSD 6 malloc/calloc should give. There is
even a difference between calloc'ing 10 block of 10 MB and 1 block of
100 MB, which shouldn't make a difference here. calloc doesn't have any
performance-advantage here, because it just calls malloc/bzero.
When looking at FreeBSD -CURRENT's calloc (won't paste it; too long), it
just does a arena_malloc/memset (which is malloc/bzero) for small
allocations but a huge_malloc for big allocations (say, multiple pages
big). The latter one already returns pages that are zero'd by the
kernel, so I suspect the calloc performance for big allocations on
-CURRENT is a lot better than on FreeBSD 6. As with FreeBSD 6, it
wouldn't matter if you calloc 10 pieces of 10 MB or one piece of 100 MB.
Yours,
--
Ed Schouten [EMAIL PROTECTED]
WWW: http://g-rave.nl/
pgpk3IoTh9vfe.pgp
Description: PGP signature
Re: Using shell commands versus C equivalents
Quoting Garrett Cooper [EMAIL PROTECTED] (from Tue, 12 Jun 2007 22:55:18 -0700): Another simple question (I hope): Is there any reason why shell commands should be used in place of a C command (in this case chmod via vsystem instead of the chmod(2) function)? It seems like the fork / exec would be more expensive with the shell command, but any area where code could be optimized is more than welcome I would think. If it is just one file, I don't see the reason to use the shell command, but if you need to reinvent chmod -R, I don't see a reason to forbid the use of the shell command (pragmatic programming). Bye, Alexander. -- Revenge is a form of nostalgia. http://www.Leidinger.netAlexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: building images for running from flash
On Wed, Jun 13, 2007 at 02:06:42AM -0400, Z.C.B. wrote: Just been looking at rebuilding my router. I understand building the image and etc, but the issue I am running into is what to do about ports. To solve this, I am looking at building the image and using mdconfig, unionfs, and jail to solve building ports. Also looking at the possibility of qemu. Any thoughts or suggestions? Another annoying issue is the muckery involved with /etc/make.conf... The issue I am running into that is the CPUTYPE? option is different for the build machine and router that the flash will be used on. Any nice way to deal with this? nanobsd(8) Eugene Grosbein ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Tue, Jun 12, 2007 at 10:55:18PM -0700, Garrett Cooper wrote: Another simple question (I hope): Is there any reason why shell commands should be used in place of a C command (in this case chmod via vsystem instead of the chmod(2) function)? It seems like the fork / exec would be more expensive with the shell command, but any area where code could be optimized is more than welcome I would think. There often are reasons to prefer using shell commands to C. There typically exist many tradeoffs involved in evaluating one over the other, and machine efficiency is not always the highest goal. (For example, it may be better to reduce complexity in favor of having a simpler structure that is easier for people to understand and maintain with confidence that the changes they make have the desired results. This is, of course, not to try to claim that shell scripts are inherently easier to understand than C code; that would be a silly stance to take.) I commend to your attention Geoff Collyer and Henry Spencer's C News (a successor to Rick Adams' B News) implementation, a great deal of which was written as shell scripts (ca. 1988 or so). (Yes, I realize that that was almost 20 years ago, and that it didn't involve FreeBSD per se. Ignoring the lessons of history is rather short-sighted and foolish: despite using shell scripts for so much of the code, the machine I was then running went from being extremely busy all the time to having a couple of bursts of activity per day for about an hour each time -- with more news flowing with C News vs. B News.) Peace, david -- David H. Wolfskill [EMAIL PROTECTED] Anything and everything is a (potential) cat toy. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgpdvOp2LwU42.pgp Description: PGP signature
Re: building images for running from flash
On Wed, 13 Jun 2007 15:11:15 +0800 Eugene Grosbein [EMAIL PROTECTED] wrote: On Wed, Jun 13, 2007 at 02:06:42AM -0400, Z.C.B. wrote: Just been looking at rebuilding my router. I understand building the image and etc, but the issue I am running into is what to do about ports. To solve this, I am looking at building the image and using mdconfig, unionfs, and jail to solve building ports. Also looking at the possibility of qemu. Any thoughts or suggestions? Another annoying issue is the muckery involved with /etc/make.conf... The issue I am running into that is the CPUTYPE? option is different for the build machine and router that the flash will be used on. Any nice way to deal with this? nanobsd(8) I am aware of it, but it does not cover what I need for either questions, except for possibly the make part. From the looks of it, I assume I can override what make.conf is used using __MAKE_CONF? Also any thing to be aware of when using DESTDIR with make kernel? Just tried it here and it did not work. It just installed over my current one. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: building images for running from flash
On Wed, Jun 13, 2007 at 09:07:45AM -0400, Z.C.B. wrote: nanobsd(8) I am aware of it, but it does not cover what I need for either questions, except for possibly the make part. It does. Eugene ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
--Rgf3q3z9SdmXC6oT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 12, 2007 at 10:55:18PM -0700, Garrett Cooper wrote: Another simple question (I hope): Is there any reason why shell commands should be used in place of a=20 C command (in this case chmod via vsystem instead of the chmod(2)=20 function)? It seems like the fork / exec would be more expensive with=20 the shell command, but any area where code could be optimized is more=20 than welcome I would think. There often are reasons to prefer using shell commands to C. There typically exist many tradeoffs involved in evaluating one over the other, and machine efficiency is not always the highest goal. (For example, it may be better to reduce complexity in favor of having a simpler structure that is easier for people to understand and maintain with confidence that the changes they make have the desired results. This is, of course, not to try to claim that shell scripts are inherently easier to understand than C code; that would be a silly stance to take.) I commend to your attention Geoff Collyer and Henry Spencer's C News (a successor to Rick Adams' B News) implementation, a great deal of which was written as shell scripts (ca. 1988 or so). (Yes, I realize that that was almost 20 years ago, and that it didn't involve FreeBSD per se. Ignoring the lessons of history is rather short-sighted and foolish: despite using shell scripts for so much of the code, the machine I was then running went from being extremely busy all the time to having a couple of bursts of activity per day for about an hour each time -- with more news flowing with C News vs. B News.) read the question again, though it is not absolutely clear/correct, the question was: chmod(path, mode) vs system(chmod ...) and not wheather to write a program or a shell script. danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: building images for running from flash
On Wed, 13 Jun 2007 21:42:12 +0800 Eugene Grosbein [EMAIL PROTECTED] wrote: On Wed, Jun 13, 2007 at 09:07:45AM -0400, Z.C.B. wrote: nanobsd(8) I am aware of it, but it does not cover what I need for either questions, except for possibly the make part. It does. Hmmm on a interesting note, make kernel with DESTDIR does not work, but installkernel does. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
In [EMAIL PROTECTED], Garrett Cooper [EMAIL PROTECTED] typed:
Alexander Leidinger wrote:
Quoting Garrett Cooper [EMAIL PROTECTED] (from Tue, 12 Jun
2007 22:55:18 -0700):
Another simple question (I hope):
Is there any reason why shell commands should be used in place of a
C command (in this case chmod via vsystem instead of the chmod(2)
function)? It seems like the fork / exec would be more expensive with
the shell command, but any area where code could be optimized is more
than welcome I would think.
If it is just one file, I don't see the reason to use the shell
command, but if you need to reinvent chmod -R, I don't see a reason
to forbid the use of the shell command (pragmatic programming).
Bye,
Alexander.
Exactly my thinking (ch{own,mod} for one file, not reinventing the
-R 'wheel'). After looking over style(7) I don't see anything about not
doing that.
While I agree in principal - if chmod already implements some
hard-to-implement functionality, then using it only makes sense - I
don't think -R qualifies. It's implemented via the fts library, which
makes recursing through a directory tree about as difficult as
scanning a directory. In fact, fts has features that dir* doesn't, so
there are instances where fts is preferable to the dir* routines for
scanning a directory.
mike
--
Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: building images for running from flash
In [EMAIL PROTECTED], Z.C.B. [EMAIL PROTECTED] typed: Another annoying issue is the muckery involved with /etc/make.conf... The issue I am running into that is the CPUTYPE? option is different for the build machine and router that the flash will be used on. Any nice way to deal with this? If they are different implementations of the same architecture, you can set CPUTYPE to the lower of the two. For the kernel and world, you can set TARGET_ARCH= on the make line, and it will cross-compile the world for you (assuming your compiler can generate code for that architecture. mike -- Mike Meyer [EMAIL PROTECTED] http://www.mired.org/consulting.html Independent Network/Unix/Perforce consultant, email for more information. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
Danny Braniss wrote: --Rgf3q3z9SdmXC6oT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 12, 2007 at 10:55:18PM -0700, Garrett Cooper wrote: Another simple question (I hope): Is there any reason why shell commands should be used in place of a=20 C command (in this case chmod via vsystem instead of the chmod(2)=20 function)? It seems like the fork / exec would be more expensive with=20 the shell command, but any area where code could be optimized is more=20 than welcome I would think. There often are reasons to prefer using shell commands to C. There typically exist many tradeoffs involved in evaluating one over the other, and machine efficiency is not always the highest goal. (For example, it may be better to reduce complexity in favor of having a simpler structure that is easier for people to understand and maintain with confidence that the changes they make have the desired results. This is, of course, not to try to claim that shell scripts are inherently easier to understand than C code; that would be a silly stance to take.) I commend to your attention Geoff Collyer and Henry Spencer's C News (a successor to Rick Adams' B News) implementation, a great deal of which was written as shell scripts (ca. 1988 or so). (Yes, I realize that that was almost 20 years ago, and that it didn't involve FreeBSD per se. Ignoring the lessons of history is rather short-sighted and foolish: despite using shell scripts for so much of the code, the machine I was then running went from being extremely busy all the time to having a couple of bursts of activity per day for about an hour each time -- with more news flowing with C News vs. B News.) read the question again, though it is not absolutely clear/correct, the question was: chmod(path, mode) vs system(chmod ...) and not wheather to write a program or a shell script. danny Sorry -- actually I meant that (along similar lines), there was a program with the following lines: vsystem(/bin/chmod +x %s, filename); and I replaced it with: chmod(filename, (mode_t) ( S_IXUSR | S_IXGRP | S_IXOTH )); Probably won't yield much gain overall, but every drop counts and there are quite a few iterations performed in the pkg_* programs, in particular dealing with X.org. Next step, eliminating the linked list structure in favor or red-black trees, maybe. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, Jun 13, 2007 at 08:26:38AM -0700, Garrett Cooper wrote: Sorry -- actually I meant that (along similar lines), there was a program with the following lines: vsystem(/bin/chmod +x %s, filename); and I replaced it with: chmod(filename, (mode_t) ( S_IXUSR | S_IXGRP | S_IXOTH )); You supposedly mean stat + chmod here, right? Trivial errors like this are easy to make. Next step, eliminating the linked list structure in favor or red-black trees, maybe. Due to the way pkg_install works, this most likely is just adding complexity for no gain, it might actually slow it down. Joerg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
Sorry -- actually I meant that (along similar lines), there was a program with the following lines: vsystem(/bin/chmod +x %s, filename); and I replaced it with: chmod(filename, (mode_t) ( S_IXUSR | S_IXGRP | S_IXOTH )); Those two lines have different effects. The first adds the execute bit to the file. The second replaces the current permissions with only the execute bit. To have the same affect, you need to stat() the file, and then use bitwise-or to add the S_IXUSR | S_IXGRP | S_IXOTH bits to st_mode and use that new st_mode with chmod(). ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, Jun 13, 2007 at 08:26:38AM -0700, Garrett Cooper wrote: vsystem(/bin/chmod +x %s, filename); and I replaced it with: chmod(filename, (mode_t) ( S_IXUSR | S_IXGRP | S_IXOTH )); Another improvement made by using stat(2)/chmod(2) over chmod(1) using system(3) variants is the protection against malicious filenames. The original code should have used fork/execv instead anyway. But yeah, saving the fork, time to load and execute the shell, and parse and perform the chmod is probably worth the effort to use the syscalls directly from C, especially if this operation happens often enough. I agree with the other poster(s) who said that if it prevents complex code, it might be worth it. Your case isn't complex enough to warrant the spawning of a shell process, especially when one of your primary goals is to reduce total runtime. -- Rick C. Petty ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, Jun 13, 2007 at 11:15:52AM -0500, Rick C. Petty wrote: Another improvement made by using stat(2)/chmod(2) over chmod(1) using system(3) variants is the protection against malicious filenames. The original code should have used fork/execv instead anyway. To be precise, this case should use open/fstat/fchmod to avoid another bunch of race conditions. Joerg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, 13 Jun 2007, Joerg Sonnenberger wrote: On Wed, Jun 13, 2007 at 08:26:38AM -0700, Garrett Cooper wrote: Sorry -- actually I meant that (along similar lines), there was a program with the following lines: vsystem(/bin/chmod +x %s, filename); and I replaced it with: chmod(filename, (mode_t) ( S_IXUSR | S_IXGRP | S_IXOTH )); You supposedly mean stat + chmod here, right? Trivial errors like this are easy to make. Yes . Good catch, thanks :). Next step, eliminating the linked list structure in favor or red-black trees, maybe. Due to the way pkg_install works, this most likely is just adding complexity for no gain, it might actually slow it down. Hmmm... the only thing is that it does the linked list traversal a number of times per dependency. I'll take a look in gdb at the size of each dependency and then confer with Kirill (my mentor) about that a bit more. -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, 13 Jun 2007, Joerg Sonnenberger wrote: On Wed, Jun 13, 2007 at 11:15:52AM -0500, Rick C. Petty wrote: Another improvement made by using stat(2)/chmod(2) over chmod(1) using system(3) variants is the protection against malicious filenames. The original code should have used fork/execv instead anyway. To be precise, this case should use open/fstat/fchmod to avoid another bunch of race conditions. Joerg Should I briefly lock (flock) the file when running open/fstat/fchmod then to avoid issues? This may become a problem as pkg_*/make becomes more parallelized (another student's goals for his SoC project). Needless to say, pkg_* is by no means threadsafe in its current form though. It uses some global vars that are currently not mutex locked, and this type of file access is another issue (I wonder if spinlocking or sleeping waiting for flock to finish would be better in this case). -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Reason for doing malloc / bzero over calloc (performance)?
On Wed, 13 Jun 2007, Ed Schouten wrote:
* Garrett Cooper [EMAIL PROTECTED] wrote:
Garrett Cooper wrote:
Title says it all -- is there a particular reason why malloc/bzero
should be used instead of calloc?
-Garrett
As someone just brought to my attention, I should do some Googling.
Initial results brought up this:
http://boredzo.org/blog/archives/2006-11-26/calloc-vs-malloc.
To be more precise; I took a look at the source code of calloc on my
FreeBSD 6 box:
| void *
| calloc(num, size)
| size_t num;
| size_t size;
| {
| void *p;
|
| if (size != 0 SIZE_T_MAX / size num) {
| errno = ENOMEM;
| return (NULL);
| }
|
| size *= num;
| if ( (p = malloc(size)) )
| bzero(p, size);
| return(p);
| }
This means that the results on that website would be quite different
than the the ones that the FreeBSD 6 malloc/calloc should give. There is
even a difference between calloc'ing 10 block of 10 MB and 1 block of
100 MB, which shouldn't make a difference here. calloc doesn't have any
performance-advantage here, because it just calls malloc/bzero.
When looking at FreeBSD -CURRENT's calloc (won't paste it; too long), it
just does a arena_malloc/memset (which is malloc/bzero) for small
allocations but a huge_malloc for big allocations (say, multiple pages
big). The latter one already returns pages that are zero'd by the
kernel, so I suspect the calloc performance for big allocations on
-CURRENT is a lot better than on FreeBSD 6. As with FreeBSD 6, it
wouldn't matter if you calloc 10 pieces of 10 MB or one piece of 100 MB.
Yours,
--
Ed Schouten [EMAIL PROTECTED]
WWW: http://g-rave.nl/
Hmmm... I wonder what the Mach kernel in OSX does to allocate memory then. I'll
have to take a look at OpenDarwin's source sometime and see what it does.
-Garrett
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, Jun 13, 2007 at 10:23:36AM -0700, [EMAIL PROTECTED] wrote: Should I briefly lock (flock) the file when running open/fstat/fchmod then to avoid issues? This may become a problem as pkg_*/make becomes more parallelized (another student's goals for his SoC project). Looking does not change the issue. The problem here to protect against is that a process renames a file between the stat and the chmod. See the classic tmp file class of security vulnerabilities. Needless to say, pkg_* is by no means threadsafe in its current form though. It uses some global vars that are currently not mutex locked, and this type of file access is another issue (I wonder if spinlocking or sleeping waiting for flock to finish would be better in this case). I'm perfectly aware of the state of pkg_install. I also believe that it is a bad idea to parallelise it, but I don't want to argue with FreeBSD/Ports about that. Joerg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: building images for running from flash
2007/6/13, Z.C.B. [EMAIL PROTECTED]: Just been looking at rebuilding my router. I understand building the image and etc, but the issue I am running into is what to do about ports. To solve this, I am looking at building the image and using mdconfig, unionfs, and jail to solve building ports. Also looking at the possibility of qemu. Any thoughts or suggestions? Another annoying issue is the muckery involved with /etc/make.conf... The issue I am running into that is the CPUTYPE? option is different for the build machine and router that the flash will be used on. Any nice way to deal with this? tinybsd.org ;-) ports/sysutils/tinybsd -- Zavam, VinÃcius ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, Jun 13, 2007 at 10:23:36AM -0700, [EMAIL PROTECTED] wrote: On Wed, 13 Jun 2007, Joerg Sonnenberger wrote: Should I briefly lock (flock) the file when running open/fstat/fchmod then to avoid issues? This may become a problem as pkg_*/make becomes more parallelized (another student's goals for his SoC project). I wouldn't bother. What issue are you trying to avoid? If one process is trying to chmod +x and another is trying to do a chmod -x, it shouldn't matter if you lock between the fstat/fchmod-- someone is going to win anyway. This operation is not something that needs to be thread-safe. Needless to say, pkg_* is by no means threadsafe in its current form though. It uses some global vars that are currently not mutex locked, and this type of file access is another issue (I wonder if spinlocking or sleeping waiting for flock to finish would be better in this case). Does pkg_* use multiple threads? I was under the impression each pkg tool used a single thread (i.e. no threads) to do its operations and that they wait for system(2)-type calls as needed. Maybe I'm not clear by what you mean when you say global vars. Now another question is whether the pkg_* tools can handle multiple processes managing the ports at the same time. For the mostpart, this is true. Without looking at the code, I would expect that the only contentions would be when trying to update the +REQUIRED_BY files. Everything else should be just fine; you're not supposed to be installing the same port multiple times at the exact same time, but maybe a lock could be held on the package directory (i.e. /var/db/pkg/$PKG_NAME). Again, I don't believe this is strictly necessary. -- Rick C. Petty ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
Next step, eliminating the linked list structure in favor or red-black trees, maybe. Due to the way pkg_install works, this most likely is just adding complexity for no gain, it might actually slow it down. Hmmm... the only thing is that it does the linked list traversal a number of times per dependency. I'll take a look in gdb at the size of each dependency and then confer with Kirill (my mentor) about that a bit more. If you tend to look for the same few items over and over, just move items to the front of the list as soon as you find them. Alternatively, if you know you won't look for this item ever again, move it to the end of the list or remove it from the list entirely. Simple tricks like this can greatly speed things up with almost no effort. Better yet, do easy things like this first; if they help, then look at more complex approaches. As Joerg said, though, you're not likely to gain much from this. pkg_install is almost entirely disk bound. I spent a lot of time recently in libarchive/bsdtar optimizing the disk handling; I can share lots of ideas for improving performance of disk-bound operations like this. Tim Kientzle ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, 13 Jun 2007, Rick C. Petty wrote: On Wed, Jun 13, 2007 at 10:23:36AM -0700, [EMAIL PROTECTED] wrote: On Wed, 13 Jun 2007, Joerg Sonnenberger wrote: Should I briefly lock (flock) the file when running open/fstat/fchmod then to avoid issues? This may become a problem as pkg_*/make becomes more parallelized (another student's goals for his SoC project). I wouldn't bother. What issue are you trying to avoid? If one process is trying to chmod +x and another is trying to do a chmod -x, it shouldn't matter if you lock between the fstat/fchmod-- someone is going to win anyway. This operation is not something that needs to be thread-safe. Needless to say, pkg_* is by no means threadsafe in its current form though. It uses some global vars that are currently not mutex locked, and this type of file access is another issue (I wonder if spinlocking or sleeping waiting for flock to finish would be better in this case). Does pkg_* use multiple threads? I was under the impression each pkg tool used a single thread (i.e. no threads) to do its operations and that they wait for system(2)-type calls as needed. Maybe I'm not clear by what you mean when you say global vars. Well, I mean that as it currently stands there are several variables used globally for setting attributes per package (I'm not at my machine right now so I can't look them up until ~6pm PST). Now another question is whether the pkg_* tools can handle multiple processes managing the ports at the same time. For the mostpart, this is true. Without looking at the code, I would expect that the only contentions would be when trying to update the +REQUIRED_BY files. Everything else should be just fine; you're not supposed to be installing the same port multiple times at the exact same time, but maybe a lock could be held on the package directory (i.e. /var/db/pkg/$PKG_NAME). Again, I don't believe this is strictly necessary. Currently, no, and this is a condition that's contingent for a fellow SoC'er's project. The mentor said that all that *should* occur is there should be an flock, but that was it. So instead of making more work for him and since I am modifying pkg_* already, I thought it would be best to just make my modifications to simplify his end (he still has a ways to go on the dependency tracking I think). It goes a bit deeper than the +REQUIRED_BY files, in particular with the +CONTENTS, etc files as the pkg_* tools are enumerating the packages currently on the system, their dependencies, owning files, etc. Perhaps a global .lock file of some kind in the package directories would be the way to go though. Thanks, -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, 13 Jun 2007, Tim Kientzle wrote: Next step, eliminating the linked list structure in favor or red-black trees, maybe. Due to the way pkg_install works, this most likely is just adding complexity for no gain, it might actually slow it down. Hmmm... the only thing is that it does the linked list traversal a number of times per dependency. I'll take a look in gdb at the size of each dependency and then confer with Kirill (my mentor) about that a bit more. If you tend to look for the same few items over and over, just move items to the front of the list as soon as you find them. Alternatively, if you know you won't look for this item ever again, move it to the end of the list or remove it from the list entirely. Simple tricks like this can greatly speed things up with almost no effort. Better yet, do easy things like this first; if they help, then look at more complex approaches. As Joerg said, though, you're not likely to gain much from this. pkg_install is almost entirely disk bound. I spent a lot of time recently in libarchive/bsdtar optimizing the disk handling; I can share lots of ideas for improving performance of disk-bound operations like this. Tim Kientzle Thank you Tim. I'll definitely keep that in mind :). -Garrett PS I'm looking at pkg_install and pkg_version mostly, but I'll be looking into the other package utilities closely in the next couple weeks, evaluating what approaches I should take in solving some bottlenecks with installing packages and ports. My goals are up on http://wiki.freebsd.org/GarrettCooper, and will be modified soon. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: building images for running from flash
On Wed, 13 Jun 2007 09:58:50 -0400 Mike Meyer [EMAIL PROTECTED] wrote: In [EMAIL PROTECTED], Z.C.B. [EMAIL PROTECTED] typed: Another annoying issue is the muckery involved with /etc/make.conf... The issue I am running into that is the CPUTYPE? option is different for the build machine and router that the flash will be used on. Any nice way to deal with this? If they are different implementations of the same architecture, you can set CPUTYPE to the lower of the two. For the kernel and world, you can set TARGET_ARCH= on the make line, and it will cross-compile the world for you (assuming your compiler can generate code for that architecture. Setting CPUTYPE in /etc/make.conf is exactly what I am trying to avoid. The big question is still what to do about the ports. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: kern/83807: [sis] [patch] if_sis: Wake On Lan support for FreeBSD
On Sun, Jun 10, 2007 at 04:45:04PM +0200, Stefan Sperling wrote:
I usually use either if_em or if_xl chipsets, so I hoped landing this code
in at least -CURRENT (should go there first, I guess) would result in more
chipsets supported ;)
There is code for enabling wake on lan in the Linux drivers for both
if_xl and if_em cards. See drivers/net/3c59x.c and
drivers/net/e1000/e1000_ethtool.c in the linux source tree.
So I don't think adding support for these cards is a problem.
Just need to find some time to do it...
Updated patch attached. Also available at http://stsp.name/wol/
Now contains untested support for if_xl. Please test.
I have no such card so I cannot test this myself.
Note that apparently only 3C905B type cards support wake on lan.
And they only support magic packet events, no unicast, broadcast
or multicast events.
The patch is against RELENG_6_2. Sorry I have no -CURRENT system
set up at the moment. I don't know if the patch even applies to
-CURRENT.
Once if_xl is working I'll look into getting if_em supported.
If any nve users are reading this, I still need feedback
on if_nve as well. I cannot test nve myself either.
Thanks,
--
stefan
http://stsp.name PGP Key: 0xF59D25F0
Index: sbin/ifconfig/Makefile
===
RCS file: /usr/local/ncvs/src/sbin/ifconfig/Makefile,v
retrieving revision 1.29
diff -u -u -r1.29 Makefile
--- sbin/ifconfig/Makefile 5 Jun 2005 03:32:51 - 1.29
+++ sbin/ifconfig/Makefile 6 May 2006 11:08:41 -
@@ -28,6 +28,8 @@
SRCS+= ifbridge.c # bridge support
+SRCS+= ifwol.c # wake on lan support
+
.if !defined(RELEASE_CRUNCH)
SRCS+= af_ipx.c # IPX support
DPADD= ${LIBIPX}
Index: sbin/ifconfig/ifconfig.8
===
RCS file: /usr/local/ncvs/src/sbin/ifconfig/ifconfig.8,v
retrieving revision 1.95.2.17
diff -u -u -r1.95.2.17 ifconfig.8
--- sbin/ifconfig/ifconfig.8 3 Nov 2006 09:14:24 - 1.95.2.17
+++ sbin/ifconfig/ifconfig.8 13 Jan 2007 12:18:33 -
@@ -939,6 +939,27 @@
If that is the case, then the first four keys
(1-4) will be the standard temporary keys and any others will be adaptor
specific keys such as permanent keys stored in NVRAM.
+.It Cm wakeon Ar events
+Enable Wake On Lan support, if available. The
+.Ar events
+argument is a comma seperated list of package types that shall
+trigger wake events. The set of valid package types is
+.Dq Li unicast ,
+.Dq Li multicast ,
+.Dq Li broadcast ,
+and
+.Dq Li magic .
+These enable wake on unicast, multicast, broadcast and Magic Packet(tm),
+respectively.
+A SecureOn password, if supported, can be be enabled using the
+.Dq Li sopasswd:password
+event.
+SecureOn passwords only work in combination with
+.Dq Li magic .
+The password must consist of 12 hexadecimal digits.
+.It Fl wakeon
+Disable Wake On Lan.
+.Pp
.It Cm wme
Enable Wireless Multimedia Extensions (WME) support, if available,
for the specified interface.
@@ -946,7 +967,6 @@
efficient communication of realtime and multimedia data.
To disable WME support, use
.Fl wme .
-.Pp
The following parameters are meaningful only when WME support is in use.
Parameters are specified per-AC (Access Category) and
split into those that are used by a station when acting
Index: sbin/ifconfig/ifwol.c
===
RCS file: sbin/ifconfig/ifwol.c
diff -N sbin/ifconfig/ifwol.c
--- /dev/null 1 Jan 1970 00:00:00 -
+++ sbin/ifconfig/ifwol.c 20 Oct 2006 10:29:10 -
@@ -0,0 +1,229 @@
+/* $Id$ */
+
+/*
+ * Copyright (c) 2005 Stefan Sperling.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ *documentation and/or other materials provided with the distribution.
+ * 3. The name of the author may not be used to endorse or promote products
+ *derived from this software without specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
+ * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+ * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
+ * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+ * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+ * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT
Re: Using shell commands versus C equivalents
On Wed, 13 Jun 2007, [EMAIL PROTECTED] wrote: PS I'm looking at pkg_install and pkg_version mostly, but I'll be looking into the other package utilities closely in the next couple weeks, evaluating what approaches I should take in solving some bottlenecks with installing packages and ports. My goals are up on http://wiki.freebsd.org/GarrettCooper, and will be modified soon. Since you are interested in speeding up the pkg_* stuff, I thought I would bring these to your attention (speed ups to the pkg_create and port registration processes, not quite what you are doing, but related). http://www.freebsd.org/cgi/query-pr.cgi?pr=112765 http://www.freebsd.org/cgi/query-pr.cgi?pr=112630 Let me also wish you good luck in your speed improvements. I was rather encouraged by these two, but all my future searches proved somewhat negative. In particular I looked rather hard at trying to speed up make (required for pkg_version when it calls make -V PKGNAME multiple times). I can tell you that in that instance, replacing linear lists by berkeley DB made no difference at all - it was very disappointing. In short, you really want to do a lot of profiling and see where the speed bottlenecks really are before writing lots of code. Stephen ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Using shell commands versus C equivalents
On Wed, 13 Jun 2007, Stephen Montgomery-Smith wrote: On Wed, 13 Jun 2007, [EMAIL PROTECTED] wrote: PS I'm looking at pkg_install and pkg_version mostly, but I'll be looking into the other package utilities closely in the next couple weeks, evaluating what approaches I should take in solving some bottlenecks with installing packages and ports. My goals are up on http://wiki.freebsd.org/GarrettCooper, and will be modified soon. Since you are interested in speeding up the pkg_* stuff, I thought I would bring these to your attention (speed ups to the pkg_create and port registration processes, not quite what you are doing, but related). http://www.freebsd.org/cgi/query-pr.cgi?pr=112765 http://www.freebsd.org/cgi/query-pr.cgi?pr=112630 Let me also wish you good luck in your speed improvements. I was rather encouraged by these two, but all my future searches proved somewhat negative. In particular I looked rather hard at trying to speed up make (required for pkg_version when it calls make -V PKGNAME multiple times). I can tell you that in that instance, replacing linear lists by berkeley DB made no difference at all - it was very disappointing. In short, you really want to do a lot of profiling and see where the speed bottlenecks really are before writing lots of code. Stephen Stephen, I have no control over the Makefile revisions, but I'll talk with Kirill about the pkg_* integration / testing, and be sure to note you on the change if it is integrated. I'll take your comments into consideration though :). -Garrett ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Disk block or sector to file mapping?
Greetings, I have a drive that failed and fsck and dump both report the failed sector or block (the term seems to be used interchangeably at times), but how can I find out what file(s) were using that block? I have a file-based backup and I could possibly replace the bad files if I know which ones were affected by the bad blocks. Thanks, Matthew ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Disk block or sector to file mapping?
On Wed, Jun 13, 2007 at 11:14:03PM -0400, Matthew Hagerty wrote: Greetings, I have a drive that failed and fsck and dump both report the failed sector or block (the term seems to be used interchangeably at times), but how can I find out what file(s) were using that block? I have a file-based backup and I could possibly replace the bad files if I know which ones were affected by the bad blocks. There's apparently a way to work out what block on the disk is used by a specific inode using some math and numerous parameters taken from the drive, filesystems, and other such things. It might be mentioned in the URL I've included below (for Linux though, not BSD), so I'd peek there. Anyways, I'd do the following: * Run the disk manufacturer's native disk analysis utility. Many of them will do some extra magic (particularly for PATA/SATA disks; with SCSI there's no magic, you can do it yourself by manipulating the grown defect list) to try and work around a full bad block/remapped sector list. Besides, when RMA'ing the disk, the manu. will usually ask if you've run their analysis tool and what the result was. * You might be able to use smartctl (ports/sysutils/smartmontools) to run a selective LBA test (smartctl -t select,X-Y /dev/adN, where X-Y are starting and ending LBAs to do checks on). Not all drives support this though. If select isn't permitted, you can try -t long which should work on most disks, but scans the entire disk (takes a long time). Then you can use smartctl -a /dev/adN and see if the last test you ran was successful or if an error was encountered, hopefully what LBA it's at. This document might also come in handy: http://smartmontools.sourceforge.net/badblockhowto.html * There's also ports/sysutils/drivecheckd which I've never used, but looks like it might possibly provide more detailed info. * The purpose of doing any of the above is to try and get the drive mark the block in question as bad, thus not access it any longer. It may have already done that when the OS reported an issue[1]. That should (hopefully) cause fsck to notice inconsistencies in filesystem data, and give you a filename that used the aforementioned block, telling you the file is inaccessible or should move to lost+found and so on. (I'm sure someone will correct me on the last part :) ) * Now try fsck -f on each unmounted filesystem and see if any errors come up, with filenames referenced. Realistically, what we need on FreeBSD is a tool similar to Solaris's format(8) analyze command, which does a raw disk scan (r, r/w, and a couple other operations). For those not familiar with it, I'll include a sample session of a disk being analysed at the bottom of this Email. Sorry if this is too verbose, but I quite often deal with disks going bad during my day job. [1] - If the OS is seeing bad blocks on a PATA/SATA disk, usually it means that the internal remapping table is full, which means that there were other bad blocks on the disk which it has silently remapped for you to avoid pain -- and space for those blocks has been exhausted. Sometimes you can work around this as mentioned, but most of the time you can't, and you're stuck simply replacing the disk entirely. Bad blocks have a tendency to spread too... -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | bash# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t0d0 DEFAULT cyl 4464 alt 2 hd 255 sec 63 /[EMAIL PROTECTED],0/pci8086,[EMAIL PROTECTED]/pci8086,[EMAIL PROTECTED]/pci9005,[EMAIL PROTECTED]/[EMAIL PROTECTED],0 Specify disk (enter its number): 0 selecting c0t0d0 [disk formatted] Warning: Current Disk has mounted partitions. FORMAT MENU: disk - select a disk type - select (define) a disk type partition - select (define) a partition table current- describe the current disk format - format and analyze the disk fdisk - run the fdisk program repair - repair a defective sector label - write label to the disk analyze- surface analysis defect - defect list management backup - search for backup labels verify - read and display labels save - save new disk/partition definitions inquiry- show vendor, product and revision volname- set 8-character volume name !cmd - execute cmd, then return quit format analyze ANALYZE MENU: read - read only test (doesn't harm SunOS) refresh - read then write (doesn't harm data) test - pattern testing (doesn't harm data) write- write then read (corrupts data)
Re: HP SmartArray ( CISS ) and CAMCONTROL
On Mon, Jun 11, 2007 at 06:55:58PM -0400, Mark Saad wrote: Wilko Thats great to hear hpacucli and hpasm would rock. Wilko Bulte wrote: On Mon, Jun 11, 2007 at 06:35:28PM -0400, Mark Saad wrote.. Matt What will raidutill offer ? Also does anyone know if HP provides hpacucli for FreeBSD ,but only to Japanese customers ? Well.. we just landed ourselves a new committer from HP India who is sponsored by HP ISS in Houston to provide (binary only) Proliant mgmt / monitoring tools for FreeBSD. In due time you will see his work I imagine. I hope this will be the start of a fruitful vendor support. Wooohohohohohooh! I would love to see those. It would give me reason to convert some linux boxes to FreeBSD. -- Mark Saad Managed UNIX Support DataPipe Managed Global IT Services [EMAIL PROTECTED] 1.201.792.4847 (international) 1.888.749.5821 (toll free) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] -- Regards, Ulf. - Ulf Zimmermann, 1525 Pacific Ave., Alameda, CA-94501, #: 510-865-0204 You can find my resume at: http://www.Alameda.net/~ulf/resume.html ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Disk block or sector to file mapping?
Hello Matthew, You wrote on Thursday, June 14, 2007, 7:14:03 AM: MH I have a drive that failed and fsck and dump both report the failed MH sector or block (the term seems to be used interchangeably at times), MH but how can I find out what file(s) were using that block? fsdb findblk disk_block_number -- Anton Yuzhaninov.
Re: Using shell commands versus C equivalents
I spent a lot of time recently in libarchive/bsdtar optimizing the disk handling; I can share lots of ideas for improving performance of disk-bound operations like this. One thing you might find useful: libarchive has an API for creating things on disk, which handles a lot of trivia (creating parent directories transparently, restoring permissions, users, file flags, etc.). It is, of course, especially useful if you're pulling things out of tar archives, but can be used separately from that. Since libarchive is a standard part of FreeBSD, you should feel free to use it. Tim K ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
