Re: Idea for FreeBSD
On Thu, Aug 07, 2008 at 07:02:30PM +1000, Peter Jeremy wrote: On 2008-Aug-06 19:14:51 -0400, [EMAIL PROTECTED] wrote: In Solaris 10 the Services Management Facility (SMF) was introduced. The main purpose of SMF appears to be to drum up business for Sun's training courses by radically changing Sol10 Administration for little benefit. The main purpose of SMF was to make it possible to programmatically control the system and deal with the myriad of different types of faults from the gazillion different things that people want to run on machines. It's complex because it has to deal with the real world. Basically what it does, is take all the rc.d scripts and puts them into a database to manage. Everything is converted to XML and two basic commands (svcs and svcadm) are used to manage everything. Actually, the inputs to the database are in XML, and this is distilled down to a binary representation in a sqlite database that actually drives the system. While the svccfg and svcadm interfaces do give you a single manner of dealing with the database (svccfg) and then the state of a given service (svcadm). The other thing that the SMF system captures entirely is the dependencies between the different daemons and services. I'm not sure that the rcorder stuff in FreeBSD is quite as complete. It could be, I just don't know. Also, there is versioning for the changes of the sqlite database in Solaris, so you can punt back to a earlier configuration without much hassle. With FreeBSD, I can configure virtually all the system via a single text file - which is easily found and kepy under configuration control. With Sol10, there are random bits of configuration spread all over the system and there is no obvious way to control configuration. Well, realistically, the sum of the files in /etc/rc.d and /usr/local/etc/rc.d are also needed, and you need a snapshot of all of those at a given instant in time to provably know how the system is going to configure when booted. -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Wed, May 14, 2008 at 12:25:16AM -0700, Garrett Cooper wrote: Most of the complaints about other DBs is licensing related, but SQLite's complaint was also the fact that the past stability record was a bit rocky. One other thing to watch for in SQLite is the lack of atomicity in updates. It's not ACID, just like BDB 1.8x isn't ACID. Without a write-ahead log, you cannot be sure that the data written actually made it to stable storage, and as such, you cannot be sure that your database didn't get corrupted when the process stops in a non-optimal way. I view SQLite as adding syntactic sugar to your data access language. It doesn't give you real tranactions, it doesn't give you a write-ahead log, it doesn't give you referential integrity. Heck, the last time I looked (admitted, a while ago), it didn't even enforce column type checking on tables. (Put this string in the INT column? No problem!) -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 10:06:21PM +0100, James Mansion wrote: Kurt J. Lidl wrote: There are known problems with certain keys corrupting the DB 1.8x series code. In fact, the release of the 1.86 was an attempt to solve this problem when the KerberosV people at MIT found a repeatable key insert sequence that would corrupt things. (Or at least that's what I remember, it was a long time ago, and I might have the details wrong.) Have to say its a little concerning that such 'mature' code is actually problematic. Particularly since I'm not aware of a non-LGPL alternative. Do you have anything by way of a pointer? Google didn't help me here. This is somewhat alluded to here: http://www.mail-archive.com/[EMAIL PROTECTED]/msg01560.html You might want to read the entire thread on that issue. There is the comment in the Oracle web pages about 1.86 vs 1.85: http://www.oracle.com/technology/software/products/berkeley-db/db/index.html Do not upgrade to the 1.86 release other than to fix specific hash access method problems found in the 1.85 release. And in the Berkeley DB: A Retrospective paper (http://sites.computer.org/debull/A07Sept/seltzer.pdf), Margo notes: Db-1.85 enjoyed widespread adoption after its release with 4.4BSD. In particular, both the MIT Kerberos project and the University of Michigan LDAP project incorporated it. A group at Harvard released a minor patch version db-1.86 for the Kerberos team in 1996. So, I think my recollection is correct. -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Thu, May 15, 2008 at 05:20:26AM +1000, Peter Jeremy wrote:
On 2008-May-14 09:50:52 -0400, Kurt J. Lidl [EMAIL PROTECTED] wrote:
One other thing to watch for in SQLite is the lack of atomicity
in updates. It's not ACID, just like BDB 1.8x isn't ACID.
This isn't true. SQLite does provide full ACID. One difference from
(eg) Oracle is that you need to explicitly begin a transaction, rather
than a transaction implicitly commencing with the first DML statement.
(I don't know what the SQL standard requires).
Generally, you get either implicit transactions, or you need
to put your database handle into explicit transaction mode,
typically by bracketing your sql with:
begin transaction;
stuff;
stuff;
commit;
if (error) { rollback; whine() }
Without a write-ahead log,
It does have a log to record incomplete transactions.
Well, thanks for the various pointers. I see that it grew a
transaction log since the last time I bothered to look at it
in depth. That's a very good thing.
I'll retract my assertion that it doesn't have a commit log.
-Kurt
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote: My requirements is 1. there is no need for SQL 2. processes are sharing db file in concurrent mode 3. reading/writing = 60%/40% With BDB clause 1 - satisfied clause 3 - satisfied (databases of relatively small items that are changed infrequently). Is there a problem with concurrent access? And, most important, is this the ONLY problem? (I still don't understand because of blurred mention of data corruption.) If concurrency is the only problem then: 1. Can data corruption be avoided? Or this is impossible? 2. How? If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it guarantee for data integrity? This is probably good enough, if your writers also do a fflush() before releasing the lock. If the writing process ever dies horribly, you can still have database corruption. You could turn the thing that does the actual I/O into the database into daemon that fullfils read/write requests from a unix domain socket. In fact, there is a sample application that does exactly that for the newer Sleepycat BDB code. It could probably be adapted or re-written for the 1.8x series code. If it were me, I'd just punt on using the 1.8x code, drop straight into using the BDB 4.6 codebase, and use their transactioning code. It's mature, robust, etc, etc, for a much large set of operations than the 1.8x code ever was. But there's still that licensing issue. -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: BDB corrupt
On Tue, May 13, 2008 at 05:14:52AM -0700, Jeremy Chadwick wrote:
On Tue, May 13, 2008 at 03:44:06PM +0400, Anthony Pankov wrote:
If concurrency is the only problem then:
1. ?an data corruption be avoided? Or this is impossible?
2. How?
Use Sleepycat/Oracle DB instead? The libc DB1.x, despite being
mature, really should be deprecated in some manner.
This catapults back into the arena of stuff that isn't in the
base system. Not to mention I'm not sure that the Oracle BDB
license would allow bundling in the OS as a binary. I doubt it,
but that's a different bikeshed to paint :-)
I'm sure there are others I've forgotten, but the only thing I can think
of in the base system which relies on DB1.x is sendmail (which IMHO
should really be removed from the base system and replaced with a small
standalone mailer -- but that's been discussed in a previous thread in
the past). Even simple ports like postgrey pull in db41, even though
they could technically work fine with DB1.x.
Well, on a 7.0 machine, it looks like the following:
/etc/mail/aliases.db
/etc/{s,}pwd.db
/usr/share/misc/{termcap,vgrindefs}.db
And of course, nvi uses the db code's recno interface
to do all its work.
If all BDB readers would use O_SHLOCK and all writers O_EXLOCK is it
guarantee for data integrity?
The corruption I've seen in the past results in DB operations failing
for no particular reason (what do you mean those are all the records?
No! I just inserted a bunch more!). It turns out some of the data in
the actual .db file is corrupt -- even when locking is used everywhere.
It's as if the code has some weird bug where it'll write bogus data to
the file for some reason.
There are known problems with certain keys corrupting the DB 1.8x
series code. In fact, the release of the 1.86 was an attempt
to solve this problem when the KerberosV people at MIT found
a repeatable key insert sequence that would corrupt things.
(Or at least that's what I remember, it was a long time ago, and
I might have the details wrong.)
-Kurt
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [PATCH] automated make -j value
On Wed, Dec 13, 2006 at 08:19:52PM -0800, David O'Brien wrote:
With multi-socket systems becoming more prevalent, and the continued
increase in cores per processors, I thought it would be nice for
'make -j' to gain some automation.
Attached is a patch that makes -j- be the same as
-j `sysctl -n kern.smp.cpus` and -j= be twice that.
comments? (redirected back to list)
I think you can do it better than this, by just support
setting the concurrancy level via reading a environmental
variable. Like say NPROC, which is what CrayOS used, and
also appeared in the BSD/OS pmake varient several years ago.
Then you can just do this:
export NPROC=`/sbin/sysctl hw.ncpu | awk '{print $3*2}'`
or this:
export NPROC=`/sbin/sysctl hw.ncpu | awk '{print $3}'`
in your shells .rc files.
(Obviously, change the sysctl node as appropriate for your OS.)
I found this really useful when compiling a large tree of
sources, where some of the Makefiles didn't have their dependencies
written correctly, sucht that a parallel make wouldn't work
properly. It's easy to turn off, just by unsetting the
environmental variable. It's also easy to iterate over a set
of values to figure out which one will compile a tree the
fastest. (FYI -- setting 3*hw.ncpu was optimal for BSD/OS.)
If you hack on make to put in automagic around -j, you should
add the environmental variable support too. It's actually more
useful in a lot of cases. (Mostly cause you don't have to
touch any Makefile to turn it on, it just works...)
-Kurt
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: cloning a FreeBSD HDD
On Wed, Mar 29, 2006 at 10:14:19AM -0300, Patrick Tracanelli wrote: Daniel O'Connor wrote: On Wednesday 29 March 2006 14:34, M. Warner Losh wrote: dump + restore is slow but reliabe. Faster than dd for disks that aren't full :) It also gives you a defrag as well as allowing you to change FS options. Yes, pretty much faster for non-full disks, even compared to paralell dd(1). And we always have the -L option to snapshot and dump(1) from live file systems, what makes it an interesting and completly viable choice to clone the disks in multiuser mode (no need to go single user). In a prior life, I had to generate a bunch (50 or 60) disk images from a master server image. The server image was updated periodically, so we decided to always go for doing it on the fly, rather than just restoring a known-good dumpfile from some place. (Questionable in hindsight, but...) Anyhow, we were using SCSI disks, so I got a shelf full of scsi disk canisters (since we had standardized on a particular one) and then wrote a zsh script to do the dumping. Zsh has a particular ability to have it duplicate the contents of a single input stream to multiple output streams. So we would fire up one dump on the master disk, and then pipe the output to multiple copies of restore running (one per disk) simultaneously. It was way faster than doing them sequentially. And, impressive to watch the access lights on the drives when you were making seven disk drives copies at once... -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FS impl.
On Fri, May 06, 2005 at 02:01:35PM -0600, Warner Losh wrote: I have been trying to write my own UFS-like filesystem implementation for fun. I had read somewhere that UFS was developed in user space (correct me if I'm wrong on that one) and then moved over to kernel-space. I was wondering if there are any existing facilities in the kernel source tree that would allow me to develop an fs in user space easily or with a little tweaking? As of right now, I have to develop, compile, panic, reboot, debug etc. which is frustrating and time consuming. Maybe you are thinking of NFS :-). You can use the same hooks that amd and similar programs to implement your code in userland. It's pretty well known that Kirk did the 4.2 FFS implementation as a user-mode process. This statement is made directly in Luke Mewburn's paper on cross-building NetBSD: http://www.usenix.org/events/bsdcon03/tech/full_papers/mewburn/mewburn.pdf (page 9). It doesn't say in the original 4.2 FFS paper. -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: ttyd0/cuad0 - why is there still this duality ?
On Mon, Jan 24, 2005 at 04:16:13PM +0100, Bernd Walter wrote: On Mon, Jan 24, 2005 at 09:30:43AM +0100, Christoph P. Kukulies wrote: Just a question. Maybe it isn't true but to me it seems there is still this duality between ttyd and cuad serial devices. Why is that? I'm just asking because someone I was talking with about modems an comm programs was 'criticising' this fact in FreeBSD while other systems long have abandoned this dualism? Because modems are still used for dial-in and dial-out. tty handing out to getty and cua to the dial out process. Moreover this handling was recently added for usb serials under -current. If other systems loose features - well it's their problem. That's a very limited way of looking at the functionality. If you want to support the functions of both dialin and dialout on one serial port, there doesn't need to be more than one kernel device. Just because support for this got hacked into 4.2BSD in a gross manner doesn't mean that there isn't a better of doing this. Having seperate dialout and dialin devices really are just a kludge for having the kernel doing locking that could be done in userland code. Just because FreeBSD does this the same way it's been done on BSD-ish systems for the last 15 years doesn't mean there isn't a better way of doing it. -Kurt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Where is the source to the system calls?
On Sun, Nov 07, 2004 at 04:29:42PM -0800, Dan Strick wrote: Does anyone know where the system calls are really defined? I followed open() to _open() to __sys_open() which seems to be part of something called libc_r before I ran into a blank wall. I grepped all of the regular files in /usr/src and /usr/include and turned up nothing. I even tried grepping for open in the output of nm -g /usr/lib/libc.a. There is no __sys_open() in libc. Am I dealing with C-compiler magic? Secret macro instructions invoking undocumented gnu C-compiler asm() features? A CIA plot? Is any of this low level structure documented anywhere? Grok the following: /usr/src/lib/libc/sys/Makefile.inc /usr/src/sys/sys/syscall.mk -Kurt ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: please help with: warning: initialization makes integer from pointer
On Fri, Oct 08, 2004 at 12:25:45PM -0500, Sam wrote: Sick! Are there actually systems out there that don't have all-zero NULL pointers? You have officially shattered my previously held beliefs about the sacredness of memset :( If there are, I'd be interested to know of them. See the C Faq, question 1.14: http://www.lysator.liu.se/c/c-faq/c-1.html -Kurt ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Sticky/sgid/suid bits safe on regular files?
On Fri, Jun 18, 2004 at 01:41:05AM +0200, Cyrille Lefevre wrote: Clifton Royston [EMAIL PROTECTED] wrote: [snip] Can anybody confirm for me that the suid, sgid, and sticky bit are in fact no-ops for FreeBSD on regular non-executable files, as it appears they should be? how about the use of hard link which is not os depedent ? Then you have twice as many directory entries for the system to manage, which might be a resource hog if you have many files. -Kurt ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: dd to floppies broken?
On Thu, Aug 21, 2003 at 02:39:44PM -0400, Leo Bicknell wrote: fd0 is block buffered. Try: dd bs=18k of=/dev/rfd0c if=memtest86-2.9/precomp.bin I forget why, but 18k maximizes performance on (some?) floppies. Because a 1440 kbyte floppy has 80 tracks, and it's double sided. Thus, 9kbyte per side, both sides get done at once == 18k per track. So, you schedule a full track of data to be written at one go, the head seeks, and you start the next transfer to the device. -Kurt ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Lower power SMP boxes?
On Sat, Feb 01, 2003 at 10:17:23AM +0100, Mats Larsson wrote: Via just recently announced their new Nehemiah processor capable of smp, presumably slow as its precursor but also the lowest power consuming processor at the market (at least with standard socket fcpga motherboard) [...] http://www.via.com.tw/en/viac3/c3.jsp It says IO/APIC support in future versions. So, it's not an SMP option today, as I understand it. -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Lower power SMP boxes?
On Sat, Feb 01, 2003 at 11:21:49AM -0800, Matthew Dillon wrote: :On Sat, Feb 01, 2003 at 10:17:23AM +0100, Mats Larsson wrote: : Via just recently announced their new Nehemiah processor capable of smp, : presumably slow as its precursor but also the lowest power consuming : processor at the market (at least with standard socket fcpga motherboard) :[...] : http://www.via.com.tw/en/viac3/c3.jsp : :It says IO/APIC support in future versions. So, it's not an SMP option :today, as I understand it. : :-Kurt Although, this is more a deficiency in the way FreeBSD is designed. Using an APIC is nice, but not absolutely necessary. All we need are good specs on how VIA's SMP cpus interact with each other and we could support it. How else are you going to do the physical interrupt steering? Unless they have gone through the effort of implementing a whole new and different steering mechanism -- which would fly in the face of having off-the-shelf OS support from the people in Redmond, at the very least. I like the 11 watts specified in the paper. That *is* low power for the class of system they are selling. I don't see a clock specification but I assume it is going to be at least as fast as the ~900MHz M-9000. It says the new generation VIA C3 processor is available at speeds starting at 1GHz in the paragraph under Enhanced Digital Media Performance. -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Hardlinks...
On Mon, Apr 08, 2002 at 11:41:44AM -0700, Michael Smith wrote: You could also use this technique to maliciously exhaust a user's quota, by linking to their temporary files. I'm not sure what the standards have to say about this, but I don't much like the current behaviour. The truely paranoid ftruncate the file size to zero if the link count is larger than one. -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: EBCDIC - ASCII
On Mon, Feb 12, 2001 at 02:01:55PM +, Aled Morris wrote: On Thu, 1 Feb 2001, Warner Losh wrote: Even the name (dd) comes from IBM's control language (JSYS?). I don't disagree, but someone once told me the name came from what it does "Convert and Copy a file" - see dd(1) - but "cc" was already taken... While not completely on topic, here's the question thread from alt.folklore.computer that started (and answered it): http://www.web.us.uu.net/staff/djm/lore/dd-origin -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Extremely large (70TB) File system/server planning
On Mon, Feb 05, 2001 at 09:50:35AM -0800, Matt Dillon wrote: :70TB is the size of the sum of all files, access or no access. :(They still want to maintain accessibility even though the chances are slim.) This doesn't sound like something you can just throw together with off-the-shelf PCs and still have something reliable to show for it. You need a big honking RAID system - maybe a NetApp, maybe something else. You have to look at the filesystem and file size limitations of the unit and the client(s). NetApp's biggest box can "only" handle 6TB of data, currently, using the latest and greatest software. They claim (and I believe them) that 12TB will be the limit later this year. So FreeBSD could be used as an NFS client, but probably not a server for your application. Considering the number of disks you need to manage, something like a NetApp or other completely self contained RAID-5-capable system for handling the disks is mandatory. Netapps are actually RAID-4 (dedicated parity disk), not RAID-5 (parity data is recorded across all drives). -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: UDP limits in dns server?
On Tue, Nov 21, 2000 at 05:54:06AM +, Tony Finch wrote: I'm looking up the IP addresses with up to 1500 or so processes each taking a list of addresses and running gethostbyaddr() on them. That's stupid. Use adns instead. http://www.chiark.greenend.org.uk/~ian/adns/ That's stupid, use something advanced (and already debugged) built on top of ADNS: http://www.web.us.uu.net/sw/fastresolve/ -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: sfork() ??
On Tue, Nov 21, 2000 at 08:27:06PM -0500, Pedro F Giffuni wrote: rfork comes from plan 9 and along with sfork it wasn't part of the 4.4BSDlite 2 release, OTOH if both are the same, why aren't we referencing it in our syscalls for compatibility with BSDI ? I can't find a reference to sfork elsewhere, but anyone with BSD/OS 2.x or later should know for sure. sfork(), in the not-quite-released BSD/OS 4.2 has gotten an overhaul, to the point where fork() and vfork() are just sfork() wrappers. I've heard the comment before that sfork() is just the BSD way of pronouncing rfork()! So in the 4.2 release, BSD/OS will finally have a fully variable weight fork() call, which would be suitable for implementing all sorts of lightweight processes, threads and so forth. For those unfamiliar with variable weight fork semantics, bastically, you get to specify which parts of the process space you want to share and which parts you want to have a new version of in the child process. So, you can trivially share data, memory, etc... (Stack, of course, cannot be shared :-) The linux clone() call is something similar to sfork()/vfork(). It has some extra grot pasted onto the side to allow for pid hiding, for the wretched hack of a way to do pthreads support there. Chris Torek has described this in amazing detail in the past, while answering a different question. Look for the message with the Subject line "why fork?" in the following digest message: http://faqchest.dynhost.com/prgm/perlu-l/perl-99/perl-9906/perl-990602/perl99061420_14433.html -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Anybody working on FreeBSD BIOS?
On Sat, Jun 17, 2000 at 06:13:32PM +0100, void wrote: On Thu, Jun 15, 2000 at 07:29:53PM -0700, Mike Smith wrote: If your customer's not _desperate_ for a super-low-cost solution, I'd suggest any of the Intel boards that offer EMP (most of these also offer BIOS-over-serial support, actually - as do a number of other vendors, IIRC AMI do this on some of their boards as well). See http://www.realweasel.com/ for one such vendor. Pricing is available only by request, but check this out: Yeah, well, as a supposed customer of theirs, let me add this tidbit. I ordered 5 of these cards for evaluation purposes in mid April. I still do not have them. After having our purchasing people hassle them, they claimed manufacturing difficulties, and the current production run has been pushed to at least the first week in July. So, once again, we are reminded -- just because you saw it on the web, doesn't mean that it is really shipping. I'm still hoping they do ship me the boards and they work as advertised, but I figure I would save other folks some hassle. -Kurt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
