Re: (Semi-random) thoughts on device tree structure and devfs
On Mon, Mar 08, 2010 at 12:34:23PM -0700, Ted Lemon wrote: > On Mar 8, 2010, at 12:47 AM, Masao Uebayashi wrote: > > How do you write a kernel config which can always identify my USB disk as > > sd0a, even if I plug random devices? > > You'd need to put the UUID in the kernel config. I'd go further and say that we should be able to supply a set of device properties (such as drvctl -p prints) to the kernel. Let us match a device by its intrinsic properties (MAC address, serial number, and/or GUID), and set the unit number according to the device property. Quentin is right that this *only* helps us to fix the unit number, but I think that in itself is an important, *feasible* step forward. Dave -- David Young OJC Technologies dyo...@ojctech.com Urbana, IL * (217) 278-3933
UBC and page replacement
Hi, I've sent a similar message to netbsd-users@ list, but didn't get any responses, maybe somebody on tech-kern@ knows the answer? I've been looking at the following paper: "UBC: An Efficient Unified I/O and Memory Caching Subsystem for NetBSD" However I'm not clear about what strategy NetBSD uses for page replacements. For example, a web server does mainly read operations from filesystem, so the kernel will eventually fill up buffer cache. When the buffer cache is full and there is no free RAM, the kernel would need to replace some of the pages. How does it decide what pages to replace? Does it look at the usage pattern or the underlying file size? Does it try to steal pages from a single large (say 2GB) file, rather than stealing them from many small files?
Re: removing aiboost(4) as redundant
On Mon, 8 Mar 2010 06:58:01 -0500 Constantine Aleksandrovich Murenin wrote: > > acpi0: entering state 3 > > Devices without power management support: aibs0 > > acpi0: aborting suspend > > Should be fixed in atk0110.c#rev1.10. It is indeed. Thanks! dieter > > C. >
Re: (Semi-random) thoughts on device tree structure and devfs
On Mar 8, 2010, at 12:47 AM, Masao Uebayashi wrote: > How do you write a kernel config which can always identify my USB disk as > sd0a, even if I plug random devices? You'd need to put the UUID in the kernel config.
Re: (Semi-random) thoughts on device tree structure and devfs
der Mouse wrote: Linux had a devfs and [dropped] it. Now it has udevd(8). Most likely the penguins had a reason for this. Surely there are mailing list messages or something that outline that reason? (Not that I have any idea where they'd be, but don't we have at least a few people with feet in both camps?) This page has links to various bits about the move from devfs to udev. http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html --Dan
re: config(5) break down
i don't think we're going to *give up* static kernel configs, regardless of how far we come modularwise. .mrg.
Re: config(5) break down
On Mon, 8 Mar 2010, Quentin Garnier wrote: > On Mon, Mar 08, 2010 at 04:43:17PM +, Eduardo Horvath wrote: > > Allright. I have to ask: > > > > If the plan is to go to a dynamically probed system with loadable modules, > > why keep config around at all? It's only useful for custom kernels. Why > > is it useful to give config a facelift instead of doing away with it > > entirely? > > config(5) files carry information on what source file to build, under > what conditions as well as information about drivers and relationships > between devices. Why would you do away with it entirely? All that > information will have to be stored somewhere anyway. Yes. It also specifies what drivers attach where. If you have a modular kernel and want to add a new driver, you could compile it and stick it into the module directory tree, but if the config file didn't have that module in it when the kernel was originally built, your driver can never be loaded and your new device is useless. config was a solution to the problem of needing to edit headers and source files to add a new device to the system. In a modular world I think it's a bit dated and there will be a need to figure all this stuff out dynamically. Eduardo
Re: (Semi-random) thoughts on device tree structure and devfs
On Mon, Mar 08, 2010 at 10:54:13AM -0500, der Mouse wrote: > > > BTW: OSF/1 aka DEC-Unix aka Tru64-Unix did somthing like Linux + > > udevd(8) over 10 years ago. > > Another reason to think that it's likely worth trying. So did SGI, and it was a disaster. If you're going to break the common Unix idiom (single directory full of nodes for devices) you'd better be prepared to replace it with something that's very, very easy and intuitive for experienced Unix administrators to learn.
Re: config(5) break down
On Mon, Mar 08, 2010 at 04:43:17PM +, Eduardo Horvath wrote: > If the plan is to go to a dynamically probed system with loadable modules, > why keep config around at all? It's only useful for custom kernels. Why > is it useful to give config a facelift instead of doing away with it > entirely? You can use config to generate the glue code that modules needs. eric
Re: config(5) break down
On Mon, Mar 08, 2010 at 04:43:17PM +, Eduardo Horvath wrote:
> On Fri, 5 Mar 2010, Masao Uebayashi wrote:
>
> > I've found that the difficulty of understanding config(5) is due to its
> > flexibility; it can do one thing in many ways. You can define a collection
> > of sources with define, defflag, device, defpseudo{,dev}, devfs. OTOH you
> > can only write dependency on attributes (define). Another example is, you
> > can write interface with define, device, defpseudodev.
> >
> > I'd propose to make a rule to simplify things (at the cost of a little
> > redundancy of config(5) files).
>
> Allright. I have to ask:
>
> If the plan is to go to a dynamically probed system with loadable modules,
> why keep config around at all? It's only useful for custom kernels. Why
> is it useful to give config a facelift instead of doing away with it
> entirely?
config(5) files carry information on what source file to build, under
what conditions as well as information about drivers and relationships
between devices. Why would you do away with it entirely? All that
information will have to be stored somewhere anyway.
--
Quentin Garnier - c...@cubidou.net - c...@netbsd.org
"See the look on my face from staying too long in one place
[...] every time the morning breaks I know I'm closer to falling"
KT Tunstall, Saving My Face, Drastic Fantastic, 2007.
pgph9qv88iLea.pgp
Description: PGP signature
Re: (Semi-random) thoughts on device tree structure and devfs
Aleksej Saushev wrote: Johnny Billquist writes: I seem to remember that way back there was even a tool (in pkgsrc?) which extracted your current device setup, and created a config file from that, so that you would always get the same enumeration, no matter what else showed up on the machine. The point is, the config file totally, and exactly describes your hardware setup. It takes only a minute to move the very same external HDD from one bus to another one, you only need to pull FireWire connector off and connect with USB cable. No screws involved. No argument on that one. And your point was? If you change the hardware configuration, your config will no longer match. And neither will the same thing mirrored in a file system. The disk will move there too. Johnny
Re: config(5) break down
On Fri, 5 Mar 2010, Masao Uebayashi wrote:
> I've found that the difficulty of understanding config(5) is due to its
> flexibility; it can do one thing in many ways. You can define a collection
> of sources with define, defflag, device, defpseudo{,dev}, devfs. OTOH you
> can only write dependency on attributes (define). Another example is, you
> can write interface with define, device, defpseudodev.
>
> I'd propose to make a rule to simplify things (at the cost of a little
> redundancy of config(5) files).
Allright. I have to ask:
If the plan is to go to a dynamically probed system with loadable modules,
why keep config around at all? It's only useful for custom kernels. Why
is it useful to give config a facelift instead of doing away with it
entirely?
Eduardo
Re: (Semi-random) thoughts on device tree structure and devfs
On Mon, Mar 08, 2010 at 10:54:13AM -0500, der Mouse wrote: > > Linux had a devfs and [dropped] it. Now it has udevd(8). Most > > likely the penguins had a reason for this. > > Surely there are mailing list messages or something that outline that > reason? (Not that I have any idea where they'd be, but don't we have > at least a few people with feet in both camps?) It is more like: Linux had a devfs and [dropped] it. Now it has udevd(8). Most likely the penguins had a reason for this. Linux had udevd(8) and reintroduced devfs. Now it has udevd(8) and some kind of devfs. Most likely the penguins had a reason for this. - Jukka. http://lwn.net/Articles/331818/
Re: (Semi-random) thoughts on device tree structure and devfs
Johnny Billquist writes: > I seem to remember that way back there was even a tool (in > pkgsrc?) which extracted your current device setup, and created > a config file from that, so that you would always get the same > enumeration, no matter what else showed up on the machine. > > The point is, the config file totally, and exactly describes > your hardware setup. It takes only a minute to move the very same external HDD from one bus to another one, you only need to pull FireWire connector off and connect with USB cable. No screws involved. -- HE CE3OH...
Re: (Semi-random) thoughts on device tree structure and devfs
> Linux had a devfs and [dropped] it. Now it has udevd(8). Most > likely the penguins had a reason for this. Surely there are mailing list messages or something that outline that reason? (Not that I have any idea where they'd be, but don't we have at least a few people with feet in both camps?) > udevd(8) gives the user land control over device enumeration. Maybe > no bad idea. (Disclaimer: I don't like Linux.) I think it probably is no bad idea. I don't like Linux either, but I don't think it's so irremediably disastrous that there's nothing at all NetBSD could learn from it. > BTW: OSF/1 aka DEC-Unix aka Tru64-Unix did somthing like Linux + > udevd(8) over 10 years ago. Another reason to think that it's likely worth trying. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Potential re(4) / netbsd-4 / i386 problem?
On 3/5/2010 12:18 PM, Izumi Tsutsui wrote:
I've been seeing panics on a netbsd-4/i386 machine which appears to be
related to the reception of oversized frames:
re0: discarding oversize frame (len=8813)
:
Can anyone help me?
- options DIAGNOSTIC might help debug
Had a kernel running with options DIAGNOSTIC and while I didn't see any
printout other than those
I saw before, I was able to get a kgdb session up and I have a back
trace. Printed out a few bits
and pieces, not sure what would be of real interest:
Program received signal SIGSEGV, Segmentation fault.
0xc05dba53 in ether_input (ifp=0xc21f503c, m=0xc24a4800) at
../../../../net/if_ethersubr.c:648
648 etype = ntohs(eh->ether_type);
(gdb) bt
#0 0xc05dba53 in ether_input (ifp=0xc21f503c, m=0xc24a4800) at
../../../../net/if_ethersubr.c:648
#1 0xc0382d93 in re_rxeof (sc=0xc21f5000) at
../../../../dev/ic/rtl8169.c:1374
#2 0xc03832a2 in re_intr (arg=0xc21f5000) at
../../../../dev/ic/rtl8169.c:1565
#3 0xc062a5d5 in intr_biglock_wrapper (vp=0xc21aef80) at
../../../../arch/x86/x86/intr.c:544
#4 0xc0108198 in Xintr_ioapic_level11 ()
#5 0xc21aef80 in ?? ()
#6 0x in ?? ()
(gdb) p eh
$1 = (struct ether_header *) 0x58a07f87
(gdb) p m
$2 = (struct mbuf *) 0xc24a4800
(gdb) p *eh
Cannot access memory at address 0x58a07f87
(gdb) p *m
$3 = {m_hdr = {mh_next = 0x388a43eb, mh_nextpkt = 0x3bd2a781,
mh_data = 0x58a07f87 , mh_owner =
0x8e878f3c, mh_len = 1082,
mh_flags = -187037215, mh_paddr = 515969279, mh_type = 27862},
M_dat = {MH = {MH_pkthdr = {
rcvif = 0xc21f503c, tags = {slh_first = 0xaf76}, len = 1082,
csum_flags = 197,
csum_data = 2687232, segsz = 419436127}, MH_dat = {MH_ext = {
ext_buf = 0x1d010ad0 ,
ext_free = 0x450008,
ext_arg = 0x84912800, ext_size = 104857600, ext_type =
0xa8c03063, ext_nextref = 0xa8c00102,
ext_prevref = 0x657dca02, ext_un = {extun_paddr = 1729288689,
extun_pgs = {0x6712d9f1,
0x9d4a3f62, 0x1050e731, 0xeae401b, 0x0, 0x0, 0xd0dcbbd0,
0x5b2500f, 0x0, 0x290100,
0x1900165f, 0x80010ad0, 0x450008, 0x7dc2a005, 0x640,
0xa8c0b12c, 0xa8c00f02}},
ext_ofile = 0x667dca02 ,
ext_nfile = 0xd00e21ff ,
ext_oline = 2029390742,
ext_nline = 407935915},
MH_databuf =
"Ð\n\001\035\b\000E\000\000(\221\204\000\...@\006c0À¨\002\001À¨\002Ê}eñÙ\022gb?j\2351
çp\020\...@®\016\000\000\000\000\000\000\000\000лÜÐ\017p²\005\000\000\000\000\000\001)\000_\026\000\031Ð\
n\001\200\b\000E\000\005
Â}\000\...@\006,±À¨\002\017À¨\002Ê}fÿ!\016Ð\226\vöx«\233p\030\...@ÕÏ\000\000<3--+
,.+,*')\001\017\017\020\020\020\020\021\017\020\020\020\020\020\017\017\017\021\020\020\020\017\017\020\02
0\017\020\020\021\017\f\020\020\020\020\020\017\017\017\020\021\017\017\022\031'=Rbmty\177|~}\200\202\200\
177\177\201~}~\201\201\201\203\205"}},
M_databuf =
"
00\000(\221\204\000\...@\006c0À¨\002\001À¨\002Ê}eñÙ\022gb?j\2351çp\020\033@®\016\000\000\000\000\000\000\0
00\000лÜÐ\017P²\005\000\000\000\000\000\001)\000_\026\000\031Ð\n\001\200\b\000E\000\005 Â}\000\...@\006,±
À¨\002\017À¨\002Ê}fÿ!\016Ð\226\vöx«\233p\030\...@ÕÏ\000\000<3--+,.+,*')\001\017\017\020\020\020\020\021\01
7\020\020\020\020\020\017\017\017\021\020\020\020\017\017\020\020\017\020\020\021\017\f\020\020\020\020\02
0\017\017\017\020\021\017\017\022\031'"...}}
(gdb) up
#1 0xc0382d93 in re_rxeof (sc=0xc21f5000) at
../../../../dev/ic/rtl8169.c:1374
1374(*ifp->if_input)(ifp, m);
(gdb) p ifp
$4 = (struct ifnet *) 0xc21f503c
(gdb) p *ifp
$5 = {if_softc = 0xc21f5000, if_list = {tqe_next = 0xc2253400, tqe_prev
= 0xc0c98f80}, if_addrlist = {
tqh_first = 0xc21dfe80, tqh_last = 0xc22dbb10}, if_xname = "re0",
'\0' ,
if_pcount = 0, if_bpf = 0x0, if_index = 1, if_timer = 0, if_flags =
-30653, if__pad1 = 0, if_data = {
ifi_type = 6 '\006', ifi_addrlen = 6 '\006', ifi_hdrlen = 14
'\016', ifi_link_state = 2,
ifi_mtu = 1500, ifi_metric = 0, ifi_baudrate = 10,
ifi_ipackets = 13724322, ifi_ierrors = 5,
ifi_opackets = 4730575, ifi_oerrors = 0, ifi_collisions = 0,
ifi_ibytes = 11913686395,
ifi_obytes = 256544336, ifi_imcasts = 12006, ifi_omcasts = 5,
ifi_iqdrops = 0, ifi_noproto = 0,
ifi_lastchange = {tv_sec = 0, tv_usec = 0}}, if_output = 0xc05db01c
,
if_input = 0xc05dba19 , if_start = 0xc0383326 ,
if_ioctl = 0xc0384654 , if_init = 0xc0383ca1 ,
if_stop = 0xc03847e6 ,
if_watchdog = 0xc0384774 , if_drain = 0, if_snd =
{ifq_head = 0x0, ifq_tail = 0x0,
ifq_len = 0, ifq_maxlen = 512, ifq_drops = 0, altq_type = 0,
altq_flags = 1, altq_disc = 0x0,
altq_ifp = 0xc21f503c, altq_enqueue = 0, altq_dequeue = 0,
altq_request = 0, altq_clfier = 0x0,
altq_classify = 0, altq_tbr = 0x0, altq_cdnr = 0x0}, if_sadl =
0xc21dfec4,
if_broadcastaddr = 0xc0a1e82
Re: (Semi-random) thoughts on device tree structure and devfs
It would help if you started by showing where your disk would be in the device tree. Then I can tell you what (more or less) you need in your config file. USB, or whatever else, is no magic. You can specify explicitly where your disk is, and have it show up with a specific device number even with other devices attached anywhere. I seem to remember that way back there was even a tool (in pkgsrc?) which extracted your current device setup, and created a config file from that, so that you would always get the same enumeration, no matter what else showed up on the machine. The point is, the config file totally, and exactly describes your hardware setup. Your suggestion would simply mean that this information would be duplicated in the file system. The config file have the additional "feature" of actually making the device appear with the same name, even if you move it around, by just changing the config file. Everything else in the system will not have to be told after that. And the names exposed, and referred to, are simple and short, even though you do have the full tree described in the config file. Someone else mentioned that the problem have grown for the simple reason that hardware configurations change much more often now than in the past. I would agree with that. However, for vital pieces of the hardware, the setup normally don't change that much (such as the disks normally used by the system). So, the device configuration and enumeration is only random so far as that if you tell the system that it is okay to give a device a random number, it will actually possibly do that. Otherwise it is totally predictable. If you, on the other hand, do move your disk around (be that by using USB and different ports and hubs, or different controllers), neither the old config, nor your new solution will help. The disk will change identity (or path) (well, with the old config, it might actually keep it's identity, but that's a chancy proposition at best). This is why a way to refer to disks by some other property would be nice (such as disk labels). (DEC actually solved this a long time ago, by letting disks have identities, which were not in any way related to anything else than the disk, and that was the unit numbers on MSCP disks, which you preferrably kept unique for the whole system, but as with many other nice things, it fell out of fashion with revolution of PC commodity hardware, which have a foresight shorter than my nose.) Johnny Masao Uebayashi wrote: Have everyone forgotten how to set up their own kernel? Is everyone now booting GENERIC? (Or just making a copy of GENERIC, with a few patches without understanding what they are editing?) The whole point being that if you boot a kernel, in which you have configured the whole system to connect anything anywhere, you should not be surprised if the device enumeration might seem random. If you want predictable device enumetaion, you can have that, and have been able to have that for over twenty years... The line wd* at atabus? drive ? flags 0x (to use one example) says that match any wd type disk to any unit number on any atabus, without doing any closer matching. Ie. kindof unpredictable. The asterisks and question marks means exactly that. If you want predictable matching that stays the same at every boot, no matter what hardware you put on the system, you write explicit lines in the config instead. Imagine if I want to use a USB disk as / on my DELL OptiPlex 745. The device tree of that machine looks like: /mainbus0 /pci0 /puhb0 /agp0 /ppb0 /pci0 /vge0 /ukphy0 /vga0 /wsdisplay0 /drm0 /uhci0 /azalia0 /ppb0 /pci0 /ppb1 /pci0 /uhci1 /uhci2 /uhci3 /uhci4 /ppb2 /pci0 /ichlpcib0 /isa0 /lpt0 /com0 /piixide0 /atabus0 /wd0 /atabus1 /atapibus0 /cd0 /ichsmb0 /piixide1 /atabus0 /atabus1 How do you write a kernel config which can always identify my USB disk as sd0a, even if I plug random devices? Masao
Re: (Semi-random) thoughts on device tree structure and devfs
> That doesn't work. Suppose I plug disk B into slot 2, then disk A into > slot 1, but tomorrow I plug disk A in first? Or, suppose when I reset > the machine the vagaries of power distribution or who knows what cause > either one disk or the other to spin up and come ready first on > different days? In my devfs, devices are enumerated in the local connection. /dev/mainbus0/.../piixide0/ata0/wd0 /dev/mainbus0/.../piixide0/ata1/wd0 /dev/mainbus0/.../piixide1/ata0/wd0 /dev/mainbus0/.../piixide1/ata1/wd0 > Also, sometimes you can't tell which is slot 1 and > which is slot 2... Use IDs. > I think an explicit graph (rather than trees with symlinks) is a > better approach in the long term. See the original post. I showed the reversed pseudobus, which I've found very powerful. More examples: /dev/mainbus0/.../screen0/vt100emul0 /dev/mainbus0/.../screen0/wsmuxout0 /dev/mainbus0/.../kbd0/wsmuxin0 /dev/mainbus0/.../mouse0/wsmuxin0 /dev/pseudobus0/wsmux0/wsmuxout0 -> /dev/mainbus0/.../screen0/wsmuxout0 /dev/pseudobus0/wsmux0/wsmuxin0 -> /dev/mainbus0/.../kbd0/wsmuxin0 /dev/pseudobus0/wsmux0/wsmuxin1 -> /dev/mainbus0/.../mouse0/wsmuxin0 Where screen0 has two children, vt100emul0 and wsmuxout0. wsmuxout0 *joins* wsmux0. kbd0's child wsmuxin0 joins wsmux0 too. When kbd0 receives a character, it delivers it to wsmuxin0, which in turn delivers it to wsmux0, which in turn delivers it to wsmuxout0, then finally screen0. screen0 sends the received character to its child vt100emul0. Now how multi-head support looks is pretty much straightforward. bridge(4) + tap(4) + some ether(4) would look exactly same manner. Masao -- Masao Uebayashi / Tombi Inc. / Tel: +81-90-9141-4635
Re: removing aiboost(4) as redundant
On 07/03/2010, dieter roelants wrote: > On Sat, 6 Mar 2010 01:47:00 -0500 > Constantine Aleksandrovich Murenin wrote: > > > has suggested that aiboost(4) be removed unless there are > > objections within a week from this notice. > > > I have an objection: > > acpi0: entering state 3 > Devices without power management support: aibs0 > acpi0: aborting suspend Should be fixed in atk0110.c#rev1.10. http://cvsweb.netbsd.org/cgi-bin/cvsweb.cgi/src/sys/dev/acpi/atk0110.c#rev1.10 C.
Re: config(5) break down
> define foo: bar > : > define bar: baz > : > > I have no idea if config currently allows this It doesn't allow, BTW /src/netbsd/src.TNF/sys/conf/files:289: undefined attribute `scsi_core' Masao
Re: config(5) break down
> Well, first of all nothing says you can't read the whole file before > resolving dependencies; there's nothing inherently wrong with > > define foo: bar > : > define bar: baz > : > > I have no idea if config currently allows this but it's not exactly > difficult to arrange. I don't think it's worth. Split *.conf files can be also distributed with *.[ch]. Its notation is self-descriptive. The only problem I'm aware of is it consumes more disk space. > No, it seems like you're intending to use whole files to define groups > instead of braces or some other punctuation. There is no need to split > things into separate files just to show grouping. True. > (Besides, it's not necessarily as flat as all that, either.) It's necessary to be flat to be modular. > I thought you said all you needed to do was grep for the logical > operators... I said it's possible. Again, I don't want to change config(1) at all. And I don't need. Masao -- Masao Uebayashi / Tombi Inc. / Tel: +81-90-9141-4635
Re: (Semi-random) thoughts on device tree structure and devfs
> http://www.netbsd.org/~cegger/pmem2.diff > May not cleanly apply against -current. Ah. You provide API for MD codes. (I thought you wanted to provide a new API for drivers, and was about to write a compllainment. :P) I wonder if we want to make device memory allocation very smart, using vmem(9). It's done in very low level; very probably handled in bootstrap code. We don't want to introduce unnecessary dependencies of kernel subsystems. Masao -- Masao Uebayashi / Tombi Inc. / Tel: +81-90-9141-4635
Re: config(5) break down
On Mon, Mar 08, 2010 at 10:53:16AM +0200, Antti Kantee wrote: > > (FFS_EI isn't the only such option either, it's just one I happen to > > have already banged heads with.) > > This one is easy, no need to make it difficult. Sure, but as I said it was just an example; what about the next one? > Things like wapbl are currently an actual problem, since it is multiply > owned (conf/files *and* ufs/files.ufs). I don't see why this is a problem either. The way things are right now, vfs_wapbl.c is conditional on "wapbl" the same as the rest; enforcing a hierarchical decomposition by source directory would break this but that's part of why such hierarchical decompositions are a bad idea. (I've also never fully understood why wapbl has to have so many tentacles hanging out of ffs, either.) -- David A. Holland dholl...@netbsd.org
Re: config(5) break down
On Mon, Mar 08, 2010 at 05:46:21PM +0900, Masao Uebayashi wrote: > > Again: why? You've latched onto this as (apparently) a key point of > > what you're doing, except that I don't see why it's necessary or even > > desirable or how it relates to any of the other things you've > > suggested. > > Dependency. When module foo depends on bar, bar has to be defined. > > include "bar.conf" > define foo: bar Well, first of all nothing says you can't read the whole file before resolving dependencies; there's nothing inherently wrong with define foo: bar : define bar: baz : I have no idea if config currently allows this but it's not exactly difficult to arrange. > > The braces, which group the related material together and can be used > > for scoping too? > > We don't need braces because modules grouping is flat. No, it seems like you're intending to use whole files to define groups instead of braces or some other punctuation. There is no need to split things into separate files just to show grouping. (Besides, it's not necessarily as flat as all that, either.) > > Sure. But what do you propose to do about them? > > Splitting helps to realize where bad instances exist. How to fix them is > another story. I thought you said all you needed to do was grep for the logical operators... > > Yes, that's the two-copies-each method. > > If you know, why did you ask??? Such techniques are already throughtly > discuessed everywhere, not only in pkgsrc... Because I'm trying to figure out what you're suggesting doing? -- David A. Holland dholl...@netbsd.org
Re: (Semi-random) thoughts on device tree structure and devfs
On 07.03.10 13:02, Masao Uebayashi wrote: >> The good news: >> dyoung@ and I started with prototyping pmem(9) that provides an MI >> physical address space management. >> >> The bad news: >> We had no time to continue on this for more than a year now. > > What have you achieved? > http://www.netbsd.org/~cegger/pmem2.diff May not cleanly apply against -current. Christoph
Re: (Semi-random) thoughts on device tree structure and devfs
Lookup-by-ID should be symlink, like: /dev/id/guid/25892e17-80f6-415f-9c65-7395632f0223 -> /dev/mainbus0/.../wd0/disk0/gpt0 /dev/id/ieee802mac/00-b0-d0-86-bb-f7 -> /dev/mainbus0/.../bge0 /dev/id/ipv4/172.16.0.1 -> /dev/mainbus0/.../bge0/ether0/net0 Masao -- Masao Uebayashi / Tombi Inc. / Tel: +81-90-9141-4635
Re: config(5) break down
On Mon Mar 08 2010 at 07:09:07 +, David Holland wrote: > Meanwhile, I think trying to wipe out all the boolean dependency logic > in favor of a big graph of modules and submodules is also likely to > make a mess. What happens to e.g. > > fileufs/ffs/ffs_bswap.c (ffs | mfs) & ffs_ei > > especially given that the ffs code is littered with FFS_EI conditional > compilation? You can make ffs_bswap its own module, but that doesn't > really serve any purpose. You could try making an FFS_EI module that > works by patching the ffs module on the fly or something, and then > include ffs_bswap.o into that, but that would be both very difficult > and highly gross. You could compile two copies each of ffs and mfs, > with and without FFS_EI support, but that wastes space. Or you could > make FFS_EI no longer optional, which would be a regression. > > (FFS_EI isn't the only such option either, it's just one I happen to > have already banged heads with.) This one is easy, no need to make it difficult. The NetBSD-supplied module is always compiled with FFS_EI (if you don't like it, you can always compile your own just like you can compile your own kernel now). We don't care about mfs here, since it's not reasonable to want to mount a memory file system in the opposite byte order (technically I guess you could mmap an image instead of malloc+newfs and then mount(MOUNT_MFS), but you might just as well use ffs). Things like wapbl are currently an actual problem, since it is multiply owned (conf/files *and* ufs/files.ufs). The easy solution (and my vote) would be to make vfs_wapbl.c always included in the base kernel. If someone feels it's worth their salt to make it into two modules with all the dependency hum-haa, that would be a good place to start practicing instead of ffs_ei.
Re: config(5) break down
> Again: why? You've latched onto this as (apparently) a key point of > what you're doing, except that I don't see why it's necessary or even > desirable or how it relates to any of the other things you've > suggested. Dependency. When module foo depends on bar, bar has to be defined. include "bar.conf" define foo: bar > The braces, which group the related material together and can be used > for scoping too? We don't need braces because modules grouping is flat. > Of course. Not only that, I've implemented quite a few kernel config > and build systems of my own, and I've waded into how it's done in > Linux as well. > > % wc -l files > 1700 files > > That's not small, but it's certainly not prohibitively large. Sure, it > could be organized better. I have no problem with that. Then see dependency again. > Sure. But what do you propose to do about them? Splitting helps to realize where bad instances exist. How to fix them is another story. > Yes, that's the two-copies-each method. If you know, why did you ask??? Such techniques are already throughtly discuessed everywhere, not only in pkgsrc... Masao -- Masao Uebayashi / Tombi Inc. / Tel: +81-90-9141-4635
Re: test wanted: module plists
On Mon Mar 08 2010 at 02:37:05 +, David Holland wrote:
> The code for loading a module plist from a file system is messed up in
> that it calls namei() and then it calls vn_open() on the same
> nameidata without reinitializing it or cleaning up the previous
> results. I'm surprised this didn't result in fireworks, but apparently
> it didn't.
>
> The following patch fixes that, and compiles, but I'm not set up to be
> able to test this -- is there anyone who can do so easily/quickly?
When I was playing with that code, I used atf on tests/modules. I can't
remember if it tests loading from .prop, but a .prop file isn't exactly
hard to create.
Dunno what the canonical in-tree feature using this is, though.
> Index: kern_module_vfs.c
> ===
> RCS file: /cvsroot/src/sys/kern/kern_module_vfs.c,v
> retrieving revision 1.3
> diff -u -p -r1.3 kern_module_vfs.c
> --- kern_module_vfs.c 16 Feb 2010 05:47:52 - 1.3
> +++ kern_module_vfs.c 8 Mar 2010 02:33:36 -
> @@ -147,23 +147,18 @@ module_load_plist_vfs(const char *modpat
> NDINIT(&nd, LOOKUP, FOLLOW | (nochroot ? NOCHROOT : 0),
> UIO_SYSSPACE, proppath);
>
> - error = namei(&nd);
> - if (error != 0) {
> - goto out1;
> + error = vn_open(&nd, FREAD, 0);
> + if (error != 0) {
> + goto out1;
> }
>
> error = vn_stat(nd.ni_vp, &sb);
> if (error != 0) {
> - goto out1;
> + goto out;
> }
> if (sb.st_size >= (plistsize - 1)) {/* leave space for term \0 */
> error = EFBIG;
> - goto out1;
> - }
> -
> - error = vn_open(&nd, FREAD, 0);
> - if (error != 0) {
> - goto out1;
> + goto out;
> }
>
> base = kmem_alloc(plistsize, KM_SLEEP);
>
>
>
> --
> David A. Holland
> dholl...@netbsd.org
Re: config(5) break down
On Mon, Mar 08, 2010 at 05:08:56PM +0900, Masao Uebayashi wrote:
> > That line of reasoning only makes sense if splitting things into
> > multiple files provides some kind of scoping, or encapsulation, or
> > other form of abstraction.
>
> One *.conf matches one module.
Again: why? You've latched onto this as (apparently) a key point of
what you're doing, except that I don't see why it's necessary or even
desirable or how it relates to any of the other things you've
suggested.
> We'll always build modules as single object
> like intermediate *.a in userland. Expose only necessary symbols. Multiple
> *.c are optimized by global optimizer. We can identify API - what symbols
> it exposes or references. Modules are not only for dynamic loading.
That's a fine thing(*) but I *still* don't see how it relates to
splitting up conf/files.
(*) Although I think it'd be more trouble to maintain than it's worth.
Source modularity begins with structuring header files carefully, and
we've got a long, long way to go there before we're ready to think
about explicit linker-level ABI control.
> > Given that config language has no scoping at all right now, the first
> > order of business would be to add some of that, like
> >
> > module vfs {
> >file kern/vfs_vnops.c
> >file kern/vfs_xattr.c
> > :
> > }
>
> It can be written as:
>
> define vfs
> file kern/vfs_vnops.c vfs
> file kern/vfs_xattr.c vfs
>
> I don't see what's different.
The braces, which group the related material together and can be used
for scoping too?
> > but putting each one of those in its own file isn't going to serve any
> > purpose and it'll make everything that much harder to examine all at
> > once.
>
> Have you ever examined sys/conf/files?
Of course. Not only that, I've implemented quite a few kernel config
and build systems of my own, and I've waded into how it's done in
Linux as well.
% wc -l files
1700 files
That's not small, but it's certainly not prohibitively large. Sure, it
could be organized better. I have no problem with that.
> > Meanwhile, I think trying to wipe out all the boolean dependency logic
> > in favor of a big graph of modules and submodules is also likely to
> > make a mess. What happens to e.g.
> >
> > fileufs/ffs/ffs_bswap.c (ffs | mfs) & ffs_ei
> >
> > especially given that the ffs code is littered with FFS_EI conditional
> > compilation? You can make ffs_bswap its own module, but that doesn't
> > really serve any purpose. You could try making an FFS_EI module that
> > works by patching the ffs module on the fly or something, and then
> > include ffs_bswap.o into that, but that would be both very difficult
> > and highly gross. You could compile two copies each of ffs and mfs,
> > with and without FFS_EI support, but that wastes space. Or you could
> > make FFS_EI no longer optional, which would be a regression.
> >
> > (FFS_EI isn't the only such option either, it's just one I happen to
> > have already banged heads with.)
>
> Yes, we have to hunt all these dirtinesses.
Sure. But what do you propose to do about them? Whatever source file
list framework we have needs to handle these cases somehow. It's not
an improvement to "simplify" the framework so it can't handle the
cases we have to support.
> One idea is to provide "alternative" modules. Modules that provide/require
> identical API, but have different signature (-DXXX, #ifdef).
Yes, that's the two-copies-each method.
--
David A. Holland
dholl...@netbsd.org
Re: (Semi-random) thoughts on device tree structure and devfs
> Linux had a devfs and droped it. Now it has udevd(8). Most likely the > penguins had a reason for this. udevd(8) gives the user land control > over device enumeration. Maybe no bad idea. (Disclaimer: I don't like > Linux.) I took a little glance at OpenSolaris/FreeBSD devfs man pages, and quickly stopped. They're all overly complicated. Those who complain about redundant device paths exposed should look at other implementations. I don't really like to have /etc/devfsd.conf and bikeshed its format. Exposing device tree works, because NetBSD's hardware device abstraction is done pretty much sanely (software abstraction has problems, tho). > BTW: OSF/1 aka DEC-Unix aka Tru64-Unix did somthing like Linux + > udevd(8) over 10 years ago. Good point. We're hopelessly behind. Masao -- Masao Uebayashi / Tombi Inc. / Tel: +81-90-9141-4635
Re: (Semi-random) thoughts on device tree structure and devfs
At the risk of being a wet blanket: On Sun, Mar 07, 2010 at 06:43:49PM +0900, Masao Uebayashi wrote: > I've been spending LOTS of time to investigate various devicess sources, to > understand some questions I've had, like: > > - Why NetBSD/arm has no bus_space_mmap(4)? hellifIknow; > - Why tty locking is messy? because ttys are messy, which is because they haven't had a big rototill in some twenty years or more; > - Why sys/dev/wscons has so many #ifdef's? (Modular unfriendly!) dunno; > - How dk(4) is enumerated? in the order found; > a) Device enumeration is unstable / unpredictable > > dk(4) is a pseudo device, and its instances are numbered in the order it's > created. This is fine when you manually / explicitly add wedges(4) by using > "dkctl addwedge". This is not fine, if I have a gpt(4) disk label which has > ordered partitions. I expect disks to be created in the order I write in > the gpt(4) disk label. It's annoying the numbering changes when I add a new > disk. Same for raidframe(4). Why doesn't gpt(4) create the wedges in that order? If it did that they'd come out numbered the way you'd expect. Having the numbering change when you add a new disk is unavoidable. See further notes below. > b) Consistent device topology management is missing > > The reason why NetBSD/arm has no bus_space_mmap(9) has turned out to be the > fact that we have no consistent (MI) way to manage physical address space of > devices. NetBSD/mips has a working bus_space_mmap(9) in > sys/arch/mips/mips/bus_space_alignstride_chipdep.c. It defines address > windows and manage it by itself. > > Who wants to reimplement it on all cpus/ports/platforms? > Considering physical address space is a pretty much simple concept > - a single linear address space. Except when it isn't really; consider for example NUMA systems. I think there have also been systems where different CPUs see a different physical address space view. Whether any of these are systems we care about, I dunno. > And we already manage (kind of) tree of devices in autoconf(9). Do > we want to manage such a topology in many places? No. That said, it's still useful to have MI code for common constructs even if it doesn't work with every platform. > c) Control / data flow is unclear > > I've never remembered what wscons command/device to configure wscons to add > screen, load font, change encoding. It's a total mess. I don't know how > the ioctl I send via wscons command is delivered to device. Same for data. > Even by looking at sys/dev/wscons. Why it it so complicated? Dunno. Is it? ioctl routing is gross (almost inevitably/inherently) but this has never seemed particularly worse than anything else. > Our tty locking code has so many hacks. See grep XXX sys/kern/tty*. And we > have to fix all serial devices. How should serial devices deal with tty > lock? How ioctl works? How its callback is called and when? How to avoid > deadlock? This is almost hopeless. Yes, there's a reason "fix ttys" is fairly high on my cleanup list. > Same for network devices's ioctl handling. > > d) Abstraction of combined/aggregated device is inconsistent > > We have some *special* devices that combine/aggregate multiple > devices and make it look like a single device. For example > wsmux(4), ccd(4), raidframe(4), lvm(4), bridge(4), agr(4), ... Now > these do almost random way to manage its components, and its > behaviour is hard to guess. You have to learn how to add/delete > components to some combining device, its limitation, etc. Unifying the access methods for these (or at least some of these) seems like a good idea, yes. > The enumeration of these is also hard to predict. > > > e) Random way of abstraction > > We have many non-real devices used to abstract real devices. For > example audio, tty, wsdisplay, network interfaces, wedges, scsipi, > com and friends, usb, pseudo devices, ... We have to learn how to > use them and their behavior respectively. Yes, and? > Developers have to decide how your device is represented to user. > If you write a serial device, you have to implement all the syscall > nobs, buffer management, tty interaction. You'll surely end up > having a big modified copy of com.c, which is almost impossible to > maintain. See "ttys are a mess". > I want to fix all of these. All at once? If you have a single overarching framework in mind that's going to address all the preceding stuff, I'd really suggest hacking it up as a research project first. (That is, do it in a context where you aren't committed to maintaining expensive production system guarantees, like backwards compatibility or the ability to run perl, and where you can remove or disable things that get in the way instead of having to stop and deal with them.) This will give you at least three things; first, a proof of concept that your architecture is viable, which many people are not
Re: config(5) break down
> That line of reasoning only makes sense if splitting things into
> multiple files provides some kind of scoping, or encapsulation, or
> other form of abstraction.
One *.conf matches one module. We'll always build modules as single object
like intermediate *.a in userland. Expose only necessary symbols. Multiple
*.c are optimized by global optimizer. We can identify API - what symbols
it exposes or references. Modules are not only for dynamic loading.
> Given that config language has no scoping at all right now, the first
> order of business would be to add some of that, like
>
> module vfs {
>file kern/vfs_vnops.c
>file kern/vfs_xattr.c
> :
> }
It can be written as:
define vfs
file kern/vfs_vnops.c vfs
file kern/vfs_xattr.c vfs
I don't see what's different.
> but putting each one of those in its own file isn't going to serve any
> purpose and it'll make everything that much harder to examine all at
> once.
Have you ever examined sys/conf/files?
> Meanwhile, I think trying to wipe out all the boolean dependency logic
> in favor of a big graph of modules and submodules is also likely to
> make a mess. What happens to e.g.
>
> fileufs/ffs/ffs_bswap.c (ffs | mfs) & ffs_ei
>
> especially given that the ffs code is littered with FFS_EI conditional
> compilation? You can make ffs_bswap its own module, but that doesn't
> really serve any purpose. You could try making an FFS_EI module that
> works by patching the ffs module on the fly or something, and then
> include ffs_bswap.o into that, but that would be both very difficult
> and highly gross. You could compile two copies each of ffs and mfs,
> with and without FFS_EI support, but that wastes space. Or you could
> make FFS_EI no longer optional, which would be a regression.
>
> (FFS_EI isn't the only such option either, it's just one I happen to
> have already banged heads with.)
Yes, we have to hunt all these dirtinesses.
One idea is to provide "alternative" modules. Modules that provide/require
identical API, but have different signature (-DXXX, #ifdef).
Masao
--
Masao Uebayashi / Tombi Inc. / Tel: +81-90-9141-4635
