Re: Moving Lua source codes
Mouse mouse at Rodents-Montreal.ORG writes: ... I think the biggest reasons I haven't tried to push any of those into NetBSD are (1) a perception that NetBSD doesn't want my changes, combined with (2) a lack of motivation to, a sort of NetBSD no longer cares about me; why should I care about it? feeling. (Note that I'm not saying anything about how justifiable - or correct! - these feelings are.) That said I'd certainly be happy to offer any-to-all of my changes to anyone who wants to bring them into NetBSD. ... If anyone's interested, of course. My expectation has been that nobody is, but this email makes me think that could well be wrong. I can easily pull a full list of changes I've made and mail it wherever. Mouse, I'm not an EdgeBSD advocacy, but when i've heard of it and briefly looked at i got a feeling that they're kinda open to bringing of various experimental things, even kernel-side ones. I don't know their relations with NetBSD project, but there was an info that they will try to push well-tried bits back to the NetBSD. If all of this is true, maybe it's a more suitable place for your stuff now?
Re: storage-class memory (was: Re: state of XIP?)
On Oct 17, 2013, at 10:41 PM, David Holland dholland-t...@netbsd.org wrote: If the XIP code is not mergeable, what's entailed in doing a different implementation that would be? Also, is the getpages/putpages interface expressive enough to allow doing this without major UVM surgery? For now I'm assuming a file system that knows about storage-class memory and can fetch the device physical page that corresponds to any particular file and offset. ISTM that at least in theory it ought to be sufficient to implement getpages by doing this, and putpages by doing nothing at all, but I don't know that much specifically about UVM or the pager interface. IMO, no, getpages interface is not sufficient. You also have the problem that the pages to be mapping are not managed pages. Additionally, you know these pages are almost certainly going to be physically contiguous so you really to use large page sizes to map them. So you don't want UVM allocating pages nor do you want to deal with unified buffer cache. Indeed, it might be cheaper to avoid uvm_fault to map the pages and just map them. The only problem is marking data as copy-on-write but again these pages aren't managed so the current COW code won't be happy.
Re: storage-class memory (was: Re: state of XIP?)
On Fri, Oct 18, 2013 at 12:34:24AM -0700, Matt Thomas wrote: If the XIP code is not mergeable, what's entailed in doing a different implementation that would be? Also, is the getpages/putpages interface expressive enough to allow doing this without major UVM surgery? For now I'm assuming a file system that knows about storage-class memory and can fetch the device physical page that corresponds to any particular file and offset. ISTM that at least in theory it ought to be sufficient to implement getpages by doing this, and putpages by doing nothing at all, but I don't know that much specifically about UVM or the pager interface. IMO, no, getpages interface is not sufficient. You also have the problem that the pages to be mapping are not managed pages. Ugh. Additionally, you know these pages are almost certainly going to be physically contiguous so you really to use large page sizes to map them. So you don't want UVM allocating pages nor do you want to deal with unified buffer cache. Who says they're going to be physically contiguous? It would be nice if they are, for that reason, but we're mapping files, not a device; it's up to the FS to allocate intelligently and it doesn't necessarily happen. Plus, if I map a 4M (or whatever appropriate size) region from a single contiguous source, hopefully this will result in a superpage mapping independently of what the underlying material is. These are *supposed* to be UBC pages though; they're file data and it's important for read(2) and write(2) operations to maintain consistency with mappings. It seems like what's wanted is to give UVM the page instead of having it allocate one; but again, I don't know that much about UVM on the inside. Indeed, it might be cheaper to avoid uvm_fault to map the pages and just map them. Probably, but faulting them has to work so this is probably an independent issue. The only problem is marking data as copy-on-write but again these pages aren't managed so the current COW code won't be happy. We shouldn't have to care about that unless we want to move to MAP_COPY from MAP_PRIVATE. -- David A. Holland dholl...@netbsd.org
Re: Lua in-kernel (lbuf library)
Marc Balmer marc at msys.ch writes: Justin Cormack justin at specialbusservice.com writes: I have been using the luajit ffi and luaffi, which let you directly use C structs (with bitfields) in Lua to do this. It makes it easier to reuse stuff that is already defined in C. (luaffi is not in its current state portable but my plan is to strip out the non portable bits, which are the function call support). Justin I had successfully used more lightweight solution called Lua AutoC [1] with Marc's lua(4). Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be used in kernel, does the things in runtime, which gives the flexibility. Cons: not widely tested, again does the things in runtime, which on other side may give performance penalty. I never used luaffi. It sounds very interesting and I think it could be very useful to bind already defined C structs, but my purpose is to dynamically define data layouts using Lua syntax (without parsing C code). FFI in the kernel can be dangerous. Pure Lua is a perfect confinment for code, but with an FFI a Lua script can access almost anything in the kernel. One has to think twice if one wants that. Well, assuming it would be module, so I would not have to load it if I don't want to. It's desirable if you're writing a device driver in Lua, as you can do most of work from Lua code (e.g. call C methods of NetBSD driver API and feed them with C structs and pointers). States and explicit exports of a certain foreign functions makes things a bit less dangerous. But in general you're right, one should do this with care. [1] https://github.com/orangeduck/LuaAutoC
Re: Lua in-kernel (lbuf library)
Lourival Vieira Neto lourival.neto at gmail.com writes: On Wed, Oct 16, 2013 at 11:45 AM, Justin Cormack justin at specialbusservice.com wrote: (...) Yes absolutely it makes more sense if already defined in C. For parsing binary stuff I would look at Erlang for inspiration too, it is one of the nicer designs. Justin I never gone that far in Erlang. It looks really interesting [1]. I'll take a deeper look later. Thanks! Regards, I think a hybrid approach with mutable C-like data structs for holding of binary data and Erlang's binary pattern matching would be cool. -- dukzcry
Re: Lua in-kernel (lbuf library)
Am 18.10.13 10:43, schrieb Artem Falcon: Marc Balmer marc at msys.ch writes: Justin Cormack justin at specialbusservice.com writes: I have been using the luajit ffi and luaffi, which let you directly use C structs (with bitfields) in Lua to do this. It makes it easier to reuse stuff that is already defined in C. (luaffi is not in its current state portable but my plan is to strip out the non portable bits, which are the function call support). Justin I had successfully used more lightweight solution called Lua AutoC [1] with Marc's lua(4). Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be used in kernel, does the things in runtime, which gives the flexibility. Cons: not widely tested, again does the things in runtime, which on other side may give performance penalty. I never used luaffi. It sounds very interesting and I think it could be very useful to bind already defined C structs, but my purpose is to dynamically define data layouts using Lua syntax (without parsing C code). FFI in the kernel can be dangerous. Pure Lua is a perfect confinment for code, but with an FFI a Lua script can access almost anything in the kernel. One has to think twice if one wants that. Well, assuming it would be module, so I would not have to load it if I don't want to. It's desirable if you're writing a device driver in Lua, as you can do most of work from Lua code (e.g. call C methods of NetBSD driver API and feed them with C structs and pointers). States and explicit exports of a certain foreign functions makes things a bit less dangerous. But in general you're right, one should do this with care. lua(4) has a mechanism for Lua's 'require' statement. Normally, when you require 'foo', it looks up wheter a kernel module name luafoo exists and loads it. This automatic loading of modules can be turned off, to make a module available to a state, it has to be specifically assigned. So when you turn autoloading off, a script could not simply call a ffi module by requiring it. Maybe Lua kernel modules should carry a flag whether they should allow autoloading or not? This way, an ffi module would still be loaded into the kernel when Lua code requires it, but lua(4) would detect the don't autoload flag and would then not_ assign the module to the Lua state. [1] https://github.com/orangeduck/LuaAutoC
Re: Moving Lua source codes
Date: Mon, 14 Oct 2013 22:46:53 +0200 From: Marc Balmer m...@msys.ch It is entirely plausible to me that we could benefit from using Lua in base, or sysinst, or maybe even in the kernel. But that argument must be made by showing evidence of real, working code that has compelling benefits, together with confidence in its robustness -- not by saying that if we let users do it then it will happen. There is real word, real working code. In userland and in kernel space. There are developers waiting for the kernel parts to be committet, so they can continue their work as well. Where is the real, working application code?
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Date: Thu, 17 Oct 2013 19:16:16 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something.
RAIDframe: stripe unit per reconstruction unit
The man page says: The stripe units per parity unit and stripe units per reconstruction unit are normally each set to 1. While certain values above 1 are permitted, a discussion of valid values and the consequences of using anything other than 1 are outside the scope of this document. I noticed that reconstruction seems to read/write in units of one stripe unit, which in my case (being 4k) seems rather inefficient. Is it possible to accelerate recontruction by using a figure larger than 1 here?
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hi, The linked research was performed on Linux, which has NFsv4.1 and pNFS client implementations. Evidently, you can do this kind of thing with an out-of-tree Lua kernel extension. Matt - Taylor R Campbell riastr...@netbsd.org wrote: [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. -- Matt Benjamin The Linux Box 206 South Fifth Ave. Suite 150 Ann Arbor, MI 48104 http://linuxbox.com tel. 734-761-4689 fax. 734-769-8938 cel. 734-216-5309
Re: Moving Lua source codes
[...] If all of this is true, maybe [EdgeBSD i]s a more suitable place for your stuff now? Possibly. They're welcome to my changes if they want them too. I'll send a ping thataway. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: storage-class memory (was: Re: state of XIP?)
On Oct 18, 2013, at 1:06 AM, David Holland dholland-t...@netbsd.org wrote: The only problem is marking data as copy-on-write but again these pages aren't managed so the current COW code won't be happy. We shouldn't have to care about that unless we want to move to MAP_COPY from MAP_PRIVATE. Huh? I'm was talking about an executable's .data section. Since we are talking about execute-in-place.
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hello, Lourival Vieira Neto lourival.n...@gmail.com writes: On Thu, Oct 17, 2013 at 1:26 PM, Jeff Rizzo r...@tastylime.net wrote: On 10/14/13 1:46 PM, Marc Balmer wrote: There is real word, real working code. In userland and in kernel space. There are developers waiting for the kernel parts to be committet, so they can continue their work as well. *Where* is this code? The pattern I see happening over and over again is: NetBSD Community: Please show us the real working code that needs this mbalmer: the code is there! (pointer to actual code not in evidence) I do not doubt that something exists, but the onus is on the person proposing the import to convince the skeptics, or at least to make an actual effort. I see lots of handwaving, and little actual code. YEARS after the import of lua into the main tree, I see very little in-tree evidence of its use. In fact, what I see is limited to : 1) evidence of lua bindings for netpgp. 2) evidence of some tests in external/bsd/lutok 3) the actual lua arc in external/mit/lua 4) gpio and sqlite stuff in liblua 5) some lua bindings in libexec/httpd (bozohttpd) 6) two example files in share/examples/lua 7) the luactl/lua module/lua(4) stuff you imported yesterday ...and counting. There is also ongoing working happening =). As Jeff points what is counting is support code. Am I missing something major here? The only actual usage I see is netpgp and httpd; the rest is all in support of lua itself. I do not see evidence that anyone is actually using lua in such a way that requires it in-tree. When you originally proposed importing lua back in 2010, you talked a lot about how uses would materialize. It's now been 3 years, and I just don't see them. If I am wrong about this, I would love some solid pointers to evidence of my wrongness. Now you're using very similar arguments for bringing lua into the kernel; I would very much like to see some real, practical, *useful* code demonstrating just why this is a good thing. Beyond the 'gee, whiz' factor, I just don't see it. Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. -- BCE HA MOPE!
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Fri, 18 Oct 2013 20:31:05 +0400 Aleksej Saushev a...@inbox.ru wrote: I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. I think I agree with your sentiment but it seems like you are misusing the word agile here. The term refers to a very specific programming methodology and would be very beneficial if applied to kernel programming. Throwing random code into a system does not constitute agile development. http://en.wikipedia.org/wiki/Agile_software_development -- D'Arcy J.M. Cain da...@netbsd.org http://www.NetBSD.org/ IM:da...@vex.net
Re: Lua in-kernel (lbuf library)
On Oct 18, 11:03am, Marc Balmer wrote: } Am 18.10.13 10:43, schrieb Artem Falcon: } Marc Balmer marc at msys.ch writes: } Justin Cormack justin at specialbusservice.com writes: } I have been using the luajit ffi and luaffi, which let you directly } use C structs (with bitfields) in Lua to do this. It makes it easier } to reuse stuff that is already defined in C. (luaffi is not in its } current state portable but my plan is to strip out the non portable } bits, which are the function call support). } } Justin } } I had successfully used more lightweight solution called Lua AutoC [1] with } Marc's lua(4). } Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be } used in kernel, does the things in runtime, which gives the flexibility. } Cons: not widely tested, again does the things in runtime, which on other } side may give performance penalty. } } } I never used luaffi. It sounds very interesting and I think it could } be very useful to bind already defined C structs, but my purpose is to } dynamically define data layouts using Lua syntax (without parsing C } code). } } FFI in the kernel can be dangerous. Pure Lua is a perfect confinment } for code, but with an FFI a Lua script can access almost anything in the } kernel. One has to think twice if one wants that. } } Well, assuming it would be module, so I would not have to load it if I } don't want to. } } It's desirable if you're writing a device driver in Lua, as you can do } most of work from Lua code (e.g. call C methods of NetBSD driver API } and feed them with C structs and pointers). } States and explicit exports of a certain foreign functions makes things } a bit less dangerous. } But in general you're right, one should do this with care. } } lua(4) has a mechanism for Lua's 'require' statement. Normally, when } you require 'foo', it looks up wheter a kernel module name luafoo exists } and loads it. This automatic loading of modules can be turned off, to } make a module available to a state, it has to be specifically assigned. } So when you turn autoloading off, a script could not simply call a ffi } module by requiring it. } } Maybe Lua kernel modules should carry a flag whether they should allow } autoloading or not? This way, an ffi module would still be loaded into } the kernel when Lua code requires it, but lua(4) would detect the don't } autoload flag and would then not_ assign the module to the Lua state. There is already a mechanism for this, see module_autoload(9). You should always be using module_autoload() to load a module from inside the kernel. If the no autoload flag is set, then the call will fail. Thus, there is no need for lua(4) to try managing this itself. It should just attempt to load the module. If successful, great. If not, then the feature being requested isn't available. } [1] https://github.com/orangeduck/LuaAutoC } }-- End of excerpt from Marc Balmer
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hi, On Fri, Oct 18, 2013 at 11:09 AM, Matt W. Benjamin m...@linuxbox.com wrote: Hi, The linked research was performed on Linux, which has NFsv4.1 and pNFS client implementations. Evidently, you can do this kind of thing with an out-of-tree Lua kernel extension. Matt Evidently. I'm not arguing that we need that. I'm just arguing that I see benefits and none harm. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
On Fri, Oct 18, 2013 at 9:08 AM, Taylor R Campbell riastr...@netbsd.org wrote: Date: Thu, 17 Oct 2013 19:16:16 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. I don't see how an optional module could be used for attacks. If users enable that, they should know what they are doing (such as loading a kernel module). [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. I'm just showing examples of how it could be useful for user applications. I understand that you do not agree with that. But I'm not arguing that we have to add these applications into the tree. I'm arguing that we could benefit users with such a tool. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hi, On Fri, Oct 18, 2013 at 1:31 PM, Aleksej Saushev a...@inbox.ru wrote: (...) Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. No. I'm talking about put just one egg, just a device driver. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. Really don't know. I'm not a member of these communities neither I'm claiming to incorporate such works here. However, I think that there was a discussion about PacketScript on OpenWRT, but I don't know how it evolved. I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. I agree with you in this point: experimental work should be done aside from the tree. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. I guess you misunderstood that. I'm not arguing that we should do it just because there is no contrary argument. I sincerely asked 'why not?' trying to understand the contrary argumentation. Also, I'm not saying that you should tolerate my experimentation. Far away from that. I haven't committed anything nor tried to impose nothing. I'm just trying to make a point of view and understand yours. When I talked about experimentation, I was trying to say that providing support for that kind of experimentation for users sounds a good idea for me and I don't see how it is prejudicial. Which doesn't mean that I'm proposing that my personal experimentation should be in tree. Regards, -- Lourival Vieira Neto
Re: Lua in-kernel (lbuf library)
Terry Moore t...@mcci.com wrote: ... But it was really, really difficult for our embedded C programmers to use. We found the following areas of constant problem: 1) zero origin versus 1-origin arrays were a constant source of bugs, particularly when interoperating with C code. 2) the string escape sequences are almost but not completely compatible with C. In particular, in C, \123 is interpreted as octal, but in Lua \123 is decimal. This was a constant source of trouble. 3) the fact that a reference uninitialized variable or missing array element returns nil instead of throwing an exception is problematic for writing significant code that is intended to be reliable -- errors are hidden. 4) the Lua APIs are not a platform -- they change from version to version. (Working with Lua APIs and trying to track current is like working with certain open-source kernels -- you really can't maintain your investment.) And so forth. All of these decisions are actually fine, in context of a scripting language that's a world unto itself. But as a tool to give C programmers, they collectively were a real barrier to adoption. To address this, and to make it more palatable to our developers, after two years or so of experience (in 2002) MCCI forked the Lua interpreter and created a C-like variant (with renamed, stable APIs to address #4, and C-like syntax). There was a paper on this at the Lua conference in 2008: http://www.lua.org/wshop08.html#moore With that, we've been happily using something based on Lua, and I can agree with all the Lua enthusiasts: it's a great tool. If we weren't an entirely C-based company, we would not have bothered with the language changes (though we might have created stable APIs with non-clashing names). I'm not promoting our version as a solution. I am, however, pointing out that items #1 and #2 above merit careful consideration before putting Lua into a security-critical environment; C programmers won't like reviewing the code (and will tend to miss things). Yet again, this whole story just makes me wonder why AWK has never evolved to be more powerful language for this kind of purpose (Perl is not the direction I am talking about). Certainly, simplicity of AWK is valued. However, it does not mean that it should have just stopped evolving at some point in history. Even if its functions use 1-origin for arrays, the language itself is much more natural for the C programmers and the UNIX world. Personally, I would love to see more advanced AWK with just-in-time compilation.. -- Mindaugas
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hello, Lourival Vieira Neto lourival.n...@gmail.com writes: On Fri, Oct 18, 2013 at 1:31 PM, Aleksej Saushev a...@inbox.ru wrote: (...) Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. No. I'm talking about put just one egg, just a device driver. Sorry, but this is not just one egg. And counting was your reaction to complaints that almost all the code related to Lua is the code to support Lua itself rather than anything else. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). I'm of opinion that this device driver can and should stay outside the tree until its utility can be demonstrated without this much strain. At last this is one of the reasons why we support kernel modules. You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. Really don't know. I'm not a member of these communities neither I'm claiming to incorporate such works here. However, I think that there was a discussion about PacketScript on OpenWRT, but I don't know how it evolved. This demonstrates that Lua isn't actually useful in the kernel. I doubt very much that we want such unreliable development practices like agile ones in the kernel, and experimentation work can be done easier and better in a branch or a personal repository. I agree with you in this point: experimental work should be done aside from the tree. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. I guess you misunderstood that. I'm not arguing that we should do it just because there is no contrary argument. I sincerely asked 'why not?' trying to understand the contrary argumentation. Also, I'm not saying that you should tolerate my experimentation. Far away from that. I haven't committed anything nor tried to impose nothing. On my side it sounded like that, sorry, if I'm wrong. I'm just trying to make a point of view and understand yours. When I talked about experimentation, I was trying to say that providing support for that kind of experimentation for users sounds a good idea for me and I don't see how it is prejudicial. Which doesn't mean that I'm proposing that my personal experimentation should be in tree. The problem as I see it is that we have one developer (two at most) pushing hard for Lua in base and in kernel and providing no satisfactory arguments why this is to be done at all. Lack of any real code for years reinforces such doubts. Why not sounds as an argument for highly experimental work in this context. And I wouldn't have anything against this why not if all the work were dressed accordingly. For now I'd say that Lua support hasn't demonstrated any benefit. I'd say that it should be removed and the work continued in a branch until benefits become more clear. -- BCE HA MOPE!
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Hello, Lourival Vieira Neto lourival.n...@gmail.com writes: On Fri, Oct 18, 2013 at 9:08 AM, Taylor R Campbell riastr...@netbsd.org wrote: Date: Thu, 17 Oct 2013 19:16:16 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). We didn't import yet another programming language interpreter for driver development previously. Besides, what are drivers developed in Lua so far? If I understand it correctly, the only driver is the Lua interpreter itself. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. I don't see how an optional module could be used for attacks. If users enable that, they should know what they are doing (such as loading a kernel module). Was anything done to warn users? [1] https://github.com/dergraf/PacketScript [2] http://www.pdsw.org/pdsw12/papers/grawinkle-pdsw12.pdf In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. I'm just showing examples of how it could be useful for user applications. I understand that you do not agree with that. But I'm not arguing that we have to add these applications into the tree. I'm arguing that we could benefit users with such a tool. The problem is that the number of such users is negligible and all of them are developers that are able to build their kernel module outside the tree. -- BCE HA MOPE!
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Am 19.10.13 00:14, schrieb Aleksej Saushev: [...] I'm of opinion that this device driver can and should stay outside the tree until its utility can be demonstrated without this much strain. At last this is one of the reasons why we support kernel modules. The inclusion and use of Lua in base, for use in userland and the kernel, has been the subject to public discussion, it has been the topic of a GSoC project, it has been presented at many conferences, it is well received by the community at large, it has users, it has attracted new users to NetBSD, it has attracted users that are now developers, and it has, last but not least, core's blessing. Can we now please stop this useless discussion? Lua is part of NetBSD. [...]
Re: Lua in-kernel (lbuf library)
On Oct 19, 12:13am, Artem Falcon wrote: } 18.10.2013, × 21:03, John Nemeth jnem...@cue.bc.ca wrote: } On Oct 18, 11:03am, Marc Balmer wrote: } } Am 18.10.13 10:43, schrieb Artem Falcon: } } Marc Balmer marc at msys.ch writes: } } Justin Cormack justin at specialbusservice.com writes: } } I have been using the luajit ffi and luaffi, which let you directly } } use C structs (with bitfields) in Lua to do this. It makes it easier } } to reuse stuff that is already defined in C. (luaffi is not in its } } current state portable but my plan is to strip out the non portable } } bits, which are the function call support). } } } } Justin } } } } I had successfully used more lightweight solution called Lua AutoC [1] with } } Marc's lua(4). } } Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be } } used in kernel, does the things in runtime, which gives the flexibility. } } Cons: not widely tested, again does the things in runtime, which on other } } side may give performance penalty. } } } } } } I never used luaffi. It sounds very interesting and I think it could } } be very useful to bind already defined C structs, but my purpose is to } } dynamically define data layouts using Lua syntax (without parsing C } } code). } } } } FFI in the kernel can be dangerous. Pure Lua is a perfect confinment } } for code, but with an FFI a Lua script can access almost anything in the } } kernel. One has to think twice if one wants that. } } } } Well, assuming it would be module, so I would not have to load it if I } } don't want to. } } } } It's desirable if you're writing a device driver in Lua, as you can do } } most of work from Lua code (e.g. call C methods of NetBSD driver API } } and feed them with C structs and pointers). } } States and explicit exports of a certain foreign functions makes things } } a bit less dangerous. } } But in general you're right, one should do this with care. } } } } lua(4) has a mechanism for Lua's 'require' statement. Normally, when } } you require 'foo', it looks up wheter a kernel module name luafoo exists } } and loads it. This automatic loading of modules can be turned off, to } } make a module available to a state, it has to be specifically assigned. } } So when you turn autoloading off, a script could not simply call a ffi } } module by requiring it. } } } } Maybe Lua kernel modules should carry a flag whether they should allow } } autoloading or not? This way, an ffi module would still be loaded into } } the kernel when Lua code requires it, but lua(4) would detect the don't } } autoload flag and would then not_ assign the module to the Lua state. } } Probably. It should be named as 'auto assign' for clarity, as module loading } occurs anyway. } } There is already a mechanism for this, see module_autoload(9). } You should always be using module_autoload() to load a module from } inside the kernel. If the noautoload flag is set, then the call } will fail. } } This is exactly what lua(4) does on 'requiring'. } } Thus, there is no need for lua(4) to try managing this } itself. It should just attempt to load the module. If successful, } great. If not, then the feature being requested isn't available. } } kern.lua.autoload is a safety barrier. One may wish not allow any lua kernel } script to load any given lua kernel module. The lua(4) implementers can certainly do this if they want. However, module_autoload() won't be looking at this flag and will continue to refuse to autoload any module that has the noautoload flag set. Also, there is the kern.module.autoload sysctl that can prevent any module from autoloading. }-- End of excerpt from Artem Falcon
Re: RAIDframe: stripe unit per reconstruction unit
On Fri, 18 Oct 2013 15:50:55 +0200 Edgar Fuß e...@math.uni-bonn.de wrote: The man page says: The stripe units per parity unit and stripe units per reconstruction unit are normally each set to 1. While certain values above 1 are permitted, a discussion of valid values and the consequences of using anything other than 1 are outside the scope of this document. I noticed that reconstruction seems to read/write in units of one stripe unit, which in my case (being 4k) seems rather inefficient. Is it possible to accelerate recontruction by using a figure larger than 1 here? If you go here: http://www.pdl.cmu.edu/RAIDframe/ and pull down the RAIDframe Manual and go to page 73 of that manual you will find: When specifying SUsPerRU, set the number to 1 unless you are specifically implementing reconstruction under parity declustering; if so, you should read through the reconstruction code first. The answer might be yes, but I don't know what the other implications are. I'd only recommend changing this value on a RAID set you don't care about, with data you don't care about. I'd also recommend reading through the reconstruction code, looking for SUsPerRU... Later... Greg Oster
RE: Lua in-kernel (lbuf library)
Yet again, this whole story just makes me wonder why AWK has never evolved to be more powerful language for this kind of purpose (Perl is not the direction I am talking about). Certainly, simplicity of AWK is valued. However, it does not mean that it should have just stopped evolving at some point in history. Even if its functions use 1-origin for arrays, the language itself is much more natural for the C programmers and the UNIX world. Personally, I would love to see more advanced AWK with just-in-time compilation.. Indeed, we started with Lua because AWK was not embeddable and because of the 1-origin issue. (Also, because Lua supports closures; closures are really a very economical way of expressing certain kinds of operations.) But we had 10 years or so of using AWK, wrapped by sh, as a cross UNIX platform. Unfortunately, it didn't work well for Windows in those days. And sh scripts still don't, Cygwin notwithstanding. But for many things we still use AWK. --Terry
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. The problem with your approach is that such chicken-and-egg problems are to be solved _at_once_ rather than laying eggs everywhere around and have everyone else wait till at least one chicken appears. No. I'm talking about put just one egg, just a device driver. Sorry, but this is not just one egg. And counting was your reaction to complaints that almost all the code related to Lua is the code to support Lua itself rather than anything else. And counting == there is ongoing work happening outside the tree. Sure, we do not *need* a script language interpreter embedded in the kernel, as we do not need a specific file system. But I do not get why we should not. There is current development of applications being done right now. Also, there is a few interesting works that used Lunatik in Linux [1, 2] that could be done more easily now in NetBSD just because we have the right environment for that. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). I think that is why we *should* (not need) have this on the tree. IMHO. I have to point out that interesting work is commonly used as a sort of euphemism to refer to highly experimental work with unclear future. Yes. But I'm talking about interesting *user* work. I'm not claiming that they should be in the kernel. I'm just saying that, IMHO, we should incorporate a small device driver that facilitates this kind of development (outside the tree). I'm of opinion that this device driver can and should stay outside the tree until its utility can be demonstrated without this much strain. At last this is one of the reasons why we support kernel modules. Understand. You tell that there's interesting work using Lua in Linux. Was it accepted in any experimental Linux distribution like Fedora? What was the outcome of discussion among linux kernel developers? Currently there's no indication that it was accepted anywhere. Really don't know. I'm not a member of these communities neither I'm claiming to incorporate such works here. However, I think that there was a discussion about PacketScript on OpenWRT, but I don't know how it evolved. This demonstrates that Lua isn't actually useful in the kernel. I don't think so. It could even evince that, but not demonstrate. And last. The appeal to why not is defective. NetBSD is not your personal playground, there exist other people who have to deal with the inadvertent mess you can leave after you. That's why you ought to present solid arguments that justify why other people should tolerate your experimentations. I guess you misunderstood that. I'm not arguing that we should do it just because there is no contrary argument. I sincerely asked 'why not?' trying to understand the contrary argumentation. Also, I'm not saying that you should tolerate my experimentation. Far away from that. I haven't committed anything nor tried to impose nothing. On my side it sounded like that, sorry, if I'm wrong. It could sound as you want, but it wasn't what I meant. I'm just trying to make a point of view and understand yours. When I talked about experimentation, I was trying to say that providing support for that kind of experimentation for users sounds a good idea for me and I don't see how it is prejudicial. Which doesn't mean that I'm proposing that my personal experimentation should be in tree. The problem as I see it is that we have one developer (two at most) pushing hard for Lua in base and in kernel and providing no satisfactory arguments why this is to be done at all. Lack of any real code for years reinforces such doubts. Why not sounds as an argument for highly experimental work in this context. And I wouldn't have anything against this why not if all the work were dressed accordingly. For now I'd say that Lua support hasn't demonstrated any benefit. I'd say that it should be removed and the work continued in a branch until benefits become more clear. Understand. Regards, -- Lourival Vieira Neto
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Lua is a tool, not an end in itself. I think that you are formulating a chicken-and-egg problem: we need the basic support for then having applications, and we need applications for then having basic support. This is not a chicken-and-egg problem. You can make an experimental kernel with Lua support and make an experimental application in Lua, all before anything has to be committed to HEAD[*]. Then you can show that the application serves a useful function, has compelling benefits over writing it in C, and can offer confidence in robustness. [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). We didn't import yet another programming language interpreter for driver development previously. Besides, what are drivers developed in Lua so far? If I understand it correctly, the only driver is the Lua interpreter itself. I meant traditional device driver, but never mind. That is not about needing, but it is about supporting a certain kind of agile development, prototyping, customization and experimentation in the NetBSD kernel (how could it be hurtful?). Prototyping and experimentation is great! Show examples! What hurts is getting bitrotten code that nobody actually maintains or uses (when was the last Lua update in src?) and provides a new Turing machine with device access in the kernel for attack vectors. I don't see how an optional module could be used for attacks. If users enable that, they should know what they are doing (such as loading a kernel module). Was anything done to warn users? The code is not linked yet. Regards, -- Lourival Vieira Neto
RE: Lua in-kernel (lbuf library)
Just to clarify a bit Indeed, we started with Lua because AWK was not embeddable and because of the 1-origin issue. We thought, mistakenly, that a language that didn't look very much like C would cause fewer problems because of the 0-origin / 1-origin difference. Apparently, however, it's a deeper habit of thought. Ditto for the string escape sequences. Apparently the '[' and ']' for indexing, and the '\' for character escapes, trigger very deeply seated patterns. Based on our experience, it seems risky to use Lua to implement code that interoperates with kernel-like C code in security-critical contexts. We found this risk to be insurmountable. Engineers who are used to zero-origin code, and who are looking at zero-origin code for reference, will make zero-origin mistakes. This is not an argument against using Lua in user mode. However, given that the motto of NetBSD for kernel work is read the source, I worry that this human-factors issue will be hard to mitigate when using Lua in kernel mode. --Terry
Re: Why do we need lua in-tree again? Yet another call for actual evidence, please. (was Re: Moving Lua source codes)
Date: Fri, 18 Oct 2013 18:12:29 -0300 From: Lourival Vieira Neto lourival.n...@gmail.com On Fri, Oct 18, 2013 at 9:08 AM, Taylor R Campbell riastr...@netbsd.org wrote: [*] You could do this in a branch, you could do this in a private Git repository, or you could even just do this in a local CVS checkout (since kernel Lua requires no invasive changes, right?). Yes, but how do we do device driver development? We are branching the tree for each non-intrusive and disabled-by-default device driver? If we have developed a device driver for an uncommon device, we have to put it in a branch? (Please, note I'm friendly asking that). Device drivers usually have trivially demonstrable useful functions related to physical devices that one encounters on the market. Example: I wrote uatp(4) because the trackpad in my MacBook didn't work very well. I also developed uatp(4) in a local Git branch because at first it was an experiment which I expected to throw away. In the two links you gave, I found precisely five lines of Lua code, buried in the paper, and those five lines seemed to exist only for the purpose of measuring how much overhead Lua adds to the existing pNFS code or something. I'm just showing examples of how it could be useful for user applications. I understand that you do not agree with that. But I'm not arguing that we have to add these applications into the tree. I'm arguing that we could benefit users with such a tool. I don't disagree that Lua could be useful for user applications, and I'm not asking you to propose applications to add to the tree. All I'm asking for is examples of applications at all, which I couldn't find in either of the links you gave. Where is the Lua code?
Re: storage-class memory (was: Re: state of XIP?)
On Fri, Oct 18, 2013 at 09:15:33AM -0700, Matt Thomas wrote: The only problem is marking data as copy-on-write but again these pages aren't managed so the current COW code won't be happy. We shouldn't have to care about that unless we want to move to MAP_COPY from MAP_PRIVATE. Huh? I'm was talking about an executable's .data section. Since we are talking about execute-in-place. If you support MAP_COPY, then you have to support copying-on-write on the filesystem's access path to file pages. If the file page is something physically significant, this becomes problematic. If you don't support MAP_COPY but only MAP_PRIVATE, then all that matters is the mapping, so all you have to do is enter it into the pmap as readonly when you fault it in, and it shouldn't matter what kind of page it is underneath. But, as I've said, while I've written more than one VM system I don't know that much about UVM specifically, so maybe there are issues I'm not thinking of. -- David A. Holland dholl...@netbsd.org
