Re: posix_fallocate
On Sun, Nov 17, 2013 at 02:02:15AM +0100, Emmanuel Dreyfus wrote: NetBSD-current seems to lack posix_fallocate(2) http://pubs.opengroup.org/onlinepubs/009695299/functions/posix_fallocate .html Is someone already working on it, or has thoughs about how it should be implemented? I have most of the plumbing (see DIOCGDISCARDINFO and DIOCDISCARD) but not an implementation for ffs or any other fs. I think the chief question at this level is whether to support the keep the length flag for fallocate, fdiscard, both, or neither. The linux fallocate uses this to allow allocating blocks past EOF, which strikes me as nuts; however, for discarding blocks it seems that there's no inherent problem with having a hole at EOF and that shrinking the file just because you dropped the block that contains EOF is kind of silly. (And if the block containing EOF was the only block, does it ratchet the size all the way back to zero? Ugh.) I vaguely recall that some time back somebody had a preliminary implementation of either fallocate or fdiscard or both for ffs, which was not really good enough to commit. But I forget who and in what context, so of course now I can't find it. -- David A. Holland dholl...@netbsd.org
Re: posix_fallocate
David Holland dholland-t...@netbsd.org wrote: I think the chief question at this level is whether to support the keep the length flag for fallocate, fdiscard, both, or neither. The linux fallocate uses this to allow allocating blocks past EOF, which strikes me as nuts; Why is it bad? I am interested to port software, and of course it uses FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE... -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org
Re: posix_fallocate
On Sun 17 Nov 2013 at 07:59:44 +, David Holland wrote: I think the chief question at this level is whether to support the keep the length flag for fallocate, fdiscard, both, or neither. The What keep the length flag? I don't see one at the indicated URL. The way I read it, the call just fills any holes and/or extends the length of the file. There are no blocks reserved in some magic way That seems to be indicated by the wording If the offset+ len is beyond the current file size, then posix_fallocate() shall adjust the file size to offset+ len. David A. Holland -Olaf. -- ___ Olaf 'Rhialto' Seibert -- The Doctor: No, 'eureka' is Greek for \X/ rhialto/at/xs4all.nl-- 'this bath is too hot.' pgpqZgnWVYHkC.pgp Description: PGP signature
Re: posix_fallocate
Robert Elz k...@munnari.oz.au wrote: To answer both you and the Mouse - the difference is that a user process actually writing data consumes measurable resources, and thus is easy to find and kill. When everything happens in the kernel, spotting which arrantly idle user process is making it happen is not at all easy. We could fork a kernel thread that would go to userspace to do the work with a write() loop, with appropriate credentials. Does it makes sense? -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org
Re: [patch] changing lua_Number to int64_t
Am 17.11.13 00:47, schrieb Alexander Nasonov: PS Why do you still use a shadow copy of luaconf.h? Please add your changes to the main luaconf.h. If you guard your kernel changes properly with _KERNEL, they will not affect userspace. I totally agree. It was probably just an oversight when I committed the code. It does, however, not make much sense to fix a lot in 5.1, since the move to 5.2 should happen soon.
Re: [patch] changing lua_Number to int64_t
Am 17.11.13 06:30, schrieb Terry Moore: From: Lourival Vieira Neto [mailto:lourival.n...@gmail.com] Watch out, by the way, for compiled scripts; I have not checked Lua 5.x, but you may find if not careful that the compiled binary is not loadable on machines with different choices for LP64, ILP32, etc. This is somewhat independent of the choice of lua_Number mapping. Yes, Lua bytecode isn't portable in fact. BTW, loading Lua bytecode is not recommended by Lua team. It isn't safe. It's not *much* less safe than compiling and executing a string in the kernel. The only additional attack surfaces are that you can write things that the compiler wouldn't write. This can (1) cause a crash at load time, or it can (2) cause surprising behavior later. The problem is that malicious bytecode in 5.1 is possible. That is why there is a guad against loading bytecode. I suspect that anyone who would want to allow Lua in the kernel would be somewhat indifferent to type-2 safety issues. Only if loading precompiled chunks, which btw needs a custom version of luac. You can not compile bytecode for in-kernel use using the standard luac(1) program. If you don't trust the input string, you had better not give it to the kernel (whether or not it's compiled). Type-1 safety issues ought to be fixed, as careful examination of the input data only slows down the load process -- it has no run-time effects. Given that Lua byte codes can be translated into C arrays and then loaded via the API, portability issues can creep in through the back door. Best regards, --Terry
Re: posix_fallocate
I think the chief question at this level is whether to support the keep the length flag for fallocate, fdiscard, both, or neither. The linux fallocate uses this to allow allocating blocks past EOF, which strikes me as nuts; Why is it bad? Well, I'm not dholland. But my own answer to that is that it's a _major_ change to Unix filesystem semantics for it to be possible for there to be data after EOF (st_size); the nominal file size is no longer the amount of data conceptually stored, nor a cap on the amoutn of data actually stored, nor does it describe the end of that data. (Unless you are going to try to support the proposition that it is possible for a file to have blocks allocated with no contents; storage that does not contain anything is an even more radical departure from traditional semantics.) Of course, there is nothing inherently wrong with changing semantics. But a change this fundamental will affect a _lot_ of the system, far more than just adding posix_fallocate, and should, IMO, be thought out a lot more thoroughly. My own view is that it _is_ nuts, because I can't come up with a coherent paradigm for how it should work. What is the Linux position on the presence of data past st_size? Is it accessible via read()? write()? mmap()? If there are any of those via which it is not accessible, does an ftruncate() call that increases st_size() change that? Is there a way to lower st_size that doesn't free data between the new st_size and the old stsize? ...and the old actual last data? How can one find out the actual distance between the beginning of the file and the end of data? Which command-line tools have been extended to handle such data, and how? What about data before offset 0, before what has traditionally been thought of as the beginning of the file? /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: RLIMIT_FSIZE and SIGXFSZ
On Sat, Nov 16, 2013 at 12:40:22AM -0500, Mouse wrote: The documentation I have (which is consistent across 1.4T, 4.0.1, and 5.2) says that [a] file I/O operation that would create a file larger that the process' soft limit will cause the write to fail and a signal SIGXFSZ to be generated. I looked at the code (for 4.0.1, that being what's on the machine I care about at the moment) and this appears to be accurate. It seems to me it would be more useful to do something like what RLIMIT_CPU does, and generate SIGXFSZ for such operations, but fail them only when the size exceeds the hard limit. As it stands, the hard limit is useful only as a value the soft limit can't be raised above. (More useful to me, at any rate.) While this looks like an improvement, i think we don't have it right. Quoting the OpenGroup write(2) online manual[1]: [...] If a write() requests that more bytes be written than there is room for (for example, [XSI] the process'file size limit or the physical end of a medium), only as many bytes as there is room for shall be written. For example, suppose there is space for 20 bytes more in a file before reaching a limit. A write of 512 bytes will return 20. The next write of a non-zero number of bytes would give a failure return (except as noted below). [XSI] If the request would cause the file size to exceed the soft file size limit for the process and there is no room for any bytes to be written, the request shall fail and the implementation shall generate the SIGXFSZ signal for the thread. [...] When enforcing for file size limits, we don't check for free room to write some bytes. That way, we always signal the process when limit will be reached. Writes should not fail and no signal should be sent if we have some space for at least a single byte. [1] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html -- Nicolas Joly Biology IT Center Institut Pasteur, Paris.
Re: [patch] changing lua_Number to int64_t
I agree that bigger is better and %jd is much better then % PRI/SCN. But don't you think that to know the exact width is even better? You can always use sizeof if the need to know the size arises. sizeof returns the number of bytes used to store an object. This is only loosely related to the number of data bits in the object; the latter is no more than sizeof the object times CHAR_BIT, but it may be lower. Also, using an exact-width type assumes that the hardware/compiler in question _has_ such a type. It's possible that lua, NetBSD, or the combination of the two is willing to write off portability to machines where one or both of those potential portability issues becomes actual. But that seems to be asking for trouble to me; history is full of but nobody will ever want to port this to one of _those_ that come back to bite people. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: [patch] changing lua_Number to int64_t
Am 17.11.13 11:18, schrieb Alexander Nasonov: Lourival Vieira Neto wrote: I don't think that keeping compatibility with userspace Lua is the right argument. We already have lost this kind of compatibility by using an integer type for lua_Number. Expecting that kernel lua_Number works just like userspace lua_Numbers could lead to misunderstandings. Portability with userspace will arise as soon as you decide to import some Lua library to kernel space. If kernel number type is wide enough to represent all exactly representable integers from userspace, you have less worries. The basic issue here is that Lua has only _one_ numerical data type, which is an integral type in kernel, but a floating point type in userspace. Also, some libraries work in both spaces. Lua binding for proplib(3) is a good example. Yes, if one keeps above issue in mind, a script can be written for both use cases. speaking of a proplib(3) binding, is that code available somewhere? I don't see a need for bigger type unless mainstream Lua switches to long double. I don't expect it to happen any time soon. Why it should bother if Lua switches to a bigger floating-point type? See above. If they switch to long double, they will extend integer arithmetic range. PPS %PRId64 may break in C++11, space between the literals should fix it. I don't think that are plans to change kernel language to C++ ;-), however doesn't hurt to write it in clean C. Just for curiosity.. do you know why it is not allowed in C++11? I'm not sure why. It may be related to user-defined literals: http://akrzemi1.wordpress.com/2012/08/12/user-defined-literals-part-i/
Re: [patch] changing lua_Number to int64_t
Mouse wrote: Also, using an exact-width type assumes that the hardware/compiler in question _has_ such a type. It's possible that lua, NetBSD, or the combination of the two is willing to write off portability to machines where one or both of those potential portability issues becomes actual. But that seems to be asking for trouble to me; history is full of but nobody will ever want to port this to one of _those_ that come back to bite people. I was perfectly fine with long long because it's long enough to represent all integers in range [-2^53-1, 2^53-1]. As Marc pointed out, Lua has a single numeric type which is double by default. Many Lua libraries don't need FP and they use a subset of exactly representable integers (not all of them do range checks, though). Extending the range when porting from userspace to kernel will decrease the pain factor of porting. Alex
Re: posix_fallocate
Date:Sun, 17 Nov 2013 10:33:43 +0100 From:m...@netbsd.org (Emmanuel Dreyfus) Message-ID: 1lch6me.jn7y3m16232ejm%m...@netbsd.org | We could fork a kernel thread that would go to userspace to do the work | with a write() loop, with appropriate credentials. Does it makes sense? It would need to be a read/write loop, nothing says that there cannot already be blocks allocated in the space being fallocated, and their content should not change. But yes, if implemented that way it would be much less of a problem. But if implemented that way, why bother at all? Why not just put the code in a user space libc posix_fallocate() function, and be done with it, it should not require any kernel support at all. That's not true of the inverse function that David Holland referred to (though like Rhialto, I can't see what relationship that has with the posix_fallocate() call that we were asked about) for making holes in files. That one is not a problem, and needs to be in the kernel to be implemented (as the physical structure of a file is deliberately not exposed to userland.) Implementing that (assuming there's some standard interface definition for it) might be sensible, I still see no use at all for a (kernel) posix_fallocate(). kre ps: another reason that a userland process is less of a problem than the kernel interface described in the opengroup posix_fallocate() spec, is that a user process must either do multiple sys calls (and is subject to being signalled, and hence terminated, between sys calls) or malloc (or brk(2)) enough space for a buffer as big as the write call - that is typically going to limit a single sys call to no more than a few tend of GBs (on today's systems) as that's generally as big as a process can grow. On the other hand, posix_fallocate() could allocate pitabytes in a single invocation of the sys call, assuming that the filesystem had that much space available. I haven't looked recently, but last time I did, preemptible sys calls still didn't mean that userland signals would be delivered in the middle of the operation of a single sys call, nor does anything suggest that signals are supposed to interrupt the operation of posiz_fallocate() half way through - which suggests to me, that as designed, it should continue until it is finished once invoked, whatever anyone tries to do to the process that invoked it.
A Library for Converting Data to and from C Structs for Lua
I came accross a small library for converting data to an from C structs
for Lua, written by Roberto Ierusalimschy:
http://www.inf.puc-rio.br/~roberto/struct/
I plan to import it and to make it available to both lua(1) and lua(4)
as follows:
The source code will be imported into
${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
to compile on NetBSD.
Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
respective Makefiles etc.
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 11:30 AM, Alexander Nasonov al...@yandex.ru wrote: Mouse wrote: Also, using an exact-width type assumes that the hardware/compiler in question _has_ such a type. It's possible that lua, NetBSD, or the combination of the two is willing to write off portability to machines where one or both of those potential portability issues becomes actual. But that seems to be asking for trouble to me; history is full of but nobody will ever want to port this to one of _those_ that come back to bite people. I was perfectly fine with long long because it's long enough to represent all integers in range [-2^53-1, 2^53-1]. As Marc pointed out, Lua has a single numeric type which is double by default. Many Lua libraries don't need FP and they use a subset of exactly representable integers (not all of them do range checks, though). Extending the range when porting from userspace to kernel will decrease the pain factor of porting. The range [-2^53-1, 2^53-1] is not sufficient - in kernel you need to be able to deal with the longest type the kernel uses, it is incredibly annoying to have to use userdata to deal with off_t or gratuitously hope that losing some bits is ok (as happens with Lua in userspace now). As the widest type in the kernel is int64_t thats what Lua should use. (The issue of uint64_t is left as an exercise for the Lua programmer). When/if the kernel uses something longer then Lua can change, but using intmax_t is not useful as the kernel is explicit about sizes. Justin
Re: [patch] changing lua_Number to int64_t
On Sun, 17 Nov 2013, Mouse wrote: sizeof returns the number of bytes used to store an object. This is only loosely related to the number of data bits in the object; the latter is no more than sizeof the object times CHAR_BIT, but it may be lower. Also, using an exact-width type assumes that the hardware/compiler in question _has_ such a type. Yes, that's true of C. It's possible that lua, NetBSD, or the combination of the two is willing to write off portability to machines where one or both of those potential portability issues becomes actual. But that seems to be asking for trouble to me; history is full of but nobody will ever want to port this to one of _those_ that come back to bite people. NetBSD already assumes that char is exactly 8 bits, and that integer types with exactly 16, 32, and 64 bits exist. Adding more instances of the same assumptions doesn't seem like a big problem to me. If there's ever a need to port to a machine where those assumptions do not hold, then we can worry about it at that time, but I susect that it will be possible to change to using things like int_least64_t (for a type with no less than 64 bits) instead of int64_t (for a type with exactly 64 bits). --apb (Alan Barrett)
Re: A Library for Converting Data to and from C Structs for Lua
On Sun, Nov 17, 2013 at 12:05 PM, Marc Balmer m...@msys.ch wrote:
I came accross a small library for converting data to an from C structs
for Lua, written by Roberto Ierusalimschy:
http://www.inf.puc-rio.br/~roberto/struct/
I plan to import it and to make it available to both lua(1) and lua(4)
as follows:
The source code will be imported into
${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
to compile on NetBSD.
Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
respective Makefiles etc.
I always found this library so un user friendly that I would rather
program in C. Not sure it was meant as much more than a proof of
concept. YMMV.
Justin
Re: A Library for Converting Data to and from C Structs for Lua
Marc Balmer wrote:
I came accross a small library for converting data to an from C structs
for Lua, written by Roberto Ierusalimschy:
http://www.inf.puc-rio.br/~roberto/struct/
I plan to import it and to make it available to both lua(1) and lua(4)
as follows:
The source code will be imported into
${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
to compile on NetBSD.
Shouldn't it be mit/luastruct?
Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
respective Makefiles etc.
Are you going to make it kmod? It's an overkill, IMO. Not every Lua
module a kernel module. The framework should support this distinction.
PS I don't see devel/lua-struct. It'd be nice to have it in pkgsrc.
Alex
Re: A Library for Converting Data to and from C Structs for Lua
On Sun, 17 Nov 2013, Marc Balmer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. - Hubert
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 7:35 AM, Marc Balmer m...@msys.ch wrote: Am 17.11.13 04:36, schrieb Lourival Vieira Neto: On Sat, Nov 16, 2013 at 10:44 PM, Christos Zoulas chris...@zoulas.com wrote: On Nov 16, 9:30pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | On Sat, Nov 16, 2013 at 8:52 PM, Christos Zoulas chris...@astron.com wrote: | In article 52872b0c.5080...@msys.ch, Marc Balmer m...@msys.ch wrote: | Changing the number type to int64_t is certainly a good idea. Two | questions, however: | | Why not intmax_t? | | My only argument is that int64_t has a well-defined width and, AFAIK, | intmax_t could vary. But I have no strong feelings about this. Do you | think intmax_t would be better? Bigger is better. And you can use %jd to print which is a big win. I agree that bigger is better and %jd is much better then % PRI/SCN. But don't you think that to know the exact width is even better? You can always use sizeof if the need to know the size arises. I mean know it as a script programmer. I think that would be helpful to know the exact lua_Number width when you are writing a script. AFAIK, you don't have sizeof functionality from Lua. So, IMHO, lua_Number width should be fixed and documented. Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote: Am 17.11.13 04:49, schrieb Terry Moore: I believe that if you want the Lua scripts to be portable across NetBSD deployments, you should choose a well-known fixed width. I don't see this as very important. Lua scripts will hardly depend on the size of an integer. But they could. I think that the script programmers should know if the numeric data type is enough for their usage (e.g., time diffs). Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 8:52 AM, Alexander Nasonov al...@yandex.ru wrote: Marc Balmer wrote: The basic issue here is that Lua has only _one_ numerical data type, which is an integral type in kernel, but a floating point type in userspace. Right, not everyone here knows this I guess. Thanks for making it clear. Sorry, I thought that it was clear. Regards, -- Lourival Vieira Neto
Re: A Library for Converting Data to and from C Structs for Lua
Am 17.11.13 13:21, schrieb Alexander Nasonov:
Marc Balmer wrote:
I came accross a small library for converting data to an from C structs
for Lua, written by Roberto Ierusalimschy:
http://www.inf.puc-rio.br/~roberto/struct/
I plan to import it and to make it available to both lua(1) and lua(4)
as follows:
The source code will be imported into
${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
to compile on NetBSD.
Shouldn't it be mit/luastruct?
Not imo. Under external/... I usually use the original name.
Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
respective Makefiles etc.
Are you going to make it kmod? It's an overkill, IMO. Not every Lua
module a kernel module. The framework should support this distinction.
It is to be used by lua(4), it _has_ to be a kmod.
I am right now not so sure if it will be this library or if take one of
my own, which has a saner interface imo.
I need something to decode/encode binary data, however, like pack/unpack
in python.
PS I don't see devel/lua-struct. It'd be nice to have it in pkgsrc.
Re: A Library for Converting Data to and from C Structs for Lua
Am 17.11.13 13:20, schrieb Justin Cormack:
On Sun, Nov 17, 2013 at 12:05 PM, Marc Balmer m...@msys.ch wrote:
I came accross a small library for converting data to an from C structs
for Lua, written by Roberto Ierusalimschy:
http://www.inf.puc-rio.br/~roberto/struct/
I plan to import it and to make it available to both lua(1) and lua(4)
as follows:
The source code will be imported into
${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
to compile on NetBSD.
Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
respective Makefiles etc.
I always found this library so un user friendly that I would rather
program in C. Not sure it was meant as much more than a proof of
concept. YMMV.
Well possible that I uses a different library that I wrote for data
handling. You know pack/unpack from python?
Re: A Library for Converting Data to and from C Structs for Lua
Am 17.11.13 13:32, schrieb Hubert Feyrer: On Sun, 17 Nov 2013, Marc Balmer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. Yes, we need it to handle structured binary data.
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 9:30 AM, Alexander Nasonov al...@yandex.ru wrote: Mouse wrote: Also, using an exact-width type assumes that the hardware/compiler in question _has_ such a type. It's possible that lua, NetBSD, or the combination of the two is willing to write off portability to machines where one or both of those potential portability issues becomes actual. But that seems to be asking for trouble to me; history is full of but nobody will ever want to port this to one of _those_ that come back to bite people. I was perfectly fine with long long because it's long enough to represent all integers in range [-2^53-1, 2^53-1]. As Marc pointed out, Lua has a single numeric type which is double by default. Many Lua libraries don't need FP and they use a subset of exactly representable integers (not all of them do range checks, though). Extending the range when porting from userspace to kernel will decrease the pain factor of porting. I think that it is not the main point here, to decrease the pain factor of porting. Porting Lua user-space libraries should be painful enough independently of lua_Number type. IMHO, the main point here is to define a lua_Number type adjusted to kernel needs. As, we have already broke compatibility with user space libraries (for several factors, not only the missing of floating point number), I think it shouldn't matter much now. Regards, -- Lourival Vieira Neto
Re: A Library for Converting Data to and from C Structs for Lua
On Sun, Nov 17, 2013 at 11:23 AM, Marc Balmer m...@msys.ch wrote: Am 17.11.13 13:32, schrieb Hubert Feyrer: On Sun, 17 Nov 2013, Marc Balmer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. Yes, we need it to handle structured binary data. BTW, I'm developing bitwiser library with a similar purpose.It is not ready already, but it could be helpful to handle binary data without copying it from/to Lua string in the future. LHF has lpack library [1], it could be an alternative to Roberto's struct. [1] http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack Regards, -- Lourival Vieira Neto
Re: posix_fallocate
On Nov 17, 1:15pm, k...@munnari.oz.au (Robert Elz) wrote: -- Subject: Re: posix_fallocate | ps: I have not examined the FreeBSD implementation - if they've done it the | hard, safe, way, and worked out all the potential kinks, and if it doesn't | depend too much upon other aspects of their I/O system implementation (like | whatever they have to make softdeps work) then perhaps copying that might be | feasible -- if the demand for this really exists, and it isn't being requested | just because it is in the spec and NetBSD is lacking it. From the cursory look at it, they just write. christos
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 12:58 PM, Christos Zoulas chris...@zoulas.com wrote: On Nov 17, 1:36am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | Bigger is better. And you can use %jd to print which is a big win. | | I agree that bigger is better and %jd is much better then % PRI/SCN. | But don't you think that to know the exact width is even better? Why? You can always compute it at runtime if you need to. Yes, but I think that it is not a common practice in Lua. AFAIK, Lua has only math.huge [1] to tell what is the biggest number value available (what isn't the same of having the lua_Number width). Thus, we would need to provide an interface to return this information. I'd prefer to have it defined in compile time independently of platform. Anyway, as I stated before, I have no strong feelings about that.. if everybody else prefer intmax_t, I have no objection. [1] http://www.lua.org/manual/5.1/manual.html#pdf-math.huge Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Nov 17, 10:36am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | I mean know it as a script programmer. I think that would be helpful | to know the exact lua_Number width when you are writing a script. | AFAIK, you don't have sizeof functionality from Lua. So, IMHO, | lua_Number width should be fixed and documented. Lua should provide manifest constants for it (like INTMAX_MAX). Otherwise you'd be making assumptions christos
Re: [patch] changing lua_Number to int64_t
On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote: | Am 17.11.13 04:49, schrieb Terry Moore: | I believe that if you want the Lua scripts to be portable across NetBSD | deployments, you should choose a well-known fixed width. | | I don't see this as very important. Lua scripts will hardly depend on | the size of an integer. | | But they could. I think that the script programmers should know if the | numeric data type is enough for their usage (e.g., time diffs). By making it the biggest type possible, you never need to be worried. christos
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 2:02 PM, Christos Zoulas chris...@zoulas.com wrote: On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote: | Am 17.11.13 04:49, schrieb Terry Moore: | I believe that if you want the Lua scripts to be portable across NetBSD | deployments, you should choose a well-known fixed width. | | I don't see this as very important. Lua scripts will hardly depend on | the size of an integer. | | But they could. I think that the script programmers should know if the | numeric data type is enough for their usage (e.g., time diffs). By making it the biggest type possible, you never need to be worried. Right.. you just convinced me.. if no one opposes, I'll change that to intmax_t and get rid of PRI/SCNd64 =). Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 4:52 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: On Sun, Nov 17, 2013 at 2:02 PM, Christos Zoulas chris...@zoulas.com wrote: On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote: | Am 17.11.13 04:49, schrieb Terry Moore: | I believe that if you want the Lua scripts to be portable across NetBSD | deployments, you should choose a well-known fixed width. | | I don't see this as very important. Lua scripts will hardly depend on | the size of an integer. | | But they could. I think that the script programmers should know if the | numeric data type is enough for their usage (e.g., time diffs). By making it the biggest type possible, you never need to be worried. Right.. you just convinced me.. if no one opposes, I'll change that to intmax_t and get rid of PRI/SCNd64 =). 1. Lua 5.3 will have 64 bit integer support as standard, which will make interop and reuse between kernel and userspace code much easier, iff we use int64_t 2. Code will have to handle use in the kernel of uint64_t which will potentially behave differently with Lua number type being int64_t vs if it was sat int128_t which might happen, such that its unlikely to be tested properly. There is no existing system where intmax_t is not int64_t so this breakage is untestable. Justin
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 3:10 PM, Justin Cormack jus...@specialbusservice.com wrote: On Sun, Nov 17, 2013 at 4:52 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: On Sun, Nov 17, 2013 at 2:02 PM, Christos Zoulas chris...@zoulas.com wrote: On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote: | Am 17.11.13 04:49, schrieb Terry Moore: | I believe that if you want the Lua scripts to be portable across NetBSD | deployments, you should choose a well-known fixed width. | | I don't see this as very important. Lua scripts will hardly depend on | the size of an integer. | | But they could. I think that the script programmers should know if the | numeric data type is enough for their usage (e.g., time diffs). By making it the biggest type possible, you never need to be worried. Right.. you just convinced me.. if no one opposes, I'll change that to intmax_t and get rid of PRI/SCNd64 =). 1. Lua 5.3 will have 64 bit integer support as standard, which will make interop and reuse between kernel and userspace code much easier, iff we use int64_t If they are using int64_t for integers, I think it is a good reason to us to stick to int64_t. Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Sat, Nov 16, 2013 at 9:25 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: (...) I moved strtoimax.c to common/libc. Don't know if someone sees a problem on that. BTW, is it OK? Could someone review this? Regards, -- Lourival Vieira Neto
Re: A Library for Converting Data to and from C Structs for Lua
On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. This... -- David A. Holland dholl...@netbsd.org
Re: posix_fallocate
On Sun, Nov 17, 2013 at 10:24:04AM +0100, Rhialto wrote: I think the chief question at this level is whether to support the keep the length flag for fallocate, fdiscard, both, or neither. The What keep the length flag? I don't see one at the indicated URL. It's a linuxism in linux's native fallocate(). -- David A. Holland dholl...@netbsd.org
Re: posix_fallocate
On Sun, Nov 17, 2013 at 10:24:04AM +0100, Emmanuel Dreyfus wrote: I think the chief question at this level is whether to support the keep the length flag for fallocate, fdiscard, both, or neither. The linux fallocate uses this to allow allocating blocks past EOF, which strikes me as nuts; Why is it bad? What Mouse said, pretty much. Not only is it semantic nonsense, it requires reworking fsck. I am interested to port software, and of course it uses FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE... FALLOC_FL_PUNCH_HOLE is fdiscard(). I don't believe in deleting things by making an allocate call, or perpetrating other people's bad design to avoid patching a couple packages. The question is whether FALLOC_FL_KEEP_SIZE makes sense; it does for fdiscard, but I remain unconvinced in the case of fallocate. -- David A. Holland dholl...@netbsd.org
Re: posix_fallocate
On Sun, Nov 17, 2013 at 10:33:43AM +0100, Emmanuel Dreyfus wrote: To answer both you and the Mouse - the difference is that a user process actually writing data consumes measurable resources, and thus is easy to find and kill. When everything happens in the kernel, spotting which arrantly idle user process is making it happen is not at all easy. We could fork a kernel thread that would go to userspace to do the work with a write() loop, with appropriate credentials. Does it makes sense? I do not think that makes the slightest sense. Also if you really want FALLOC_FL_KEEP_SIZE, it won't be adequate. -- David A. Holland dholl...@netbsd.org
Re: RLIMIT_FSIZE and SIGXFSZ
On Sat, Nov 16, 2013 at 12:40:22AM -0500, Mouse wrote: The documentation I have (which is consistent across 1.4T, 4.0.1, and 5.2) says that [a] file I/O operation that would create a file larger that the process' soft limit will cause the write to fail and a signal SIGXFSZ to be generated. I looked at the code (for 4.0.1, that being what's on the machine I care about at the moment) and this appears to be accurate. It seems to me it would be more useful to do something like what RLIMIT_CPU does, and generate SIGXFSZ for such operations, but fail them only when the size exceeds the hard limit. As it stands, the hard limit is useful only as a value the soft limit can't be raised above. (More useful to me, at any rate.) So I think the CPU limit signalling is treated as a special case because it's pretty useless to deliver a signal whose handler you can't execute because you're past the CPU limit. Otherwise (all other limits) you get denied once you reach the soft limit, and I think that's the way things are intended to work. I think what you want is an additional signaling threshold that can be manipulated separately from the limit. -- David A. Holland dholl...@netbsd.org
Re: posix_fallocate
[posix_fallocate] We could fork a kernel thread that would go to userspace to do the work with a write() loop, with appropriate credentials. Does it makes sense? It would need to be a read/write loop, nothing says that there cannot already be blocks allocated in the space being fallocated, and their content should not change. That's a reason to put it in the kernel, actually. The kernel can tell which ranges of a file have already had space allocated for them, so it can be just writes. (Well, for local filesystems. Throw NFS or its ilk into the mix and it gets more interesting.) But if implemented that way, why bother at all? Why not just put the code in a user space libc posix_fallocate() function, and be done with it, it should not require any kernel support at all. Well, I don't know what the point of having posix_fallocate at all would be. But the obvious answer to this is atomicity, especially in the presence of other writers: the kernel is capable of making sure it doesn't destroy someone else's write by mistake, which userland isn't (in the userland implementation, there's a window between read and write when someone else can write only to get overwritten). If that matters for the target application, that's a reason to prefer an in-kernel implementation. On the other hand, posix_fallocate() could allocate pitabytes in a single invocation of the sys call, assuming that the filesystem had that much space available. I haven't looked recently, but last time I did, preemptible sys calls still didn't mean that userland signals would be delivered in the middle of the operation of a single sys call, [...] No, it doesn't mean that, strictly. But any syscall that feels like it can be signalable during any sleep involved in its operation; this was true even before multiprocessor support. There will be a loop involved _somewhere_ in any posix_fallocate() implementation, and I can't imagine that an implementation wouldn't sleep somewhere waiting for the underlying filesystem operations. Those sleeps can be made interruptible by signals; at most it will complicate the exit path. nor does anything suggest that signals are supposed to interrupt the operation of posiz_fallocate() half way through - which suggests to me, that as designed, it should continue until it is finished once invoked, whatever anyone tries to do to the process that invoked it. Actually, I see nothing in the description that prevents that. Given what it does, a half-completed posix_fallocate is indistinguishable, to userland, from a never-started posix_fallocate, provided the former hasn't got as far as affecting st_size. It would be interesting to deal with a posix_fallocate that raises st_size being interrupted after it's written some but not all of the new space, especially in the presence of another writer writing into the same area of the file, but I feel certain those problems are solvable, even if it means pushing a small fraction of the implementation down into the filesystem - and maybe not even that; the existence of kqueue's EVFILT_VNODE NOTE_WRITE means that most of the necessary machinery is already in place. This is not to say that I support the idea of adding posix_fallocate; I'm not sure what I think on that question. But some of the arguments kre has presented here do not, IMO, hold water. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: RLIMIT_FSIZE and SIGXFSZ
[...SIGXFSZ...] It seems to me it would be more useful to do something like what RLIMIT_CPU does, and generate SIGXFSZ for such operations, but fail them only when the size exceeds the hard limit. [...] So I think the CPU limit signalling is treated as a special case because it's pretty useless to deliver a signal whose handler you can't execute because you're past the CPU limit. Otherwise (all other limits) you get denied once you reach the soft limit, and I think that's the way things are intended to work. RLIMIT_FSIZE is already special in another way, a way similar to RLIMIT_CPU: it generates a signal. You don't get a SIGXPROC when trying to fork() when you're over RLIMIT_NPROC. You don't get SIGXFD when running into RLIMIT_NOFILE. (Though perhaps you should. Maybe SIGRLIMIT, with the second handler arg saying which limit?) If the write() simply failed showing EXFSIZE or some such, I might agree with you. I think what you want is an additional signaling threshold that can be manipulated separately from the limit. Sort of. For my immediate purposes, I don't even _need_ to use RLIMIT_FSIZE at all; it's simply a covenient way that's already there to hear about the condition I care about. I've made the changes I sketched for 4.0.1 and they do give the behaviour I wanted/expected. I'm going to add them to 1.4T and 5.2 (the other two versions I run) for my own use; I'll let you people hash out whether NetBSD wants them. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: A Library for Converting Data to and from C Structs for Lua
On Sun, Nov 17, 2013 at 4:39 PM, David Holland dholland-t...@netbsd.org wrote: On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. This... I think that would be nice to have Lua kernel modules in pkgsrc, if possible. Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Nov 17, 3:36pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | 1. Lua 5.3 will have 64 bit integer support as standard, which will | make interop and reuse between kernel and userspace code much easier, | iff we use int64_t | | If they are using int64_t for integers, I think it is a good reason to us to | stick to int64_t. This is not relevant. The numeric type will still be double, so forget about compatibility between kernel and userland. There is no need for the interpreter to use a fixed width type, but rather it is convenient to use the largest numeric type the machine can represent. christos
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 7:56 PM, Christos Zoulas chris...@zoulas.com wrote: On Nov 17, 3:36pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | 1. Lua 5.3 will have 64 bit integer support as standard, which will | make interop and reuse between kernel and userspace code much easier, | iff we use int64_t | | If they are using int64_t for integers, I think it is a good reason to us to | stick to int64_t. This is not relevant. The numeric type will still be double, so forget about compatibility between kernel and userland. There is no need for the interpreter to use a fixed width type, but rather it is convenient to use the largest numeric type the machine can represent. There will be two numeric types as standard, int64_t and double. It should be possible to compile the kernel Lua with only int64_t and no double support I would think, so integer only userland programs would be compatible which is a very useful feature. But the semantics of the Lua integer type will be such that it wraps at 64 bit, unlike some hypothetical larger type (that doesn't yet exist and which the kernel doesn't yet use). Justin
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 6:13 PM, Justin Cormack jus...@specialbusservice.com wrote: On Sun, Nov 17, 2013 at 7:56 PM, Christos Zoulas chris...@zoulas.com wrote: On Nov 17, 3:36pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | 1. Lua 5.3 will have 64 bit integer support as standard, which will | make interop and reuse between kernel and userspace code much easier, | iff we use int64_t | | If they are using int64_t for integers, I think it is a good reason to us to | stick to int64_t. This is not relevant. The numeric type will still be double, so forget about compatibility between kernel and userland. There is no need for the interpreter to use a fixed width type, but rather it is convenient to use the largest numeric type the machine can represent. There will be two numeric types as standard, int64_t and double. It should be possible to compile the kernel Lua with only int64_t and no double support I would think, so integer only userland programs would be compatible which is a very useful feature. But the semantics of the Lua integer type will be such that it wraps at 64 bit, unlike some hypothetical larger type (that doesn't yet exist and which the kernel doesn't yet use). Well, I don't think I fully understood that; mainly because I'm not aware about Lua 5.3. It will provide two number types for the scripts? Or you are just talking about lua_Integer type on the C-side. Lua 5.1 already has a lua_Integer type that is defined as ptrdiff_t. Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 8:39 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: Well, I don't think I fully understood that; mainly because I'm not aware about Lua 5.3. It will provide two number types for the scripts? Or you are just talking about lua_Integer type on the C-side. Lua 5.1 already has a lua_Integer type that is defined as ptrdiff_t. Yes that is correct, so 2 will be integer and 5.3 float, operations will be defined in terms of how they convert, so there will be int and float division. The draft manual is here http://www.lua.org/work/doc/manual.html (see 3.4.1). This will not happen for a while, but it will make it much easier in future for interfaces like the kernel that need 64 bit int support, which is why it is being implemented. So not being compatible with this seems a mistake. Justin
RE: [patch] changing lua_Number to int64_t
From: Marc Balmer [mailto:m...@msys.ch] It's not *much* less safe than compiling and executing a string in the kernel. The only additional attack surfaces are that you can write things that the compiler wouldn't write. This can (1) cause a crash at load time, or it can (2) cause surprising behavior later. The problem is that malicious bytecode in 5.1 is possible. That is why there is a guad against loading bytecode. Malicious in what sense? Is this a type-1 problem or a type-2 problem? Or something else that I've not considered? --Terry
Re: [patch] changing lua_Number to int64_t
On Sun, Nov 17, 2013 at 6:45 PM, Justin Cormack jus...@specialbusservice.com wrote: On Sun, Nov 17, 2013 at 8:39 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: Well, I don't think I fully understood that; mainly because I'm not aware about Lua 5.3. It will provide two number types for the scripts? Or you are just talking about lua_Integer type on the C-side. Lua 5.1 already has a lua_Integer type that is defined as ptrdiff_t. Yes that is correct, so 2 will be integer and 5.3 float, operations will be defined in terms of how they convert, so there will be int and float division. The draft manual is here http://www.lua.org/work/doc/manual.html (see 3.4.1). This will not happen for a while, but it will make it much easier in future for interfaces like the kernel that need 64 bit int support, which is why it is being implemented. So not being compatible with this seems a mistake. Humm.. I think that ยง2.1 brings a good argument: Standard Lua uses 64-bit integers and double-precision floats, (...). I think that would not hurt to stick to the future standard; once 64 bit is good enough for kernel purposes. Regards, -- Lourival Vieira Neto
Re: [patch] changing lua_Number to int64_t
On Nov 17, 7:14pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote: -- Subject: Re: [patch] changing lua_Number to int64_t | Humm.. I think that =A72.1 brings a good argument: Standard Lua uses | 64-bit integers and double-precision floats, (...). I think that | would not hurt to stick to the future standard; once 64 bit is good | enough for kernel purposes. Ok, christos
Re: A Library for Converting Data to and from C Structs for Lua
Am 17.11.13 14:43, schrieb Lourival Vieira Neto: On Sun, Nov 17, 2013 at 11:23 AM, Marc Balmer m...@msys.ch wrote: Am 17.11.13 13:32, schrieb Hubert Feyrer: On Sun, 17 Nov 2013, Marc Balmer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. Yes, we need it to handle structured binary data. BTW, I'm developing bitwiser library with a similar purpose.It is not ready already, but it could be helpful to handle binary data without copying it from/to Lua string in the future. LHF has lpack library [1], it could be an alternative to Roberto's struct. [1] http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack So before I commit anything (if at all), we will coordinate this effort and maybe take the best of all libraries. So that we have only one module dealing with data.
Re: A Library for Converting Data to and from C Structs for Lua
Am 17.11.13 20:40, schrieb Lourival Vieira Neto: On Sun, Nov 17, 2013 at 4:39 PM, David Holland dholland-t...@netbsd.org wrote: On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. This... I think that would be nice to have Lua kernel modules in pkgsrc, if possible. No, I don't think so. They interact to much with the system, they need to be part of the system.
Re: posix_fallocate
Date:Sun, 17 Nov 2013 14:12:16 -0500 (EST) From:Mouse mo...@rodents-montreal.org Message-ID: 201311171912.oaa17...@chip.rodents-montreal.org | That's a reason to put it in the kernel, actually. The kernel can tell | which ranges of a file have already had space allocated for them, so it | can be just writes. Sure, but that's just an optimisation, and surely only matters if this is done enough that the difference is actually of significance. Do you really believe it is, or ever will be? | Well, I don't know what the point of having posix_fallocate at all | would be. Agree there, it seems mostly useless. | But the obvious answer to this is atomicity, especially in | the presence of other writers: Really? You're imagining multiple writers, writing to the same file, without any co-ordination (like locking, or whatever other way works) and you're actually worried about unpredictable results??? Really? But beyond that, since you plan on allowing signals to interrupt the operation part way through, what atomicity are you really getting anyway? | No, it doesn't mean that, strictly. But any syscall that feels like it | can be signalable during any sleep involved in its operation; Sure, though the internal implementation of the sys call has to explicitly make that happen - and to be useful, the sys call interface really has to be designed to support it. This one isn't (and as specified, cannot be.) That is, if you imagine the sys call in question being SIGKILL (as in the scenario I postulated initially) there's no problem - if the code for the sys call wants to allow that to work before it completes, it easily can. But instead imagine it is SIGALRM - now there's a problem, since there's no way in the interface to report how much of the work was done, the only thing that can be done if EINTR happens, is repeat the sys call. For a periodic sys call like SIGALRM (or any of the other timer signals) chances are that a signal will arrive during the sys call every time, resulting in an infinite loop of fallocate/signal/fallocate/signal... This thing is just poorly designed. Let's just ignore it (by all means, implement the hole making part from the linux interface if desired, but the allocation side just isn't needed). | Given what it does, a half-completed posix_fallocate is indistinguishable, | to userland, from a never-started posix_fallocate, provided the former | hasn't got as far as affecting st_size. Of course, I'm only really interested in cases where the size can't help being affected, as it starts at 0 - cases where actual holes in the middle of a file are being filled in I see as so unlikely in practice that they're totally irrelevant (like unless the app has just made the file by seeking forward and writing a byte, how does it ever know whether or not there are holes to fill? And why would it do it that way, and follow by fallocate() ? If fallocate() exists, surely it would just use that to make the file?) But as above, that it is indistinguishable is the problem. kre ps: Note that I see that the linux way of handling fallocate isn't to write blocks of 0's, but to allocate uninitialised blocks, and mark them uninitialised - I assume the way that works, is that if an app reads one of those blocks, it is just given 0's - and if it writes (the expected operation) whatever is there (0's or junk) just gets overwritten. That way they make fallocate() really fast (just assigns block numbers to the file) but it requires that the un-init flag (wheverever, and however, they keep that) is 100% reliable. Nothing is really that reliable... FFS doesn't have any mechanism to do that, so actually writing 0's would be the only way, and given that, fallocate() looks to be a total waste of time - again, given that it is an optional sys call, that no-one is required to implement, and so which no-one can assume actually exists.
Re: [patch] changing lua_Number to int64_t
Am 17.11.13 22:03, schrieb Terry Moore: From: Marc Balmer [mailto:m...@msys.ch] It's not *much* less safe than compiling and executing a string in the kernel. The only additional attack surfaces are that you can write things that the compiler wouldn't write. This can (1) cause a crash at load time, or it can (2) cause surprising behavior later. The problem is that malicious bytecode in 5.1 is possible. That is why there is a guad against loading bytecode. Malicious in what sense? Is this a type-1 problem or a type-2 problem? Or something else that I've not considered? See Peter Cawleys talk he gave during LWS 2011 in Frick. Mitigating the danger of malicious bytecode Peter Cawley (University of Oxford) In Lua 5.1 and 5.2, the family of load functions permit the loading of precompiled bytecode in addition to plain source code. We shall look at some relevant details of the bytecode, and see how these details can be maliciously abused to escape from a Lua sandbox (in both 5.1 and 5.2). With the dangers presented, we shall then move to look at two ways of mitigating them. Firstly, one can wrap the load functions as to reject bytecode before it is loaded. Secondly, and more interestingly, one can attempt to algorithmically decide whether a given piece of bytecode is malicious or not. We shall look at two such algorithms for validating bytecode: LBCV, which tries to reject as little as possible, and my current experimental verifier, which tries to accept as little as possible. http://www.lua.org/wshop11/Cawley.pdf
Re: posix_fallocate
On Mon 18 Nov 2013 at 05:11:35 +0700, Robert Elz wrote: how does it ever know whether or not there are holes to fill? And why would it do it that way, and follow by fallocate() ? Wasn't there some (proposed? actual?) interface to find holes in files? -Olaf. -- ___ Olaf 'Rhialto' Seibert -- The Doctor: No, 'eureka' is Greek for \X/ rhialto/at/xs4all.nl-- 'this bath is too hot.' pgpJE4ppPn_az.pgp Description: PGP signature
RE: [patch] safety of Lua byte code
Am 17.11.13 22:03, schrieb Terry Moore: From: Marc Balmer [mailto:m...@msys.ch] It's not *much* less safe than compiling and executing a string in the kernel. The only additional attack surfaces are that you can write things that the compiler wouldn't write. This can (1) cause a crash at load time, or it can (2) cause surprising behavior later. The problem is that malicious bytecode in 5.1 is possible. That is why there is a guad against loading bytecode. Malicious in what sense? Is this a type-1 problem or a type-2 problem? Or something else that I've not considered? See Peter Cawleys talk he gave during LWS 2011 in Frick. Mitigating the danger of malicious bytecode So we're discussing what I called a type-2 problem, escaping the sandbox. Really, however, if you load any code of any complexity into any VM, you are at risk -- no matter how it the code is presented (byte code or source). If you are loading code you don't know into a Turning-complete VM that's running in the kernel, you are making a very strong assumption (and one that seems, based on experience in the software industry to be unwarranted) about the correctness of the compiler, run-time system, etc. I am not referring to Lua in particular; the code is pretty good. I'm referring instead to the general complexity of the VM environment. I personally would never run untrusted code of any kind in the kernel, whether provided in source form or binary form, whether sandboxed or not, unless the VM is very limited (along the lines of the BPF VM). This isn't to say that Lua is not useful in the kernel. It is, however, to say that I'd be very reluctant to let an adversary have carte blanche to inject Lua code, in source or binary form, into the kernel. (This goes for *any* language, C, Python, binary. Injecting Lua code into your kernel is like doing an insmod -- you had better be asserting that you trust the code *completely*. In fact, the use case of a serial protocol coded in Lua is exactly an insmod-like application.) Nobody would seriously claim, I think, that one can algorithmically determine whether a given fragment of Lua *source* code is malicious. Yes, Lua byte codes can be malicious in additional ways, but I think worrying about this misleads us about the nature of the problem. If you're worrying about the safety of the compiled code in this sense, then you should really be worrying about the safety of having the compiler in the kernel at all. I think that the paper you refer to is more motivated by use cases such as running Lua in a web browser a-la Java, or in a game WoW, where the code is injected into a VM without any real authentication. Best regards, --Terry
Re: posix_fallocate
Date:Sun, 17 Nov 2013 23:49:50 +0100 From:Rhialto rhia...@falu.nl Message-ID: 20131117224950.gh23...@falu.nl | Wasn't there some (proposed? actual?) interface to find holes in files? I was told off-list about SEEK_HOLE and SEEK_DATA that allow (if the filesys supports them) apps to jump to the next hole, then end of hole, and that way work out where the holes are. According to the linux man page, they exist there, and FreeBSD DragonFly and Solaris - and might come to posix. Personally, I think the underlying structure of files should not be made visible to apps at all - they should just see a byte stream (perhaps with an advisory useful block size to write in). kre
Re: A Library for Converting Data to and from C Structs for Lua
On Sun, Nov 17, 2013 at 7:57 PM, Marc Balmer m...@msys.ch wrote: Am 17.11.13 14:43, schrieb Lourival Vieira Neto: On Sun, Nov 17, 2013 at 11:23 AM, Marc Balmer m...@msys.ch wrote: Am 17.11.13 13:32, schrieb Hubert Feyrer: On Sun, 17 Nov 2013, Marc Balmer wrote: I plan to import it and to make it available to both lua(1) and lua(4) I wonder if we really need to get all this into NetBSD, instead of moving it to pkgsrc somehow. Yes, we need it to handle structured binary data. BTW, I'm developing bitwiser library with a similar purpose.It is not ready already, but it could be helpful to handle binary data without copying it from/to Lua string in the future. LHF has lpack library [1], it could be an alternative to Roberto's struct. [1] http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack So before I commit anything (if at all), we will coordinate this effort and maybe take the best of all libraries. So that we have only one module dealing with data. OK.. Regards, -- Lourival Vieira Neto
Re: [patch] safety of Lua byte code
Nobody would seriously claim, I think, that one can algorithmically determine whether a given fragment of Lua *source* code is malicious. It is impossible, because the _very same code_ may be fairly called malicious when presented for execution by an attacker but not when presented for execution by a suitably authorized person. Therefore, information beyond the code itself is necessary to determine malice. Note, too, that there is nothing Lua-specific in the above argument. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: A Library for Converting Data to and from C Structs for Lua
On Nov 17, 11:02pm, Marc Balmer wrote: } Am 17.11.13 20:40, schrieb Lourival Vieira Neto: } On Sun, Nov 17, 2013 at 4:39 PM, David Holland dholland-t...@netbsd.org wrote: } On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote: }I plan to import it and to make it available to both lua(1) and lua(4) } }I wonder if we really need to get all this into NetBSD, }instead of moving it to pkgsrc somehow. } } This... } } I think that would be nice to have Lua kernel modules in pkgsrc, if possible. } } No, I don't think so. They interact to much with the system, they need } to be part of the system. Uh, no. The whole idea behind modules clearly means being able to use third party code. We should be able to have modules in pkgsrc. There are no modules in pkgsrc yet, but that's just a matter of figuring out the best way to do it. There is no reason why all modules must be included with the system. }-- End of excerpt from Marc Balmer
Re: posix_fallocate
That's a reason to put it in the kernel, actually. The kernel can tell which ranges of a file have already had space allocated for them, so it can be just writes. Sure, but that's just an optimisation, and surely only matters if this is done enough that the difference is actually of significance. Do you really believe it is, or ever will be? Not just an optimization; it also affects correctness - see below. But the obvious answer to this is atomicity, especially in the presence of other writers: Really? You're imagining multiple writers, writing to the same file, without any co-ordination (like locking, or whatever other way works) and you're actually worried about unpredictable results??? Really? Actually, thinking about it more, atomicity is the wrong word. The correct thing to worry about here is that, as I read the pointed-to webpage, posix_fallocate() is defined to do nothing to already-present data. But, when racing with another writer, a non-kernel implementation will always have conditions under which it can destroy data written by some other process. No, it doesn't mean that, strictly. But any syscall that feels like it can be signalable during any sleep involved in its operation; Sure, though the internal implementation of the sys call has to explicitly make that happen Well, sure. - and to be useful, the sys call interface really has to be designed to support it. This one isn't (and as specified, cannot be.) Huh? I don't see it that way. That is, if you imagine the [signal] in question being SIGKILL (as in the scenario I postulated initially) there's no problem - if the code for the sys call wants to allow that to work before it completes, it easily can. But instead imagine it is SIGALRM - now there's a problem, since there's no way in the interface to report how much of the work was done, the only thing that can be done if EINTR happens, is repeat the sys call. Right. That's a reason to put it in the kernel, so it doesn't need to redo past work. (This is not a complete fix, because it applies only to calls which don't increase st_size.) I note that the documentation webpage lists EINTR. For a periodic [signal] like SIGALRM (or any of the other timer signals) chances are that a signal will arrive during the sys call every time, resulting in an infinite loop of fallocate/signal/fallocate/signal... Sure. There are lots of ways programmers can write code which ends up in livelock. I don't see how this one deserves any more special treatment than the others. [...] - cases where actual holes in the middle of a file are being filled in I see as so unlikely in practice that they're totally irrelevant (like unless the app has just made the file by seeking forward and writing a byte, how does it ever know whether or not there are holes to fill? And why would it do it that way, and follow by fallocate()? If fallocate() exists, surely it would just use that to make the file?) I see it as being intended for programs doing things like databases: they may want to allocate disk space when they know they'll want it but it's still easy to back out of the operation if it fails. Once the space is allocated, then they can carry on knowing they won't run into a full disk partway through, later, when it's much harder to deal with. In this paradigm, the application doesn't know whether there used to be a hole there and doesn't care; the important thing is that, after the call, there isn't. I'm not sure why it's better than (read-and-)write for such application, but that's the use case it feels designed for to me. (Also, seeking and writing is not the only way to create a large file; truncate/ftruncate can extend files on at least some systems.) ps: Note that I see that the linux way of handling fallocate isn't to write blocks of 0's, but to allocate uninitialised blocks, and mark them uninitialised [...] That way they make fallocate() really fast (just assigns block numbers to the file) but it requires that the un-init flag (wheverever, and however, they keep that) is 100% reliable. Nothing is really that reliable... Not 100% reliable, but at least as reliable as the rest of the filesystem. FFS assumes that di_db[] in inodes, and the block allocation bitmaps, won't change behind its back, too; I don't see why this would be any different, really. I'm not sure whether I'd be willing to pay one more bit per frag in order to (greatly) speed up reads of allocated but unwritten blocks; my own guess would be that such things are rare enough that optimizing them doesn't really matter - though, of course, I don't often find uses for things I don't have. Perhaps Linux has found a real use for such things. The major use I can think of for them are things like NFS swapfiles, where you want to allocate the whole file but have no need to write it. My own livebackup is in a similar situation; it could benefit from an allocated but unwritten state
re: posix_fallocate
Personally, I think the underlying structure of files should not be made visible to apps at all - they should just see a byte stream (perhaps with an advisory useful block size to write in). i would buy this argument if mmap()ing a large sparse file and filling it up randomly (but with relatively large chunks at a time) did not lead to severely fragmented files that can take 10x to read, vs one written with plain sequential write() calls. because of this, some workaround is necessary. it is very disappointing to see an average of 120 iops of 64KB each (and only because i formatted my FS with 64kb blocks/frags!.) whe sequentially reading a file created by mmap(). posix_fallocate() is answering a real problem. the work around today is to write the file, which doubles the IO traffic, and i am not sure we can do better with FFS, due to the issues you've mentioned, but there are many other filesystems in existence that do allow block allocation without exposing prior data or initialisation. given the current issues, i'd be happy with a userspace implementation. .mrg.
