Re: posix_fallocate

2013-11-17 Thread David Holland
On Sun, Nov 17, 2013 at 02:02:15AM +0100, Emmanuel Dreyfus wrote:
  NetBSD-current seems to lack posix_fallocate(2)
  http://pubs.opengroup.org/onlinepubs/009695299/functions/posix_fallocate
  .html
  
  Is someone already working on it, or has thoughs about how it should be
  implemented?

I have most of the plumbing (see DIOCGDISCARDINFO and DIOCDISCARD)
but not an implementation for ffs or any other fs.

I think the chief question at this level is whether to support the
keep the length flag for fallocate, fdiscard, both, or neither. The
linux fallocate uses this to allow allocating blocks past EOF, which
strikes me as nuts; however, for discarding blocks it seems that
there's no inherent problem with having a hole at EOF and that
shrinking the file just because you dropped the block that contains
EOF is kind of silly. (And if the block containing EOF was the only
block, does it ratchet the size all the way back to zero? Ugh.)

I vaguely recall that some time back somebody had a preliminary
implementation of either fallocate or fdiscard or both for ffs, which
was not really good enough to commit. But I forget who and in what
context, so of course now I can't find it.

-- 
David A. Holland
dholl...@netbsd.org


Re: posix_fallocate

2013-11-17 Thread Emmanuel Dreyfus
David Holland dholland-t...@netbsd.org wrote:

 I think the chief question at this level is whether to support the
 keep the length flag for fallocate, fdiscard, both, or neither. The
 linux fallocate uses this to allow allocating blocks past EOF, which
 strikes me as nuts; 

Why is it bad?

I am interested to port software, and of course it uses
FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE...

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
m...@netbsd.org


Re: posix_fallocate

2013-11-17 Thread Rhialto
On Sun 17 Nov 2013 at 07:59:44 +, David Holland wrote:
 I think the chief question at this level is whether to support the
 keep the length flag for fallocate, fdiscard, both, or neither. The

What keep the length flag? I don't see one at the indicated URL.

The way I read it, the call just fills any holes and/or extends the
length of the file. There are no blocks reserved in some magic way
That seems to be indicated by the wording  If the offset+ len is beyond
the current file size, then posix_fallocate() shall adjust the file size
to offset+ len.

 David A. Holland
-Olaf.
-- 
___ Olaf 'Rhialto' Seibert  -- The Doctor: No, 'eureka' is Greek for
\X/ rhialto/at/xs4all.nl-- 'this bath is too hot.'



pgpqZgnWVYHkC.pgp
Description: PGP signature


Re: posix_fallocate

2013-11-17 Thread Emmanuel Dreyfus
Robert Elz k...@munnari.oz.au wrote:

 To answer both you and the Mouse - the difference is that a user process
 actually writing data consumes measurable resources, and thus is easy to
 find and kill.   When everything happens in the kernel, spotting which
 arrantly idle user process is making it happen is not at all easy.

We could fork a kernel thread that would go to userspace to do the work
with a write() loop, with appropriate credentials. Does it makes sense?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
m...@netbsd.org


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Marc Balmer
Am 17.11.13 00:47, schrieb Alexander Nasonov:

 PS Why do you still use a shadow copy of luaconf.h? Please add your
 changes to the main luaconf.h. If you guard your kernel changes properly
 with _KERNEL, they will not affect userspace.

I totally agree.  It was probably just an oversight when I committed the
code.

It does, however, not make much sense to fix a lot in 5.1, since the
move to 5.2 should happen soon.




Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Marc Balmer
Am 17.11.13 06:30, schrieb Terry Moore:
 From: Lourival Vieira Neto [mailto:lourival.n...@gmail.com]
 Watch out, by the way, for compiled scripts; I have not checked Lua 5.x,
 but
 you may find if not careful that the compiled binary is not loadable on
 machines with different choices for LP64, ILP32, etc. This is somewhat
 independent of the choice of lua_Number mapping.

 Yes, Lua bytecode isn't portable in fact. BTW, loading Lua bytecode is
 not recommended by Lua team. It isn't safe.
 
 It's not *much* less safe than compiling and executing a string in the
 kernel. The only additional attack surfaces are that you can write things
 that the compiler wouldn't write. This can (1) cause a crash at load time,
 or it can (2) cause surprising behavior later. 

The problem is that malicious bytecode in 5.1 is possible.  That is why
there is a guad against loading bytecode.

 I suspect that anyone who would want to allow Lua in the kernel would be
 somewhat indifferent to type-2 safety issues.

Only if loading precompiled chunks, which btw needs a custom version of
luac.  You can not compile bytecode for in-kernel use using the standard
luac(1) program.


If you don't trust the input
 string, you had better not give it to the kernel (whether or not it's
 compiled). Type-1 safety issues ought to be fixed, as careful examination of
 the input data only slows down the load process -- it has no run-time
 effects. 
 
 Given that Lua byte codes can be translated into C arrays and then loaded
 via the API, portability issues can creep in through the back door. 
 
 Best regards,
 --Terry
 



Re: posix_fallocate

2013-11-17 Thread Mouse
 I think the chief question at this level is whether to support the
 keep the length flag for fallocate, fdiscard, both, or neither.
 The linux fallocate uses this to allow allocating blocks past EOF,
 which strikes me as nuts;
 Why is it bad?

Well, I'm not dholland.  But my own answer to that is that it's a
_major_ change to Unix filesystem semantics for it to be possible for
there to be data after EOF (st_size); the nominal file size is no
longer the amount of data conceptually stored, nor a cap on the amoutn
of data actually stored, nor does it describe the end of that data.
(Unless you are going to try to support the proposition that it is
possible for a file to have blocks allocated with no contents; storage
that does not contain anything is an even more radical departure from
traditional semantics.)

Of course, there is nothing inherently wrong with changing semantics.
But a change this fundamental will affect a _lot_ of the system, far
more than just adding posix_fallocate, and should, IMO, be thought out
a lot more thoroughly.

My own view is that it _is_ nuts, because I can't come up with a
coherent paradigm for how it should work.  What is the Linux position
on the presence of data past st_size?  Is it accessible via read()?
write()?  mmap()?  If there are any of those via which it is not
accessible, does an ftruncate() call that increases st_size() change
that?  Is there a way to lower st_size that doesn't free data between
the new st_size and the old stsize?  ...and the old actual last data?
How can one find out the actual distance between the beginning of the
file and the end of data?  Which command-line tools have been extended
to handle such data, and how?  What about data before offset 0, before
what has traditionally been thought of as the beginning of the file?

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B


Re: RLIMIT_FSIZE and SIGXFSZ

2013-11-17 Thread Nicolas Joly
On Sat, Nov 16, 2013 at 12:40:22AM -0500, Mouse wrote:
 The documentation I have (which is consistent across 1.4T, 4.0.1, and
 5.2) says that [a] file I/O operation that would create a file larger
 that the process' soft limit will cause the write to fail and a signal
 SIGXFSZ to be generated.  I looked at the code (for 4.0.1, that being
 what's on the machine I care about at the moment) and this appears to
 be accurate.
 
 It seems to me it would be more useful to do something like what
 RLIMIT_CPU does, and generate SIGXFSZ for such operations, but fail
 them only when the size exceeds the hard limit.  As it stands, the hard
 limit is useful only as a value the soft limit can't be raised above.
 (More useful to me, at any rate.)

While this looks like an improvement, i think we don't have it right.

Quoting the OpenGroup write(2) online manual[1]:

[...]
If a write() requests that more bytes be written than there is room
for (for example, [XSI] the process'file size limit or the physical
end of a medium), only as many bytes as there is room for shall be
written. For example, suppose there is space for 20 bytes more in a
file before reaching a limit. A write of 512 bytes will return 20. The
next write of a non-zero number of bytes would give a failure return
(except as noted below).

[XSI] If the request would cause the file size to exceed the soft file
size limit for the process and there is no room for any bytes to be
written, the request shall fail and the implementation shall generate
the SIGXFSZ signal for the thread.
[...]

When enforcing for file size limits, we don't check for free room to
write some bytes. That way, we always signal the process when limit
will be reached. Writes should not fail and no signal should be sent
if we have some space for at least a single byte.

[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html

-- 
Nicolas Joly

Biology IT Center
Institut Pasteur, Paris.


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Mouse
 I agree that bigger is better and %jd is much better then %
 PRI/SCN.  But don't you think that to know the exact width is even
 better?
 You can always use sizeof if the need to know the size arises.

sizeof returns the number of bytes used to store an object.  This is
only loosely related to the number of data bits in the object; the
latter is no more than sizeof the object times CHAR_BIT, but it may be
lower.

Also, using an exact-width type assumes that the hardware/compiler in
question _has_ such a type.

It's possible that lua, NetBSD, or the combination of the two is
willing to write off portability to machines where one or both of those
potential portability issues becomes actual.  But that seems to be
asking for trouble to me; history is full of but nobody will ever want
to port this to one of _those_ that come back to bite people.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Marc Balmer
Am 17.11.13 11:18, schrieb Alexander Nasonov:
 Lourival Vieira Neto wrote:
 I don't think that keeping compatibility with userspace Lua is the
 right argument. We already have lost this kind of compatibility by
 using an integer type for lua_Number. Expecting that kernel lua_Number
 works just like userspace lua_Numbers could lead to misunderstandings.
 
 Portability with userspace will arise as soon as you decide to import
 some Lua library to kernel space. If kernel number type is wide enough
 to represent all exactly representable integers from userspace, you have
 less worries.

The basic issue here is that Lua has only _one_ numerical data type,
which is an integral type in kernel, but a floating point type in userspace.

 Also, some libraries work in both spaces. Lua binding for proplib(3) is
 a good example.

Yes, if one keeps above issue in mind, a script can be written for both
use cases.  speaking of a proplib(3) binding, is that code available
somewhere?

 
 I don't see a need for bigger type unless mainstream Lua switches to
 long double. I don't expect it to happen any time soon.

 Why it should bother if Lua switches to a bigger floating-point type?
 
 See above. If they switch to long double, they will extend integer
 arithmetic range.
 
 PPS %PRId64 may break in C++11, space between the literals should fix it.

 I don't think that are plans to change kernel language to C++ ;-),
 however doesn't hurt to write it in clean C.

 Just for curiosity.. do you know why it is not allowed in C++11?
 
 I'm not sure why. It may be related to user-defined literals:
 http://akrzemi1.wordpress.com/2012/08/12/user-defined-literals-part-i/



Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Alexander Nasonov
Mouse wrote:
 Also, using an exact-width type assumes that the hardware/compiler in
 question _has_ such a type.
 
 It's possible that lua, NetBSD, or the combination of the two is
 willing to write off portability to machines where one or both of those
 potential portability issues becomes actual.  But that seems to be
 asking for trouble to me; history is full of but nobody will ever want
 to port this to one of _those_ that come back to bite people.

I was perfectly fine with long long because it's long enough to
represent all integers in range [-2^53-1, 2^53-1].

As Marc pointed out, Lua has a single numeric type which is double
by default. Many Lua libraries don't need FP and they use a subset of
exactly representable integers (not all of them do range checks, though).
Extending the range when porting from userspace to kernel will decrease
the pain factor of porting.

Alex


Re: posix_fallocate

2013-11-17 Thread Robert Elz
Date:Sun, 17 Nov 2013 10:33:43 +0100
From:m...@netbsd.org (Emmanuel Dreyfus)
Message-ID:  1lch6me.jn7y3m16232ejm%m...@netbsd.org

  | We could fork a kernel thread that would go to userspace to do the work
  | with a write() loop, with appropriate credentials. Does it makes sense?

It would need to be a read/write loop, nothing says that there cannot already
be blocks allocated in the space being fallocated, and their content should
not change.

But yes, if implemented that way it would be much less of a problem.

But if implemented that way, why bother at all?  Why not just put the
code in a user space libc posix_fallocate() function, and be done with
it, it should not require any kernel support at all.

That's not true of the inverse function that David Holland referred to
(though like Rhialto, I can't see what relationship that has with the
posix_fallocate() call that we were asked about) for making holes in files.
That one is not a problem, and needs to be in the kernel to be implemented
(as the physical structure of a file is deliberately not exposed to userland.)
Implementing that (assuming there's some standard interface definition for
it) might be sensible, I still see no use at all for a (kernel) 
posix_fallocate().

kre

ps: another reason that a userland process is less of a problem than the
kernel interface described in the opengroup posix_fallocate() spec, is
that a user process must either do multiple sys calls (and is subject to
being signalled, and hence terminated, between sys calls) or malloc (or
brk(2)) enough space for a buffer as big as the write call - that is
typically going to limit a single sys call to no more than a few tend of GBs
(on today's systems) as that's generally as big as a process can grow.

On the other hand, posix_fallocate() could allocate pitabytes in a single
invocation of the sys call, assuming that the filesystem had that much
space available.   I haven't looked recently, but last time I did,
preemptible sys calls still didn't mean that userland signals would be
delivered in the middle of the operation of a single sys call, nor does
anything suggest that signals are supposed to interrupt the operation of
posiz_fallocate() half way through - which suggests to me, that as designed,
it should continue until it is finished once invoked, whatever anyone tries
to do to the process that invoked it.



A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Marc Balmer
I came accross a small library for converting data to an from C structs
for Lua, written by Roberto Ierusalimschy:

http://www.inf.puc-rio.br/~roberto/struct/

I plan to import it and to make it available to both lua(1) and lua(4)
as follows:

The source code will be imported into
${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
to compile on NetBSD.

Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
respective Makefiles etc.



Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Justin Cormack
On Sun, Nov 17, 2013 at 11:30 AM, Alexander Nasonov al...@yandex.ru wrote:
 Mouse wrote:
 Also, using an exact-width type assumes that the hardware/compiler in
 question _has_ such a type.

 It's possible that lua, NetBSD, or the combination of the two is
 willing to write off portability to machines where one or both of those
 potential portability issues becomes actual.  But that seems to be
 asking for trouble to me; history is full of but nobody will ever want
 to port this to one of _those_ that come back to bite people.

 I was perfectly fine with long long because it's long enough to
 represent all integers in range [-2^53-1, 2^53-1].

 As Marc pointed out, Lua has a single numeric type which is double
 by default. Many Lua libraries don't need FP and they use a subset of
 exactly representable integers (not all of them do range checks, though).
 Extending the range when porting from userspace to kernel will decrease
 the pain factor of porting.

The range [-2^53-1, 2^53-1] is not sufficient - in kernel you need to
be able to deal with the longest type the kernel uses, it is
incredibly annoying to have to use userdata to deal with off_t or
gratuitously hope that losing some bits is ok (as happens with Lua in
userspace now). As the widest type in the kernel is int64_t thats
what Lua should use. (The issue of uint64_t is left as an exercise for
the Lua programmer). When/if the kernel uses something longer then Lua
can change, but using intmax_t is not useful as the kernel is explicit
about sizes.

Justin


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Alan Barrett

On Sun, 17 Nov 2013, Mouse wrote:
sizeof returns the number of bytes used to store an object. 
This is only loosely related to the number of data bits in the 
object; the latter is no more than sizeof the object times 
CHAR_BIT, but it may be lower.


Also, using an exact-width type assumes that the 
hardware/compiler in question _has_ such a type.


Yes, that's true of C.

It's possible that lua, NetBSD, or the combination of the two is 
willing to write off portability to machines where one or both 
of those potential portability issues becomes actual.  But that 
seems to be asking for trouble to me; history is full of but 
nobody will ever want to port this to one of _those_ that come 
back to bite people.


NetBSD already assumes that char is exactly 8 bits, and that 
integer types with exactly 16, 32, and 64 bits exist.  Adding more 
instances of the same assumptions doesn't seem like a big problem 
to me.  If there's ever a need to port to a machine where those 
assumptions do not hold, then we can worry about it at that time, 
but I susect that it will be possible to change to using things 
like int_least64_t (for a type with no less than 64 bits) instead 
of int64_t (for a type with exactly 64 bits).


--apb (Alan Barrett)


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Justin Cormack
On Sun, Nov 17, 2013 at 12:05 PM, Marc Balmer m...@msys.ch wrote:
 I came accross a small library for converting data to an from C structs
 for Lua, written by Roberto Ierusalimschy:

 http://www.inf.puc-rio.br/~roberto/struct/

 I plan to import it and to make it available to both lua(1) and lua(4)
 as follows:

 The source code will be imported into
 ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
 to compile on NetBSD.

 Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
 ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
 respective Makefiles etc.


I always found this library so un user friendly that I would rather
program in C. Not sure it was meant as much more than a proof of
concept. YMMV.

Justin


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Alexander Nasonov
Marc Balmer wrote:
 I came accross a small library for converting data to an from C structs
 for Lua, written by Roberto Ierusalimschy:
 
 http://www.inf.puc-rio.br/~roberto/struct/
 
 I plan to import it and to make it available to both lua(1) and lua(4)
 as follows:
 
 The source code will be imported into
 ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
 to compile on NetBSD.

Shouldn't it be mit/luastruct?

 Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
 ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
 respective Makefiles etc.

Are you going to make it kmod? It's an overkill, IMO. Not every Lua
module a kernel module. The framework should support this distinction.

PS I don't see devel/lua-struct. It'd be nice to have it in pkgsrc.

Alex


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Hubert Feyrer

On Sun, 17 Nov 2013, Marc Balmer wrote:

I plan to import it and to make it available to both lua(1) and lua(4)


I wonder if we really need to get all this into NetBSD,
instead of moving it to pkgsrc somehow.


 - Hubert


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 7:35 AM, Marc Balmer m...@msys.ch wrote:
 Am 17.11.13 04:36, schrieb Lourival Vieira Neto:
 On Sat, Nov 16, 2013 at 10:44 PM, Christos Zoulas chris...@zoulas.com 
 wrote:
 On Nov 16,  9:30pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
 -- Subject: Re: [patch] changing lua_Number to int64_t

 | On Sat, Nov 16, 2013 at 8:52 PM, Christos Zoulas chris...@astron.com 
 wrote:
 |  In article 52872b0c.5080...@msys.ch, Marc Balmer  m...@msys.ch 
 wrote:
 | Changing the number type to int64_t is certainly a good idea.  Two
 | questions, however:
 | 
 |  Why not intmax_t?
 |
 | My only argument is that int64_t has a well-defined width and, AFAIK,
 | intmax_t could vary. But I have no strong feelings about this. Do you
 | think intmax_t would be better?

 Bigger is better. And you can use %jd to print which is a big win.

 I agree that bigger is better and %jd is much better then % PRI/SCN.
 But don't you think that to know the exact width is even better?

 You can always use sizeof if the need to know the size arises.

I mean know it as a script programmer. I think that would be helpful
to know the exact  lua_Number width when you are writing a script.
AFAIK, you don't have sizeof functionality from Lua. So, IMHO,
lua_Number width should be fixed and documented.

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote:
 Am 17.11.13 04:49, schrieb Terry Moore:
 I believe that if you want the Lua scripts to be portable across NetBSD
 deployments, you should choose a well-known fixed width.

 I don't see this as very important.  Lua scripts will hardly depend on
 the size of an integer.

But they could. I think that the script programmers should know if the
numeric data type is enough for their usage (e.g., time diffs).

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 8:52 AM, Alexander Nasonov al...@yandex.ru wrote:
 Marc Balmer wrote:
 The basic issue here is that Lua has only _one_ numerical data type,
 which is an integral type in kernel, but a floating point type in userspace.

 Right, not everyone here knows this I guess. Thanks for making it clear.

Sorry, I thought that it was clear.

Regards,
-- 
Lourival Vieira Neto


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Marc Balmer
Am 17.11.13 13:21, schrieb Alexander Nasonov:
 Marc Balmer wrote:
 I came accross a small library for converting data to an from C structs
 for Lua, written by Roberto Ierusalimschy:

 http://www.inf.puc-rio.br/~roberto/struct/

 I plan to import it and to make it available to both lua(1) and lua(4)
 as follows:

 The source code will be imported into
 ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
 to compile on NetBSD.
 
 Shouldn't it be mit/luastruct?

Not imo.  Under external/...  I usually use the original name.

 
 Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
 ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
 respective Makefiles etc.
 
 Are you going to make it kmod? It's an overkill, IMO. Not every Lua
 module a kernel module. The framework should support this distinction.

It is to be used by lua(4), it _has_ to be a kmod.

I am right now not so sure if it will be this library or if take one of
my own, which has a saner interface imo.

I need something to decode/encode binary data, however, like pack/unpack
in python.

 
 PS I don't see devel/lua-struct. It'd be nice to have it in pkgsrc.




Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Marc Balmer
Am 17.11.13 13:20, schrieb Justin Cormack:
 On Sun, Nov 17, 2013 at 12:05 PM, Marc Balmer m...@msys.ch wrote:
 I came accross a small library for converting data to an from C structs
 for Lua, written by Roberto Ierusalimschy:

 http://www.inf.puc-rio.br/~roberto/struct/

 I plan to import it and to make it available to both lua(1) and lua(4)
 as follows:

 The source code will be imported into
 ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
 to compile on NetBSD.

 Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
 ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
 respective Makefiles etc.

 
 I always found this library so un user friendly that I would rather
 program in C. Not sure it was meant as much more than a proof of
 concept. YMMV.

Well possible that I uses a different library that I wrote for data
handling.  You know pack/unpack from python?




Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Marc Balmer
Am 17.11.13 13:32, schrieb Hubert Feyrer:
 On Sun, 17 Nov 2013, Marc Balmer wrote:
 I plan to import it and to make it available to both lua(1) and lua(4)
 
 I wonder if we really need to get all this into NetBSD,
 instead of moving it to pkgsrc somehow.

Yes, we need it to handle structured binary data.




Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 9:30 AM, Alexander Nasonov al...@yandex.ru wrote:
 Mouse wrote:
 Also, using an exact-width type assumes that the hardware/compiler in
 question _has_ such a type.

 It's possible that lua, NetBSD, or the combination of the two is
 willing to write off portability to machines where one or both of those
 potential portability issues becomes actual.  But that seems to be
 asking for trouble to me; history is full of but nobody will ever want
 to port this to one of _those_ that come back to bite people.

 I was perfectly fine with long long because it's long enough to
 represent all integers in range [-2^53-1, 2^53-1].

 As Marc pointed out, Lua has a single numeric type which is double
 by default. Many Lua libraries don't need FP and they use a subset of
 exactly representable integers (not all of them do range checks, though).
 Extending the range when porting from userspace to kernel will decrease
 the pain factor of porting.

I think that it is not the main point here, to decrease the pain
factor of porting. Porting Lua user-space libraries should be painful
enough independently of lua_Number type. IMHO, the main point here is
to define a lua_Number type adjusted to kernel needs. As, we have
already broke compatibility with user space libraries (for several
factors, not only the missing of floating point number), I think it
shouldn't matter much now.

Regards,
-- 
Lourival Vieira Neto


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 11:23 AM, Marc Balmer m...@msys.ch wrote:
 Am 17.11.13 13:32, schrieb Hubert Feyrer:
 On Sun, 17 Nov 2013, Marc Balmer wrote:
 I plan to import it and to make it available to both lua(1) and lua(4)

 I wonder if we really need to get all this into NetBSD,
 instead of moving it to pkgsrc somehow.

 Yes, we need it to handle structured binary data.

BTW, I'm developing bitwiser library with a similar purpose.It is not
ready already, but it could be helpful to handle binary data without
copying it from/to Lua string in the future.

LHF has lpack library [1], it could be an alternative to Roberto's struct.

[1] http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack

Regards,
-- 
Lourival Vieira Neto


Re: posix_fallocate

2013-11-17 Thread Christos Zoulas
On Nov 17,  1:15pm, k...@munnari.oz.au (Robert Elz) wrote:
-- Subject: Re: posix_fallocate

| ps: I have not examined the FreeBSD implementation - if they've done it the
| hard, safe, way, and worked out all the potential kinks, and if it doesn't
| depend too much upon other aspects of their I/O system implementation (like
| whatever they have to make softdeps work) then perhaps copying that might be
| feasible -- if the demand for this really exists, and it isn't being requested
| just because it is in the spec and NetBSD is lacking it.

From the cursory look at it, they just write.

christos


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 12:58 PM, Christos Zoulas chris...@zoulas.com wrote:
 On Nov 17,  1:36am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
 -- Subject: Re: [patch] changing lua_Number to int64_t

 |  Bigger is better. And you can use %jd to print which is a big win.
 |
 | I agree that bigger is better and %jd is much better then % PRI/SCN.
 | But don't you think that to know the exact width is even better?

 Why? You can always compute it at runtime if you need to.

Yes, but I think that it is not a common practice in Lua. AFAIK, Lua
has only math.huge [1] to tell what is the biggest number value
available (what isn't the same of having the lua_Number width). Thus,
we would need to provide an interface to return this information. I'd
prefer to have it defined in compile time independently of platform.
Anyway, as I stated before, I have no strong feelings about that.. if
everybody else prefer intmax_t, I have no objection.

[1] http://www.lua.org/manual/5.1/manual.html#pdf-math.huge

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Christos Zoulas
On Nov 17, 10:36am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
-- Subject: Re: [patch] changing lua_Number to int64_t

| I mean know it as a script programmer. I think that would be helpful
| to know the exact  lua_Number width when you are writing a script.
| AFAIK, you don't have sizeof functionality from Lua. So, IMHO,
| lua_Number width should be fixed and documented.

Lua should provide manifest constants for it  (like INTMAX_MAX).
Otherwise you'd be making assumptions

christos


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Christos Zoulas
On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
-- Subject: Re: [patch] changing lua_Number to int64_t

| On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote:
|  Am 17.11.13 04:49, schrieb Terry Moore:
|  I believe that if you want the Lua scripts to be portable across NetBSD
|  deployments, you should choose a well-known fixed width.
| 
|  I don't see this as very important.  Lua scripts will hardly depend on
|  the size of an integer.
| 
| But they could. I think that the script programmers should know if the
| numeric data type is enough for their usage (e.g., time diffs).

By making it the biggest type possible, you never need to be worried.

christos


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 2:02 PM, Christos Zoulas chris...@zoulas.com wrote:
 On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
 -- Subject: Re: [patch] changing lua_Number to int64_t

 | On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote:
 |  Am 17.11.13 04:49, schrieb Terry Moore:
 |  I believe that if you want the Lua scripts to be portable across NetBSD
 |  deployments, you should choose a well-known fixed width.
 | 
 |  I don't see this as very important.  Lua scripts will hardly depend on
 |  the size of an integer.
 |
 | But they could. I think that the script programmers should know if the
 | numeric data type is enough for their usage (e.g., time diffs).

 By making it the biggest type possible, you never need to be worried.

Right.. you just convinced me.. if no one opposes, I'll change that to
intmax_t and get rid of PRI/SCNd64 =).

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Justin Cormack
On Sun, Nov 17, 2013 at 4:52 PM, Lourival Vieira Neto
lourival.n...@gmail.com wrote:
 On Sun, Nov 17, 2013 at 2:02 PM, Christos Zoulas chris...@zoulas.com wrote:
 On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
 -- Subject: Re: [patch] changing lua_Number to int64_t

 | On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote:
 |  Am 17.11.13 04:49, schrieb Terry Moore:
 |  I believe that if you want the Lua scripts to be portable across NetBSD
 |  deployments, you should choose a well-known fixed width.
 | 
 |  I don't see this as very important.  Lua scripts will hardly depend on
 |  the size of an integer.
 |
 | But they could. I think that the script programmers should know if the
 | numeric data type is enough for their usage (e.g., time diffs).

 By making it the biggest type possible, you never need to be worried.

 Right.. you just convinced me.. if no one opposes, I'll change that to
 intmax_t and get rid of PRI/SCNd64 =).

1. Lua 5.3 will have 64 bit integer support as standard, which will
make interop and reuse between kernel and userspace code much easier,
iff we use int64_t

2. Code will have to handle use in the kernel of uint64_t which will
potentially behave differently with Lua number type being int64_t vs
if it was sat int128_t which might happen, such that its unlikely to
be tested properly. There is no existing system where intmax_t is not
int64_t so this breakage is untestable.

Justin


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 3:10 PM, Justin Cormack
jus...@specialbusservice.com wrote:
 On Sun, Nov 17, 2013 at 4:52 PM, Lourival Vieira Neto
 lourival.n...@gmail.com wrote:
 On Sun, Nov 17, 2013 at 2:02 PM, Christos Zoulas chris...@zoulas.com wrote:
 On Nov 17, 10:46am, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
 -- Subject: Re: [patch] changing lua_Number to int64_t

 | On Sun, Nov 17, 2013 at 7:37 AM, Marc Balmer m...@msys.ch wrote:
 |  Am 17.11.13 04:49, schrieb Terry Moore:
 |  I believe that if you want the Lua scripts to be portable across NetBSD
 |  deployments, you should choose a well-known fixed width.
 | 
 |  I don't see this as very important.  Lua scripts will hardly depend on
 |  the size of an integer.
 |
 | But they could. I think that the script programmers should know if the
 | numeric data type is enough for their usage (e.g., time diffs).

 By making it the biggest type possible, you never need to be worried.

 Right.. you just convinced me.. if no one opposes, I'll change that to
 intmax_t and get rid of PRI/SCNd64 =).

 1. Lua 5.3 will have 64 bit integer support as standard, which will
 make interop and reuse between kernel and userspace code much easier,
 iff we use int64_t

If they are using int64_t for integers, I think it is a good reason to us to
stick to int64_t.

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sat, Nov 16, 2013 at 9:25 PM, Lourival Vieira Neto
lourival.n...@gmail.com wrote:
 (...) I moved strtoimax.c to common/libc. Don't know if
 someone sees a problem on that.

BTW, is it OK? Could someone review this?

Regards,
-- 
Lourival Vieira Neto


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread David Holland
On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote:
  I plan to import it and to make it available to both lua(1) and lua(4)
  
  I wonder if we really need to get all this into NetBSD,
  instead of moving it to pkgsrc somehow.

This...

-- 
David A. Holland
dholl...@netbsd.org


Re: posix_fallocate

2013-11-17 Thread David Holland
On Sun, Nov 17, 2013 at 10:24:04AM +0100, Rhialto wrote:
   I think the chief question at this level is whether to support the
   keep the length flag for fallocate, fdiscard, both, or neither. The
  
  What keep the length flag? I don't see one at the indicated URL.

It's a linuxism in linux's native fallocate().

-- 
David A. Holland
dholl...@netbsd.org


Re: posix_fallocate

2013-11-17 Thread David Holland
On Sun, Nov 17, 2013 at 10:24:04AM +0100, Emmanuel Dreyfus wrote:
   I think the chief question at this level is whether to support the
   keep the length flag for fallocate, fdiscard, both, or neither. The
   linux fallocate uses this to allow allocating blocks past EOF, which
   strikes me as nuts; 
  
  Why is it bad?

What Mouse said, pretty much. Not only is it semantic nonsense, it
requires reworking fsck.

  I am interested to port software, and of course it uses
  FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE...

FALLOC_FL_PUNCH_HOLE is fdiscard(). I don't believe in deleting things
by making an allocate call, or perpetrating other people's bad design
to avoid patching a couple packages.

The question is whether FALLOC_FL_KEEP_SIZE makes sense; it does for
fdiscard, but I remain unconvinced in the case of fallocate.

-- 
David A. Holland
dholl...@netbsd.org


Re: posix_fallocate

2013-11-17 Thread David Holland
On Sun, Nov 17, 2013 at 10:33:43AM +0100, Emmanuel Dreyfus wrote:
   To answer both you and the Mouse - the difference is that a user process
   actually writing data consumes measurable resources, and thus is easy to
   find and kill.   When everything happens in the kernel, spotting which
   arrantly idle user process is making it happen is not at all easy.
  
  We could fork a kernel thread that would go to userspace to do the work
  with a write() loop, with appropriate credentials. Does it makes sense?

I do not think that makes the slightest sense. Also if you really want
FALLOC_FL_KEEP_SIZE, it won't be adequate.

-- 
David A. Holland
dholl...@netbsd.org


Re: RLIMIT_FSIZE and SIGXFSZ

2013-11-17 Thread David Holland
On Sat, Nov 16, 2013 at 12:40:22AM -0500, Mouse wrote:
  The documentation I have (which is consistent across 1.4T, 4.0.1, and
  5.2) says that [a] file I/O operation that would create a file larger
  that the process' soft limit will cause the write to fail and a signal
  SIGXFSZ to be generated.  I looked at the code (for 4.0.1, that being
  what's on the machine I care about at the moment) and this appears to
  be accurate.
  
  It seems to me it would be more useful to do something like what
  RLIMIT_CPU does, and generate SIGXFSZ for such operations, but fail
  them only when the size exceeds the hard limit.  As it stands, the hard
  limit is useful only as a value the soft limit can't be raised above.
  (More useful to me, at any rate.)

So I think the CPU limit signalling is treated as a special case
because it's pretty useless to deliver a signal whose handler you
can't execute because you're past the CPU limit. Otherwise (all other
limits) you get denied once you reach the soft limit, and I think
that's the way things are intended to work.

I think what you want is an additional signaling threshold that can be
manipulated separately from the limit.

-- 
David A. Holland
dholl...@netbsd.org


Re: posix_fallocate

2013-11-17 Thread Mouse
 [posix_fallocate]
 We could fork a kernel thread that would go to userspace to do the
 work with a write() loop, with appropriate credentials.  Does it
 makes sense?
 It would need to be a read/write loop, nothing says that there cannot
 already be blocks allocated in the space being fallocated, and their
 content should not change.

That's a reason to put it in the kernel, actually.  The kernel can tell
which ranges of a file have already had space allocated for them, so it
can be just writes.  (Well, for local filesystems.  Throw NFS or its
ilk into the mix and it gets more interesting.)

 But if implemented that way, why bother at all?  Why not just put the
 code in a user space libc posix_fallocate() function, and be done
 with it, it should not require any kernel support at all.

Well, I don't know what the point of having posix_fallocate at all
would be.  But the obvious answer to this is atomicity, especially in
the presence of other writers: the kernel is capable of making sure it
doesn't destroy someone else's write by mistake, which userland isn't
(in the userland implementation, there's a window between read and
write when someone else can write only to get overwritten).  If that
matters for the target application, that's a reason to prefer an
in-kernel implementation.

 On the other hand, posix_fallocate() could allocate pitabytes in a
 single invocation of the sys call, assuming that the filesystem had
 that much space available.   I haven't looked recently, but last time
 I did, preemptible sys calls still didn't mean that userland signals
 would be delivered in the middle of the operation of a single sys
 call, [...]

No, it doesn't mean that, strictly.  But any syscall that feels like it
can be signalable during any sleep involved in its operation; this was
true even before multiprocessor support.  There will be a loop involved
_somewhere_ in any posix_fallocate() implementation, and I can't
imagine that an implementation wouldn't sleep somewhere waiting for the
underlying filesystem operations.  Those sleeps can be made
interruptible by signals; at most it will complicate the exit path.

 nor does anything suggest that signals are supposed to interrupt the
 operation of posiz_fallocate() half way through - which suggests to
 me, that as designed, it should continue until it is finished once
 invoked, whatever anyone tries to do to the process that invoked it.

Actually, I see nothing in the description that prevents that.  Given
what it does, a half-completed posix_fallocate is indistinguishable, to
userland, from a never-started posix_fallocate, provided the former
hasn't got as far as affecting st_size.  It would be interesting to
deal with a posix_fallocate that raises st_size being interrupted after
it's written some but not all of the new space, especially in the
presence of another writer writing into the same area of the file, but
I feel certain those problems are solvable, even if it means pushing a
small fraction of the implementation down into the filesystem - and
maybe not even that; the existence of kqueue's EVFILT_VNODE NOTE_WRITE
means that most of the necessary machinery is already in place.

This is not to say that I support the idea of adding posix_fallocate;
I'm not sure what I think on that question.  But some of the arguments
kre has presented here do not, IMO, hold water.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B


Re: RLIMIT_FSIZE and SIGXFSZ

2013-11-17 Thread Mouse
 [...SIGXFSZ...]
 It seems to me it would be more useful to do something like what
 RLIMIT_CPU does, and generate SIGXFSZ for such operations, but fail
 them only when the size exceeds the hard limit.  [...]
 So I think the CPU limit signalling is treated as a special case
 because it's pretty useless to deliver a signal whose handler you
 can't execute because you're past the CPU limit.  Otherwise (all
 other limits) you get denied once you reach the soft limit, and I
 think that's the way things are intended to work.

RLIMIT_FSIZE is already special in another way, a way similar to
RLIMIT_CPU: it generates a signal.  You don't get a SIGXPROC when
trying to fork() when you're over RLIMIT_NPROC.  You don't get SIGXFD
when running into RLIMIT_NOFILE.  (Though perhaps you should.  Maybe
SIGRLIMIT, with the second handler arg saying which limit?)

If the write() simply failed showing EXFSIZE or some such, I might
agree with you.

 I think what you want is an additional signaling threshold that can
 be manipulated separately from the limit.

Sort of.

For my immediate purposes, I don't even _need_ to use RLIMIT_FSIZE at
all; it's simply a covenient way that's already there to hear about the
condition I care about.

I've made the changes I sketched for 4.0.1 and they do give the
behaviour I wanted/expected.  I'm going to add them to 1.4T and 5.2
(the other two versions I run) for my own use; I'll let you people hash
out whether NetBSD wants them.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 4:39 PM, David Holland dholland-t...@netbsd.org wrote:
 On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote:
   I plan to import it and to make it available to both lua(1) and lua(4)
  
   I wonder if we really need to get all this into NetBSD,
   instead of moving it to pkgsrc somehow.

 This...

I think that would be nice to have Lua kernel modules in pkgsrc, if possible.

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Christos Zoulas
On Nov 17,  3:36pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
-- Subject: Re: [patch] changing lua_Number to int64_t

|  1. Lua 5.3 will have 64 bit integer support as standard, which will
|  make interop and reuse between kernel and userspace code much easier,
|  iff we use int64_t
| 
| If they are using int64_t for integers, I think it is a good reason to us to
| stick to int64_t.

This is not relevant. The numeric type will still be double, so forget
about compatibility between kernel and userland. There is no need for
the interpreter to use a fixed width type, but rather it is convenient
to use the largest numeric type the machine can represent.

christos


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Justin Cormack
On Sun, Nov 17, 2013 at 7:56 PM, Christos Zoulas chris...@zoulas.com wrote:
 On Nov 17,  3:36pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
 -- Subject: Re: [patch] changing lua_Number to int64_t

 |  1. Lua 5.3 will have 64 bit integer support as standard, which will
 |  make interop and reuse between kernel and userspace code much easier,
 |  iff we use int64_t
 |
 | If they are using int64_t for integers, I think it is a good reason to us to
 | stick to int64_t.

 This is not relevant. The numeric type will still be double, so forget
 about compatibility between kernel and userland. There is no need for
 the interpreter to use a fixed width type, but rather it is convenient
 to use the largest numeric type the machine can represent.

There will be two numeric types as standard, int64_t and double. It
should be possible to compile the kernel Lua with only int64_t and no
double support I would think, so integer only userland programs would
be compatible which is a very useful feature. But the semantics of the
Lua integer type will be such that it wraps at 64 bit, unlike some
hypothetical larger type (that doesn't yet exist and which the kernel
doesn't yet use).

Justin


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 6:13 PM, Justin Cormack
jus...@specialbusservice.com wrote:
 On Sun, Nov 17, 2013 at 7:56 PM, Christos Zoulas chris...@zoulas.com wrote:
 On Nov 17,  3:36pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
 -- Subject: Re: [patch] changing lua_Number to int64_t

 |  1. Lua 5.3 will have 64 bit integer support as standard, which will
 |  make interop and reuse between kernel and userspace code much easier,
 |  iff we use int64_t
 |
 | If they are using int64_t for integers, I think it is a good reason to us 
 to
 | stick to int64_t.

 This is not relevant. The numeric type will still be double, so forget
 about compatibility between kernel and userland. There is no need for
 the interpreter to use a fixed width type, but rather it is convenient
 to use the largest numeric type the machine can represent.

 There will be two numeric types as standard, int64_t and double. It
 should be possible to compile the kernel Lua with only int64_t and no
 double support I would think, so integer only userland programs would
 be compatible which is a very useful feature. But the semantics of the
 Lua integer type will be such that it wraps at 64 bit, unlike some
 hypothetical larger type (that doesn't yet exist and which the kernel
 doesn't yet use).

Well, I don't think I fully understood that; mainly because I'm not
aware about Lua 5.3. It will provide two number types for the scripts?
Or you are just talking about lua_Integer type on the C-side. Lua 5.1
already has a lua_Integer type that is defined as ptrdiff_t.

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Justin Cormack
On Sun, Nov 17, 2013 at 8:39 PM, Lourival Vieira Neto
lourival.n...@gmail.com wrote:
 Well, I don't think I fully understood that; mainly because I'm not
 aware about Lua 5.3. It will provide two number types for the scripts?
 Or you are just talking about lua_Integer type on the C-side. Lua 5.1
 already has a lua_Integer type that is defined as ptrdiff_t.

Yes that is correct, so 2 will be integer and 5.3 float, operations
will be defined in terms of how they convert, so there will be int and
float division. The draft manual is here
http://www.lua.org/work/doc/manual.html (see 3.4.1). This will not
happen for a while, but it will make it much easier in future for
interfaces like the kernel that need 64 bit int support, which is why
it is being implemented. So not being compatible with this seems a
mistake.

Justin


RE: [patch] changing lua_Number to int64_t

2013-11-17 Thread Terry Moore
 From: Marc Balmer [mailto:m...@msys.ch]
 
  It's not *much* less safe than compiling and executing a string in the
  kernel. The only additional attack surfaces are that you can write
things
  that the compiler wouldn't write. This can (1) cause a crash at load
time,
  or it can (2) cause surprising behavior later.
 
 The problem is that malicious bytecode in 5.1 is possible.  That is why
 there is a guad against loading bytecode.

Malicious in what sense? Is this a type-1 problem or a type-2 problem? Or
something else that I've not considered?

--Terry



Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 6:45 PM, Justin Cormack
jus...@specialbusservice.com wrote:
 On Sun, Nov 17, 2013 at 8:39 PM, Lourival Vieira Neto
 lourival.n...@gmail.com wrote:
 Well, I don't think I fully understood that; mainly because I'm not
 aware about Lua 5.3. It will provide two number types for the scripts?
 Or you are just talking about lua_Integer type on the C-side. Lua 5.1
 already has a lua_Integer type that is defined as ptrdiff_t.

 Yes that is correct, so 2 will be integer and 5.3 float, operations
 will be defined in terms of how they convert, so there will be int and
 float division. The draft manual is here
 http://www.lua.org/work/doc/manual.html (see 3.4.1). This will not
 happen for a while, but it will make it much easier in future for
 interfaces like the kernel that need 64 bit int support, which is why
 it is being implemented. So not being compatible with this seems a
 mistake.

Humm.. I think that ยง2.1 brings a good argument: Standard Lua uses
64-bit integers and double-precision floats, (...). I think that
would not hurt to stick to the future standard; once 64 bit is good
enough for kernel purposes.

Regards,
-- 
Lourival Vieira Neto


Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Christos Zoulas
On Nov 17,  7:14pm, lourival.n...@gmail.com (Lourival Vieira Neto) wrote:
-- Subject: Re: [patch] changing lua_Number to int64_t

| Humm.. I think that =A72.1 brings a good argument: Standard Lua uses
| 64-bit integers and double-precision floats, (...). I think that
| would not hurt to stick to the future standard; once 64 bit is good
| enough for kernel purposes.

Ok,

christos


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Marc Balmer
Am 17.11.13 14:43, schrieb Lourival Vieira Neto:
 On Sun, Nov 17, 2013 at 11:23 AM, Marc Balmer m...@msys.ch wrote:
 Am 17.11.13 13:32, schrieb Hubert Feyrer:
 On Sun, 17 Nov 2013, Marc Balmer wrote:
 I plan to import it and to make it available to both lua(1) and lua(4)

 I wonder if we really need to get all this into NetBSD,
 instead of moving it to pkgsrc somehow.

 Yes, we need it to handle structured binary data.
 
 BTW, I'm developing bitwiser library with a similar purpose.It is not
 ready already, but it could be helpful to handle binary data without
 copying it from/to Lua string in the future.
 
 LHF has lpack library [1], it could be an alternative to Roberto's struct.
 
 [1] http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack

So before I commit anything (if at all), we will coordinate this effort
and maybe take the best of all libraries.  So that we have only one
module dealing with data.




Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Marc Balmer
Am 17.11.13 20:40, schrieb Lourival Vieira Neto:
 On Sun, Nov 17, 2013 at 4:39 PM, David Holland dholland-t...@netbsd.org 
 wrote:
 On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote:
   I plan to import it and to make it available to both lua(1) and lua(4)
  
   I wonder if we really need to get all this into NetBSD,
   instead of moving it to pkgsrc somehow.

 This...
 
 I think that would be nice to have Lua kernel modules in pkgsrc, if possible.

No, I don't think so.  They interact to much with the system, they need
to be part of the system.


Re: posix_fallocate

2013-11-17 Thread Robert Elz
Date:Sun, 17 Nov 2013 14:12:16 -0500 (EST)
From:Mouse mo...@rodents-montreal.org
Message-ID:  201311171912.oaa17...@chip.rodents-montreal.org

  | That's a reason to put it in the kernel, actually.  The kernel can tell
  | which ranges of a file have already had space allocated for them, so it
  | can be just writes.

Sure, but that's just an optimisation, and surely only matters if this
is done enough that the difference is actually of significance.  Do you
really believe it is, or ever will be?

  | Well, I don't know what the point of having posix_fallocate at all
  | would be.

Agree there, it seems mostly useless.

  | But the obvious answer to this is atomicity, especially in
  | the presence of other writers:

Really?  You're imagining multiple writers, writing to the same file,
without any co-ordination (like locking, or whatever other way works) and
you're actually worried about unpredictable results???   Really?

But beyond that, since you plan on allowing signals to interrupt the
operation part way through, what atomicity are you really getting anyway?

  | No, it doesn't mean that, strictly.  But any syscall that feels like it
  | can be signalable during any sleep involved in its operation;

Sure, though the internal implementation of the sys call has to explicitly
make that happen - and to be useful, the sys call interface really has to
be designed to support it.   This one isn't (and as specified, cannot be.)

That is, if you imagine the sys call in question being SIGKILL (as in the
scenario I postulated initially) there's no problem - if the code for the
sys call wants to allow that to work before it completes, it easily can.

But instead imagine it is SIGALRM - now there's a problem, since there's
no way in the interface to report how much of the work was done, the only
thing that can be done if EINTR happens, is repeat the sys call.  For
a periodic sys call like SIGALRM (or any of the other timer signals) chances
are that a signal will arrive during the sys call every time, resulting
in an infinite loop of fallocate/signal/fallocate/signal...

This thing is just poorly designed.  Let's just ignore it (by all means,
implement the hole making part from the linux interface if desired, but
the allocation side just isn't needed).

  | Given what it does, a half-completed posix_fallocate is indistinguishable,
  | to userland, from a never-started posix_fallocate, provided the former
  | hasn't got as far as affecting st_size.

Of course, I'm only really interested in cases where the size can't help
being affected, as it starts at 0 - cases where actual holes in the middle
of a file are being filled in I see as so unlikely in practice that they're
totally irrelevant (like unless the app has just made the file by seeking
forward and writing a byte, how does it ever know whether or not there are
holes to fill?   And why would it do it that way, and follow by fallocate() ?
If fallocate() exists, surely it would just use that to make the file?)

But as above, that it is indistinguishable is the problem.

kre

ps: Note that I see that the linux way of handling fallocate isn't to
write blocks of 0's, but to allocate uninitialised blocks, and mark them
uninitialised - I assume the way that works, is that if an app reads one
of those blocks, it is just given 0's - and if it writes (the expected
operation) whatever is there (0's or junk) just gets overwritten.  That
way they make fallocate() really fast (just assigns block numbers to the file)
but it requires that the un-init flag (wheverever, and however, they keep that)
is 100% reliable.   Nothing is really that reliable...   FFS doesn't have
any mechanism to do that, so actually writing 0's would be the only way,
and given that, fallocate() looks to be a total waste of time - again,
given that it is an optional sys call, that no-one is required to implement,
and so which no-one can assume actually exists.



Re: [patch] changing lua_Number to int64_t

2013-11-17 Thread Marc Balmer
Am 17.11.13 22:03, schrieb Terry Moore:
 From: Marc Balmer [mailto:m...@msys.ch]

 It's not *much* less safe than compiling and executing a string in the
 kernel. The only additional attack surfaces are that you can write
 things
 that the compiler wouldn't write. This can (1) cause a crash at load
 time,
 or it can (2) cause surprising behavior later.

 The problem is that malicious bytecode in 5.1 is possible.  That is why
 there is a guad against loading bytecode.
 
 Malicious in what sense? Is this a type-1 problem or a type-2 problem? Or
 something else that I've not considered?

See Peter Cawleys talk he gave during LWS 2011 in Frick.

Mitigating the danger of malicious bytecode

Peter Cawley (University of Oxford)

In Lua 5.1 and 5.2, the family of load functions permit the loading of
precompiled bytecode in addition to plain source code. We shall look at
some relevant details of the bytecode, and see how these details can be
maliciously abused to escape from a Lua sandbox (in both 5.1 and 5.2).
With the dangers presented, we shall then move to look at two ways of
mitigating them. Firstly, one can wrap the load functions as to reject
bytecode before it is loaded. Secondly, and more interestingly, one can
attempt to algorithmically decide whether a given piece of bytecode is
malicious or not. We shall look at two such algorithms for validating
bytecode: LBCV, which tries to reject as little as possible, and my
current experimental verifier, which tries to accept as little as
possible. 


http://www.lua.org/wshop11/Cawley.pdf




Re: posix_fallocate

2013-11-17 Thread Rhialto
On Mon 18 Nov 2013 at 05:11:35 +0700, Robert Elz wrote:
 how does it ever know whether or not there are
 holes to fill?   And why would it do it that way, and follow by fallocate() ?

Wasn't there some (proposed? actual?) interface to find holes in files?

-Olaf.
-- 
___ Olaf 'Rhialto' Seibert  -- The Doctor: No, 'eureka' is Greek for
\X/ rhialto/at/xs4all.nl-- 'this bath is too hot.'



pgpJE4ppPn_az.pgp
Description: PGP signature


RE: [patch] safety of Lua byte code

2013-11-17 Thread Terry Moore
 Am 17.11.13 22:03, schrieb Terry Moore:
  From: Marc Balmer [mailto:m...@msys.ch]
 
  It's not *much* less safe than compiling and executing a string in the
  kernel. The only additional attack surfaces are that you can write
  things
  that the compiler wouldn't write. This can (1) cause a crash at load
  time,
  or it can (2) cause surprising behavior later.
 
  The problem is that malicious bytecode in 5.1 is possible.  That is why
  there is a guad against loading bytecode.
 
  Malicious in what sense? Is this a type-1 problem or a type-2 problem?
Or
  something else that I've not considered?
 
 See Peter Cawleys talk he gave during LWS 2011 in Frick.
 
 Mitigating the danger of malicious bytecode
 

So we're discussing what I called a type-2 problem, escaping the sandbox.

Really, however, if you load any code of any complexity into any VM, you are
at risk -- no matter how it the code is presented (byte code or source). If
you are loading code you don't know into a Turning-complete VM that's
running in the kernel, you are making a very strong assumption (and one that
seems, based on experience in the software industry to be unwarranted) about
the correctness of the compiler, run-time system, etc. I am not referring to
Lua in particular; the code is pretty good. I'm referring instead to the
general complexity of the VM environment. 

I personally would never run untrusted code of any kind in the kernel,
whether provided in source form or binary form, whether sandboxed or not,
unless the VM is very limited (along the lines of the BPF VM).

This isn't to say that Lua is not useful in the kernel. It is, however, to
say that I'd be very reluctant to let an adversary have carte blanche to
inject Lua code, in source or binary form, into the kernel. (This goes for
*any* language, C, Python, binary. Injecting Lua code into your kernel is
like doing an insmod -- you had better be asserting that you trust the code
*completely*. In fact, the use case of a serial protocol coded in Lua is
exactly an insmod-like application.) 

Nobody would seriously claim, I think, that one can algorithmically
determine whether a given fragment of Lua *source* code is malicious. Yes,
Lua byte codes can be malicious in additional ways, but I think worrying
about this misleads us about the nature of the problem.

If you're worrying about the safety of the compiled code in this sense, then
you should really be worrying about the safety of having the compiler in the
kernel at all.  I think that the paper you refer to is more motivated by use
cases such as running Lua in a web browser a-la Java, or in a game WoW,
where the code is injected into a VM without any real authentication.  

Best regards,
--Terry




Re: posix_fallocate

2013-11-17 Thread Robert Elz
Date:Sun, 17 Nov 2013 23:49:50 +0100
From:Rhialto rhia...@falu.nl
Message-ID:  20131117224950.gh23...@falu.nl

  | Wasn't there some (proposed? actual?) interface to find holes in files?

I was told off-list about SEEK_HOLE and SEEK_DATA that allow (if the
filesys supports them) apps to jump to the next hole, then end of hole,
and that way work out where the holes are.

According to the linux man page, they exist there, and FreeBSD DragonFly and
Solaris - and might come to posix.

Personally, I think the underlying structure of files should not be made
visible to apps at all - they should just see a byte stream (perhaps with
an advisory useful block size to write in).

kre



Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread Lourival Vieira Neto
On Sun, Nov 17, 2013 at 7:57 PM, Marc Balmer m...@msys.ch wrote:
 Am 17.11.13 14:43, schrieb Lourival Vieira Neto:
 On Sun, Nov 17, 2013 at 11:23 AM, Marc Balmer m...@msys.ch wrote:
 Am 17.11.13 13:32, schrieb Hubert Feyrer:
 On Sun, 17 Nov 2013, Marc Balmer wrote:
 I plan to import it and to make it available to both lua(1) and lua(4)

 I wonder if we really need to get all this into NetBSD,
 instead of moving it to pkgsrc somehow.

 Yes, we need it to handle structured binary data.

 BTW, I'm developing bitwiser library with a similar purpose.It is not
 ready already, but it could be helpful to handle binary data without
 copying it from/to Lua string in the future.

 LHF has lpack library [1], it could be an alternative to Roberto's struct.

 [1] http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack

 So before I commit anything (if at all), we will coordinate this effort
 and maybe take the best of all libraries.  So that we have only one
 module dealing with data.

OK..

Regards,
-- 
Lourival Vieira Neto


Re: [patch] safety of Lua byte code

2013-11-17 Thread Mouse
 Nobody would seriously claim, I think, that one can algorithmically
 determine whether a given fragment of Lua *source* code is malicious.

It is impossible, because the _very same code_ may be fairly called
malicious when presented for execution by an attacker but not when
presented for execution by a suitably authorized person.  Therefore,
information beyond the code itself is necessary to determine malice.

Note, too, that there is nothing Lua-specific in the above argument.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B


Re: A Library for Converting Data to and from C Structs for Lua

2013-11-17 Thread John Nemeth
On Nov 17, 11:02pm, Marc Balmer wrote:
} Am 17.11.13 20:40, schrieb Lourival Vieira Neto:
}  On Sun, Nov 17, 2013 at 4:39 PM, David Holland dholland-t...@netbsd.org 
wrote:
}  On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote:
}I plan to import it and to make it available to both lua(1) and lua(4)
}   
}I wonder if we really need to get all this into NetBSD,
}instead of moving it to pkgsrc somehow.
} 
}  This...
}  
}  I think that would be nice to have Lua kernel modules in pkgsrc, if 
possible.
} 
} No, I don't think so.  They interact to much with the system, they need
} to be part of the system.

 Uh, no.  The whole idea behind modules clearly means being
able to use third party code.  We should be able to have modules
in pkgsrc.  There are no modules in pkgsrc yet, but that's just a
matter of figuring out the best way to do it.  There is no reason
why all modules must be included with the system.

}-- End of excerpt from Marc Balmer


Re: posix_fallocate

2013-11-17 Thread Mouse
 That's a reason to put it in the kernel, actually.  The kernel can
 tell which ranges of a file have already had space allocated for
 them, so it can be just writes.
 Sure, but that's just an optimisation, and surely only matters if
 this is done enough that the difference is actually of significance.
 Do you really believe it is, or ever will be?

Not just an optimization; it also affects correctness - see below.

 But the obvious answer to this is atomicity, especially in the
 presence of other writers:
 Really?  You're imagining multiple writers, writing to the same file,
 without any co-ordination (like locking, or whatever other way works)
 and you're actually worried about unpredictable results???  Really?

Actually, thinking about it more, atomicity is the wrong word.

The correct thing to worry about here is that, as I read the pointed-to
webpage, posix_fallocate() is defined to do nothing to already-present
data.  But, when racing with another writer, a non-kernel
implementation will always have conditions under which it can destroy
data written by some other process.

 No, it doesn't mean that, strictly.  But any syscall that feels like
 it can be signalable during any sleep involved in its operation;
 Sure, though the internal implementation of the sys call has to
 explicitly make that happen

Well, sure.

 - and to be useful, the sys call interface really has to be designed
 to support it.   This one isn't (and as specified, cannot be.)

Huh?  I don't see it that way.

 That is, if you imagine the [signal] in question being SIGKILL (as in
 the scenario I postulated initially) there's no problem - if the code
 for the sys call wants to allow that to work before it completes, it
 easily can.

 But instead imagine it is SIGALRM - now there's a problem, since
 there's no way in the interface to report how much of the work was
 done, the only thing that can be done if EINTR happens, is repeat the
 sys call.

Right.  That's a reason to put it in the kernel, so it doesn't need to
redo past work.  (This is not a complete fix, because it applies only
to calls which don't increase st_size.)

I note that the documentation webpage lists EINTR.

 For a periodic [signal] like SIGALRM (or any of the other timer
 signals) chances are that a signal will arrive during the sys call
 every time, resulting in an infinite loop of
 fallocate/signal/fallocate/signal...

Sure.  There are lots of ways programmers can write code which ends up
in livelock.  I don't see how this one deserves any more special
treatment than the others.

 [...] - cases where actual holes in the middle of a file are being
 filled in I see as so unlikely in practice that they're totally
 irrelevant (like unless the app has just made the file by seeking
 forward and writing a byte, how does it ever know whether or not
 there are holes to fill?  And why would it do it that way, and follow
 by fallocate()?  If fallocate() exists, surely it would just use that
 to make the file?)

I see it as being intended for programs doing things like databases:
they may want to allocate disk space when they know they'll want it but
it's still easy to back out of the operation if it fails.  Once the
space is allocated, then they can carry on knowing they won't run into
a full disk partway through, later, when it's much harder to deal with.

In this paradigm, the application doesn't know whether there used to be
a hole there and doesn't care; the important thing is that, after the
call, there isn't.

I'm not sure why it's better than (read-and-)write for such
application, but that's the use case it feels designed for to me.

(Also, seeking and writing is not the only way to create a large file;
truncate/ftruncate can extend files on at least some systems.)

 ps: Note that I see that the linux way of handling fallocate isn't to
 write blocks of 0's, but to allocate uninitialised blocks, and mark
 them uninitialised [...]  That way they make fallocate() really fast
 (just assigns block numbers to the file) but it requires that the
 un-init flag (wheverever, and however, they keep that) is 100%
 reliable.  Nothing is really that reliable...

Not 100% reliable, but at least as reliable as the rest of the
filesystem.  FFS assumes that di_db[] in inodes, and the block
allocation bitmaps, won't change behind its back, too; I don't see why
this would be any different, really.

I'm not sure whether I'd be willing to pay one more bit per frag in
order to (greatly) speed up reads of allocated but unwritten blocks; my
own guess would be that such things are rare enough that optimizing
them doesn't really matter - though, of course, I don't often find uses
for things I don't have.  Perhaps Linux has found a real use for such
things.  The major use I can think of for them are things like NFS
swapfiles, where you want to allocate the whole file but have no need
to write it.  My own livebackup is in a similar situation; it could
benefit from an allocated but unwritten state 

re: posix_fallocate

2013-11-17 Thread matthew green

 Personally, I think the underlying structure of files should not be made
 visible to apps at all - they should just see a byte stream (perhaps with
 an advisory useful block size to write in).

i would buy this argument if mmap()ing a large sparse file
and filling it up randomly (but with relatively large chunks
at a time) did not lead to severely fragmented files that
can take 10x to read, vs one written with plain sequential
write() calls.  because of this, some workaround is
necessary.  it is very disappointing to see an average of
120 iops of 64KB each (and only because i formatted my
FS with 64kb blocks/frags!.) whe sequentially reading a
file created by mmap().

posix_fallocate() is answering a real problem.  the work
around today is to write the file, which doubles the IO
traffic, and i am not sure we can do better with FFS, due
to the issues you've mentioned, but there are many other
filesystems in existence that do allow block allocation
without exposing prior data or initialisation.

given the current issues, i'd be happy with a userspace
implementation.


.mrg.