Re: Lua in-kernel (lbuf library)
Hi Christoph,
Firstly, thanks for your comments. I really appreciated that =).
BTW, I renamed lbuf to Lua bitwiser and removed support for unamed
array access (buf:mask(alignment [, offset, length]) and buf:mask{
length_pos1, length_pos2, ... }). Instead, I introduced a new way to
provide array access (see below).
On Sun, Oct 20, 2013 at 8:05 PM, Christoph Badura b...@bsd.de wrote:
On Tue, Oct 15, 2013 at 06:01:29PM -0300, Lourival Vieira Neto wrote:
Also, having to switch mentally between zero-based arrays in the kernel C
code and 1-based arrays in the Lua code make my head ache.
It's something that doesn't bug me so much.. But, if necessary it
could be changed to 0-based in this userdata.
When you create your own data structures, I guess it is a wash. You have
to adjust +/-1 in infrequent circumstances in either scenario.
But in this case you are creating a special purpose language that operates
in universe of zero-based array. And that's not only the kernel code.
Every Internet protocol specification that I remember is using zero-based
indexing. For someone dealing with both sides (the world and your lua
library), it makes the difference between constantly having to be alert
to remember to do the offset adjustment. That is a lot more mental work
for anyone working with this library.
If you use 1-based indices talking to protocol people will be funny too:
``Anyone know why the flags in byte 6 of this packet are funny?''
``Sure, that's most likely because the flags are in byte 5.''
I think it is worth thinking hard about this.
Yes.. I'll keep think on this. By now, I'm using 1-based indices to
Lua array-access on buffers and 0-based offsets to bitmask
definitions.
I what is buffer and how does it relate to mbufs?
Note, non-contiguous buffer still an open problem in lbuf. I don't
know if should use a ptrdiff_t to pass the distance to 'next' field,
a 'next()' to return 'next' field or something else.
You seem to be talking about implementation. I was talking about the
interface of the library.
Yes, sorry. In that stage, sometimes, it is a little difficult to
separate these things.
However, you could create a lbuf from a mbuf header as follows:
lbuf_new(L, mbuf-m_data, mbuf-m_len, NULL, true);
I don't think that is a good way. You say you want to inspect packet
data in the kernel. Well, the packet's data can be spread over a
chain of mbufs. Also, mbufs may have internal or externel storage.
You don't want to deal with that as the user of your library.
As a user, I want an interface like this:
lbuf_from_mbuf(L, mbuf, NULL, true);
I'm thinking in providing an interface like this in an adapter layer.
Thus, we could use Lua bitwiser library in other areas. I was just
giving an example of how to use it with the current implementation
(again, sorry for don't separate interface discussion from
implementation).
Yes, mea culpa =(. I wasn't clear about that. 'net' flag was the way I
found to 'record' the buffer endianness. What means, true if the
buffer uses BE and false if it uses HE. It has the same semantics of
hton* and ntoh* functions. Don't know if it is better to pass the
endianness itself as a flag (e.g., enum { BIG_ENDIAN, LITTLE_ENDIAN,
HOST_ENDIAN }). What do you think?
For me the the most convenient interface would be if I didn't have to
mention the host byteorder. Just record what byteorder the buffer is in,
and convert when appropriate. Alan made a good point. It maybe be
convenient and/or necessary to specify a different byteorder in a mask.
I'm working on it; thinking in the following form:
bitwiser.mask{ field = { offset, length, sign, endian, step }}, where
sign is a boolean, endian is a string like 'host', 'h', 'big', 'b',
'little', 'l', 'net' or 'n', and step is a number between [1, 64].
And (to allow omitting or non-ordering of parameters):
bitwiser.mask{ field =
{ __offset = offset, __length = length, __sign = sign, __endian =
endian, __step = step }}
defaults are __sign = true, __endian = 'host' and __step = undef. If
step is present or length is omitted, the lib assumes that it is a
segment field, what means that it should return a bitwiser.buffer
userdatum if accessed, which can be accessed like an array (using
step, if it is defined, or else the step of the original buffer to
determine the length of each field). It also could be masked to use
field access. For example:
m = bitwiser.mask{
type = { 0, 4 },
flags = { 4, 4, __step = 1 },
payload = { 8 }
}
b = bitwiser.buffer{ 0xff, 0, 0xff, 0 } -- new buffers have step = 8, by default
b[1] -- 0xff
b:mask(m)
b.flags[1] = false -- unsets bit-4 (0-based)
b.flags[4] -- returns bit-7 (0-based), 1 in this case
b.payload:mask{ padding = { 0, 8 }, data = { 8, __step = 16 } }
b.payload.data[1] -- returns 2 bytes from bit-16 (0-based) of the
original buffer,
-- 0x00ff or 0xff00 depending on platform
endianess, in this case
Re: Lua in-kernel (lbuf library)
Hi Artem, On Wed, Oct 23, 2013 at 5:10 PM, Artem Falcon lo...@gero.in wrote: On Tue, Oct 15, 2013 at 06:01:29PM -0300, Lourival Vieira Neto wrote: Also, having to switch mentally between zero-based arrays in the kernel C code and 1-based arrays in the Lua code make my head ache. Above this well-discussed inconvenience there is a thing which may hurt more. It's the Lua's stack-based C API and all the stack composition pottering it imposes on you. This may be customary for those having an experience with concatenative languages, but it'll be a source of errors for the others. See [1]'s Stack based API is harder for more on it. [1] http://julien.danjou.info/blog/2011/why-not-lua -- dukzcry Here is another nice discussion about C APIs of scripting languages: www.inf.puc-rio.br/~roberto/docs/jucs-c-apis.pdf Regards, -- Lourival Vieira Neto
Re: Lua in-kernel (lbuf library)
Am 21.10.13 00:05, schrieb Christoph Badura: On Tue, Oct 15, 2013 at 06:01:29PM -0300, Lourival Vieira Neto wrote: Also, having to switch mentally between zero-based arrays in the kernel C code and 1-based arrays in the Lua code make my head ache. It's something that doesn't bug me so much.. But, if necessary it could be changed to 0-based in this userdata. When you create your own data structures, I guess it is a wash. You have to adjust +/-1 in infrequent circumstances in either scenario. But in this case you are creating a special purpose language that operates in universe of zero-based array. And that's not only the kernel code. Every Internet protocol specification that I remember is using zero-based indexing. For someone dealing with both sides (the world and your lua library), it makes the difference between constantly having to be alert to remember to do the offset adjustment. That is a lot more mental work for anyone working with this library. If you use 1-based indices talking to protocol people will be funny too: ``Anyone know why the flags in byte 6 of this packet are funny?'' ``Sure, that's most likely because the flags are in byte 5.'' I think it is worth thinking hard about this. From a cursory reading of the Wireshark Lua API, it seems to me they are using 0-based indices too. It probably depends whether you are access a Lua table (where you'd expect 1-based) or if you are accessin an in-kernel datastructure that is not strictly a Lua table. If a Lua function is to mimick an existing C function, it might be better indeed to use 0-based access, to not confuse developers who are probably familiar with the corresponding C function. As this is mostly a matter of taste/source of confusion, it should be documented. A man page for a Lua function that offers the same or similar functionality as a C function should state whether it's 0-based or 1-based, imo. [...]
Re: Lua in-kernel (lbuf library)
Am 19.10.13 00:51, schrieb John Nemeth: } kern.lua.autoload is a safety barrier. One may wish not allow any lua kernel } script to load any given lua kernel module. The lua(4) implementers can certainly do this if they want. However, module_autoload() won't be looking at this flag and will continue to refuse to autoload any module that has the noautoload flag set. Also, there is the kern.module.autoload sysctl that can prevent any module from autoloading. And I think that is a good thing. If kern.module.autoload is used to prevent module autoloading, then lua(4) must not be a way to circumvent this. For clarification: The kern.lua.autoload sysctl merely controls whether lua(4) *tries* to autload a module when it encounters the require statement. If it is set to false, it will not try it.
Re: Lua in-kernel (lbuf library)
On Tue, Oct 15, 2013 at 06:01:29PM -0300, Lourival Vieira Neto wrote: Also, having to switch mentally between zero-based arrays in the kernel C code and 1-based arrays in the Lua code make my head ache. Above this well-discussed inconvenience there is a thing which may hurt more. It's the Lua's stack-based C API and all the stack composition pottering it imposes on you. This may be customary for those having an experience with concatenative languages, but it'll be a source of errors for the others. See [1]'s Stack based API is harder for more on it. [1] http://julien.danjou.info/blog/2011/why-not-lua -- dukzcry
Re: Lua in-kernel (lbuf library)
On Tue, Oct 15, 2013 at 06:01:29PM -0300, Lourival Vieira Neto wrote:
Also, having to switch mentally between zero-based arrays in the kernel C
code and 1-based arrays in the Lua code make my head ache.
It's something that doesn't bug me so much.. But, if necessary it
could be changed to 0-based in this userdata.
When you create your own data structures, I guess it is a wash. You have
to adjust +/-1 in infrequent circumstances in either scenario.
But in this case you are creating a special purpose language that operates
in universe of zero-based array. And that's not only the kernel code.
Every Internet protocol specification that I remember is using zero-based
indexing. For someone dealing with both sides (the world and your lua
library), it makes the difference between constantly having to be alert
to remember to do the offset adjustment. That is a lot more mental work
for anyone working with this library.
If you use 1-based indices talking to protocol people will be funny too:
``Anyone know why the flags in byte 6 of this packet are funny?''
``Sure, that's most likely because the flags are in byte 5.''
I think it is worth thinking hard about this.
From a cursory reading of the Wireshark Lua API, it seems to me they are
using 0-based indices too.
I what is buffer and how does it relate to mbufs?
Note, non-contiguous buffer still an open problem in lbuf. I don't
know if should use a ptrdiff_t to pass the distance to 'next' field,
a 'next()' to return 'next' field or something else.
You seem to be talking about implementation. I was talking about the
interface of the library.
However, you could create a lbuf from a mbuf header as follows:
lbuf_new(L, mbuf-m_data, mbuf-m_len, NULL, true);
I don't think that is a good way. You say you want to inspect packet
data in the kernel. Well, the packet's data can be spread over a
chain of mbufs. Also, mbufs may have internal or externel storage.
You don't want to deal with that as the user of your library.
As a user, I want an interface like this:
lbuf_from_mbuf(L, mbuf, NULL, true);
That would make the contents of the mbuf chain starting at mbuf
available in an array-of-bytes like fashion. Length isn't needed as
it is computed from the mbuf chain.
Yes, mea culpa =(. I wasn't clear about that. 'net' flag was the way I
found to 'record' the buffer endianness. What means, true if the
buffer uses BE and false if it uses HE. It has the same semantics of
hton* and ntoh* functions. Don't know if it is better to pass the
endianness itself as a flag (e.g., enum { BIG_ENDIAN, LITTLE_ENDIAN,
HOST_ENDIAN }). What do you think?
For me the the most convenient interface would be if I didn't have to
mention the host byteorder. Just record what byteorder the buffer is in,
and convert when appropriate. Alan made a good point. It maybe be
convenient and/or necessary to specify a different byteorder in a mask.
buf:rawget(0, 9) ~ if net flag is *true*: takes 16 bits from
You know, I think rawget is badly named. Raw implies unmodified.
And byteswapping is a form of modification. Maybe you can find a better
name.
buf:mask(3)
buf[3] ~ accesses 3 bits from bit-6 position
What does that mean? Does it return the top-most 2 bits from the first
byte plus the least significant bit fom the second byte of the buffer?
It means the least-most 2 bits from the first byte and the LSB from the
second.
I don't know what least-most means. But since you don't seem to
agree with my questions I assume you intend that to be the opposite of
top-most then your statement doesn't make sense to me. Translated
into normal terms it would return bits 0 and 1 of the first byte and bit
0 of the second one. They are not even contigous.
What is 'length' for?
Offset and length could be used to impose boundaries to the mask.
Don't tell it to me. Write it into the documentation. :-)
buf:mask{ 2, 2, 32, 9 }
buf[2] ~ accesses 2 bits from bit-2 position
What exactly would buf[3] return. Please be explicit in whether you are
counting byte offsets or bit offsets. I can't figure that out from your
description.
It would return 32 bits (converted or not, depending on 'net' flag)
from bit-4 (MSB-ordered).
From bit-4? From your own use of 1-based indices don't you mean the
5th bit?
Personally, the idea of making array access to the buffer depend on
state stored in the buffer does not look appealing to me. It prevents
buffers to be passed around because consumers don't know what they will
get back on array access.
I think it could be useful to access nonaligned and aligned data
easily without caring about naming fields.
I guess I did not make myself understandable. Storing that state globally
means that every function you hand an lbuf too has to set the global
mask before it accesses the lbuf, because it can't know if the current
mask is compatible with its own use. And after call to a function, the
caller has to
Re: Lua in-kernel (lbuf library)
18.10.2013, в 21:03, John Nemeth jnem...@cue.bc.ca написал(а): On Oct 18, 11:03am, Marc Balmer wrote: } Am 18.10.13 10:43, schrieb Artem Falcon: } Marc Balmer marc at msys.ch writes: } Justin Cormack justin at specialbusservice.com writes: } I have been using the luajit ffi and luaffi, which let you directly } use C structs (with bitfields) in Lua to do this. It makes it easier } to reuse stuff that is already defined in C. (luaffi is not in its } current state portable but my plan is to strip out the non portable } bits, which are the function call support). } } Justin } } I had successfully used more lightweight solution called Lua AutoC [1] with } Marc's lua(4). } Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be } used in kernel, does the things in runtime, which gives the flexibility. } Cons: not widely tested, again does the things in runtime, which on other } side may give performance penalty. } } } I never used luaffi. It sounds very interesting and I think it could } be very useful to bind already defined C structs, but my purpose is to } dynamically define data layouts using Lua syntax (without parsing C } code). } } FFI in the kernel can be dangerous. Pure Lua is a perfect confinment } for code, but with an FFI a Lua script can access almost anything in the } kernel. One has to think twice if one wants that. } } Well, assuming it would be module, so I would not have to load it if I } don't want to. } } It's desirable if you're writing a device driver in Lua, as you can do } most of work from Lua code (e.g. call C methods of NetBSD driver API } and feed them with C structs and pointers). } States and explicit exports of a certain foreign functions makes things } a bit less dangerous. } But in general you're right, one should do this with care. } } lua(4) has a mechanism for Lua's 'require' statement. Normally, when } you require 'foo', it looks up wheter a kernel module name luafoo exists } and loads it. This automatic loading of modules can be turned off, to } make a module available to a state, it has to be specifically assigned. } So when you turn autoloading off, a script could not simply call a ffi } module by requiring it. } } Maybe Lua kernel modules should carry a flag whether they should allow } autoloading or not? This way, an ffi module would still be loaded into } the kernel when Lua code requires it, but lua(4) would detect the don't } autoload flag and would then not_ assign the module to the Lua state. Probably. It should be named as 'auto assign' for clarity, as module loading occurs anyway. There is already a mechanism for this, see module_autoload(9). You should always be using module_autoload() to load a module from inside the kernel. If the no autoload flag is set, then the call will fail. This is exactly what lua(4) does on 'requiring'. Thus, there is no need for lua(4) to try managing this itself. It should just attempt to load the module. If successful, great. If not, then the feature being requested isn't available. kern.lua.autoload is a safety barrier. One may wish not allow any lua kernel script to load any given lua kernel module. } [1] https://github.com/orangeduck/LuaAutoC } }-- End of excerpt from Marc Balmer -- dukzcry
Re: Lua in-kernel (lbuf library)
Terry Moore wrote: Just to clarify a bit Indeed, we started with Lua because AWK was not embeddable and because of the 1-origin issue. We thought, mistakenly, that a language that didn't look very much like C would cause fewer problems because of the 0-origin / 1-origin difference. Apparently, however, it's a deeper habit of thought. Ditto for the string escape sequences. Apparently the '[' and ']' for indexing, and the '\' for character escapes, trigger very deeply seated patterns. While I agree that 0-1 switch is mentally hard [*] but if you often need to access arrays by index in Lua code then you either solving a wrong problem of doing something a wrong way. Lua is designed to be a glue language. If you need arrays (as opposite to collections which can be iterated over with ipairs/pairs/iterators), then you're likely doing low-level C stuff in Lua. Data structures in kernel are often organized as linked lists. To iterate from Lua, you probably want to write an iterator and also a lookup function. Lets say you want to iterate over all ps processes from ddb (if gdb has python support, why can't we have a cooler thing?). You do this: ddb lua on ddb for p in processes:match(ps) do print(p.pid) end 906 2245 4935 ... ddb =processes:find(1).path /sbin/init ddb lua off You can do a lot without ever accessing elements by integer indices. Based on our experience, it seems risky to use Lua to implement code that interoperates with kernel-like C code in security-critical contexts. We found this risk to be insurmountable. Engineers who are used to zero-origin code, and who are looking at zero-origin code for reference, will make zero-origin mistakes. Again, don't do C stuff in Lua. However, there is a different kind of risk when developing security code in Lua: it's the layers of Lua itselft. Any complex layer introduces a non-neglidgible risk. [*] I found it out while developing mixed Lua-C module which could also detect LuaJIT and use its zero-based FFI structures. Alex
Re: Lua in-kernel (lbuf library)
Am 19.10.13 14:02, schrieb Alexander Nasonov: [...] Lets say you want to iterate over all ps processes from ddb (if gdb has python support, why can't we have a cooler thing?). You do this: ddb lua on ddb for p in processes:match(ps) do print(p.pid) end 906 2245 4935 ... ddb =processes:find(1).path /sbin/init ddb lua off You can do a lot without ever accessing elements by integer indices. Actually it would be quite nice to have a 'lua' command in ddb (instead of 'lua on' that drops you in an interactive Lua mode, like the 'lua' command line command. As the system is perhaps in a weird state when ddb is entered, it would be a good idea when the Lua state for ddb would have been created e.g. during system startup, and maybe during normal system operation, luactl(8) could be used to load scripts into it, which can define functions that will be available once ddb is entered. for those planning to tinker which such stuff, don't only look at the Lua C API, but especially try to understand Lua's metatable concept. This is where the real strength and power of Lua comes from. You can e.g. define what 'foo.bar' or 'bla.foo = 42' really means.
Re: Lua in-kernel (lbuf library)
Marc Balmer marc at msys.ch writes: Justin Cormack justin at specialbusservice.com writes: I have been using the luajit ffi and luaffi, which let you directly use C structs (with bitfields) in Lua to do this. It makes it easier to reuse stuff that is already defined in C. (luaffi is not in its current state portable but my plan is to strip out the non portable bits, which are the function call support). Justin I had successfully used more lightweight solution called Lua AutoC [1] with Marc's lua(4). Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be used in kernel, does the things in runtime, which gives the flexibility. Cons: not widely tested, again does the things in runtime, which on other side may give performance penalty. I never used luaffi. It sounds very interesting and I think it could be very useful to bind already defined C structs, but my purpose is to dynamically define data layouts using Lua syntax (without parsing C code). FFI in the kernel can be dangerous. Pure Lua is a perfect confinment for code, but with an FFI a Lua script can access almost anything in the kernel. One has to think twice if one wants that. Well, assuming it would be module, so I would not have to load it if I don't want to. It's desirable if you're writing a device driver in Lua, as you can do most of work from Lua code (e.g. call C methods of NetBSD driver API and feed them with C structs and pointers). States and explicit exports of a certain foreign functions makes things a bit less dangerous. But in general you're right, one should do this with care. [1] https://github.com/orangeduck/LuaAutoC
Re: Lua in-kernel (lbuf library)
Lourival Vieira Neto lourival.neto at gmail.com writes: On Wed, Oct 16, 2013 at 11:45 AM, Justin Cormack justin at specialbusservice.com wrote: (...) Yes absolutely it makes more sense if already defined in C. For parsing binary stuff I would look at Erlang for inspiration too, it is one of the nicer designs. Justin I never gone that far in Erlang. It looks really interesting [1]. I'll take a deeper look later. Thanks! Regards, I think a hybrid approach with mutable C-like data structs for holding of binary data and Erlang's binary pattern matching would be cool. -- dukzcry
Re: Lua in-kernel (lbuf library)
Am 18.10.13 10:43, schrieb Artem Falcon: Marc Balmer marc at msys.ch writes: Justin Cormack justin at specialbusservice.com writes: I have been using the luajit ffi and luaffi, which let you directly use C structs (with bitfields) in Lua to do this. It makes it easier to reuse stuff that is already defined in C. (luaffi is not in its current state portable but my plan is to strip out the non portable bits, which are the function call support). Justin I had successfully used more lightweight solution called Lua AutoC [1] with Marc's lua(4). Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be used in kernel, does the things in runtime, which gives the flexibility. Cons: not widely tested, again does the things in runtime, which on other side may give performance penalty. I never used luaffi. It sounds very interesting and I think it could be very useful to bind already defined C structs, but my purpose is to dynamically define data layouts using Lua syntax (without parsing C code). FFI in the kernel can be dangerous. Pure Lua is a perfect confinment for code, but with an FFI a Lua script can access almost anything in the kernel. One has to think twice if one wants that. Well, assuming it would be module, so I would not have to load it if I don't want to. It's desirable if you're writing a device driver in Lua, as you can do most of work from Lua code (e.g. call C methods of NetBSD driver API and feed them with C structs and pointers). States and explicit exports of a certain foreign functions makes things a bit less dangerous. But in general you're right, one should do this with care. lua(4) has a mechanism for Lua's 'require' statement. Normally, when you require 'foo', it looks up wheter a kernel module name luafoo exists and loads it. This automatic loading of modules can be turned off, to make a module available to a state, it has to be specifically assigned. So when you turn autoloading off, a script could not simply call a ffi module by requiring it. Maybe Lua kernel modules should carry a flag whether they should allow autoloading or not? This way, an ffi module would still be loaded into the kernel when Lua code requires it, but lua(4) would detect the don't autoload flag and would then not_ assign the module to the Lua state. [1] https://github.com/orangeduck/LuaAutoC
Re: Lua in-kernel (lbuf library)
On Oct 18, 11:03am, Marc Balmer wrote: } Am 18.10.13 10:43, schrieb Artem Falcon: } Marc Balmer marc at msys.ch writes: } Justin Cormack justin at specialbusservice.com writes: } I have been using the luajit ffi and luaffi, which let you directly } use C structs (with bitfields) in Lua to do this. It makes it easier } to reuse stuff that is already defined in C. (luaffi is not in its } current state portable but my plan is to strip out the non portable } bits, which are the function call support). } } Justin } } I had successfully used more lightweight solution called Lua AutoC [1] with } Marc's lua(4). } Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be } used in kernel, does the things in runtime, which gives the flexibility. } Cons: not widely tested, again does the things in runtime, which on other } side may give performance penalty. } } } I never used luaffi. It sounds very interesting and I think it could } be very useful to bind already defined C structs, but my purpose is to } dynamically define data layouts using Lua syntax (without parsing C } code). } } FFI in the kernel can be dangerous. Pure Lua is a perfect confinment } for code, but with an FFI a Lua script can access almost anything in the } kernel. One has to think twice if one wants that. } } Well, assuming it would be module, so I would not have to load it if I } don't want to. } } It's desirable if you're writing a device driver in Lua, as you can do } most of work from Lua code (e.g. call C methods of NetBSD driver API } and feed them with C structs and pointers). } States and explicit exports of a certain foreign functions makes things } a bit less dangerous. } But in general you're right, one should do this with care. } } lua(4) has a mechanism for Lua's 'require' statement. Normally, when } you require 'foo', it looks up wheter a kernel module name luafoo exists } and loads it. This automatic loading of modules can be turned off, to } make a module available to a state, it has to be specifically assigned. } So when you turn autoloading off, a script could not simply call a ffi } module by requiring it. } } Maybe Lua kernel modules should carry a flag whether they should allow } autoloading or not? This way, an ffi module would still be loaded into } the kernel when Lua code requires it, but lua(4) would detect the don't } autoload flag and would then not_ assign the module to the Lua state. There is already a mechanism for this, see module_autoload(9). You should always be using module_autoload() to load a module from inside the kernel. If the no autoload flag is set, then the call will fail. Thus, there is no need for lua(4) to try managing this itself. It should just attempt to load the module. If successful, great. If not, then the feature being requested isn't available. } [1] https://github.com/orangeduck/LuaAutoC } }-- End of excerpt from Marc Balmer
Re: Lua in-kernel (lbuf library)
Terry Moore t...@mcci.com wrote: ... But it was really, really difficult for our embedded C programmers to use. We found the following areas of constant problem: 1) zero origin versus 1-origin arrays were a constant source of bugs, particularly when interoperating with C code. 2) the string escape sequences are almost but not completely compatible with C. In particular, in C, \123 is interpreted as octal, but in Lua \123 is decimal. This was a constant source of trouble. 3) the fact that a reference uninitialized variable or missing array element returns nil instead of throwing an exception is problematic for writing significant code that is intended to be reliable -- errors are hidden. 4) the Lua APIs are not a platform -- they change from version to version. (Working with Lua APIs and trying to track current is like working with certain open-source kernels -- you really can't maintain your investment.) And so forth. All of these decisions are actually fine, in context of a scripting language that's a world unto itself. But as a tool to give C programmers, they collectively were a real barrier to adoption. To address this, and to make it more palatable to our developers, after two years or so of experience (in 2002) MCCI forked the Lua interpreter and created a C-like variant (with renamed, stable APIs to address #4, and C-like syntax). There was a paper on this at the Lua conference in 2008: http://www.lua.org/wshop08.html#moore With that, we've been happily using something based on Lua, and I can agree with all the Lua enthusiasts: it's a great tool. If we weren't an entirely C-based company, we would not have bothered with the language changes (though we might have created stable APIs with non-clashing names). I'm not promoting our version as a solution. I am, however, pointing out that items #1 and #2 above merit careful consideration before putting Lua into a security-critical environment; C programmers won't like reviewing the code (and will tend to miss things). Yet again, this whole story just makes me wonder why AWK has never evolved to be more powerful language for this kind of purpose (Perl is not the direction I am talking about). Certainly, simplicity of AWK is valued. However, it does not mean that it should have just stopped evolving at some point in history. Even if its functions use 1-origin for arrays, the language itself is much more natural for the C programmers and the UNIX world. Personally, I would love to see more advanced AWK with just-in-time compilation.. -- Mindaugas
Re: Lua in-kernel (lbuf library)
On Oct 19, 12:13am, Artem Falcon wrote: } 18.10.2013, × 21:03, John Nemeth jnem...@cue.bc.ca wrote: } On Oct 18, 11:03am, Marc Balmer wrote: } } Am 18.10.13 10:43, schrieb Artem Falcon: } } Marc Balmer marc at msys.ch writes: } } Justin Cormack justin at specialbusservice.com writes: } } I have been using the luajit ffi and luaffi, which let you directly } } use C structs (with bitfields) in Lua to do this. It makes it easier } } to reuse stuff that is already defined in C. (luaffi is not in its } } current state portable but my plan is to strip out the non portable } } bits, which are the function call support). } } } } Justin } } } } I had successfully used more lightweight solution called Lua AutoC [1] with } } Marc's lua(4). } } Pros: light in comparison to other FFI libs, joy in use, easy to adopt to be } } used in kernel, does the things in runtime, which gives the flexibility. } } Cons: not widely tested, again does the things in runtime, which on other } } side may give performance penalty. } } } } } } I never used luaffi. It sounds very interesting and I think it could } } be very useful to bind already defined C structs, but my purpose is to } } dynamically define data layouts using Lua syntax (without parsing C } } code). } } } } FFI in the kernel can be dangerous. Pure Lua is a perfect confinment } } for code, but with an FFI a Lua script can access almost anything in the } } kernel. One has to think twice if one wants that. } } } } Well, assuming it would be module, so I would not have to load it if I } } don't want to. } } } } It's desirable if you're writing a device driver in Lua, as you can do } } most of work from Lua code (e.g. call C methods of NetBSD driver API } } and feed them with C structs and pointers). } } States and explicit exports of a certain foreign functions makes things } } a bit less dangerous. } } But in general you're right, one should do this with care. } } } } lua(4) has a mechanism for Lua's 'require' statement. Normally, when } } you require 'foo', it looks up wheter a kernel module name luafoo exists } } and loads it. This automatic loading of modules can be turned off, to } } make a module available to a state, it has to be specifically assigned. } } So when you turn autoloading off, a script could not simply call a ffi } } module by requiring it. } } } } Maybe Lua kernel modules should carry a flag whether they should allow } } autoloading or not? This way, an ffi module would still be loaded into } } the kernel when Lua code requires it, but lua(4) would detect the don't } } autoload flag and would then not_ assign the module to the Lua state. } } Probably. It should be named as 'auto assign' for clarity, as module loading } occurs anyway. } } There is already a mechanism for this, see module_autoload(9). } You should always be using module_autoload() to load a module from } inside the kernel. If the noautoload flag is set, then the call } will fail. } } This is exactly what lua(4) does on 'requiring'. } } Thus, there is no need for lua(4) to try managing this } itself. It should just attempt to load the module. If successful, } great. If not, then the feature being requested isn't available. } } kern.lua.autoload is a safety barrier. One may wish not allow any lua kernel } script to load any given lua kernel module. The lua(4) implementers can certainly do this if they want. However, module_autoload() won't be looking at this flag and will continue to refuse to autoload any module that has the noautoload flag set. Also, there is the kern.module.autoload sysctl that can prevent any module from autoloading. }-- End of excerpt from Artem Falcon
RE: Lua in-kernel (lbuf library)
Yet again, this whole story just makes me wonder why AWK has never evolved to be more powerful language for this kind of purpose (Perl is not the direction I am talking about). Certainly, simplicity of AWK is valued. However, it does not mean that it should have just stopped evolving at some point in history. Even if its functions use 1-origin for arrays, the language itself is much more natural for the C programmers and the UNIX world. Personally, I would love to see more advanced AWK with just-in-time compilation.. Indeed, we started with Lua because AWK was not embeddable and because of the 1-origin issue. (Also, because Lua supports closures; closures are really a very economical way of expressing certain kinds of operations.) But we had 10 years or so of using AWK, wrapped by sh, as a cross UNIX platform. Unfortunately, it didn't work well for Windows in those days. And sh scripts still don't, Cygwin notwithstanding. But for many things we still use AWK. --Terry
RE: Lua in-kernel (lbuf library)
Just to clarify a bit Indeed, we started with Lua because AWK was not embeddable and because of the 1-origin issue. We thought, mistakenly, that a language that didn't look very much like C would cause fewer problems because of the 0-origin / 1-origin difference. Apparently, however, it's a deeper habit of thought. Ditto for the string escape sequences. Apparently the '[' and ']' for indexing, and the '\' for character escapes, trigger very deeply seated patterns. Based on our experience, it seems risky to use Lua to implement code that interoperates with kernel-like C code in security-critical contexts. We found this risk to be insurmountable. Engineers who are used to zero-origin code, and who are looking at zero-origin code for reference, will make zero-origin mistakes. This is not an argument against using Lua in user mode. However, given that the motto of NetBSD for kernel work is read the source, I worry that this human-factors issue will be hard to mitigate when using Lua in kernel mode. --Terry
RE: Lua in-kernel (lbuf library)
HI, I'm replying to a message that was on tech-userlevel@. I'm not sure why a discussion of in-kernel Lua was on tech-userlevel@. I see that other related messages were on tech-kern@, so I'm posting my reply there. My company has quite a bit of experience using Lua as a scripting language -- since 2000. Lua has many advantages and I like it very much. But it was really, really difficult for our embedded C programmers to use. We found the following areas of constant problem: 1) zero origin versus 1-origin arrays were a constant source of bugs, particularly when interoperating with C code. 2) the string escape sequences are almost but not completely compatible with C. In particular, in C, \123 is interpreted as octal, but in Lua \123 is decimal. This was a constant source of trouble. 3) the fact that a reference uninitialized variable or missing array element returns nil instead of throwing an exception is problematic for writing significant code that is intended to be reliable -- errors are hidden. 4) the Lua APIs are not a platform -- they change from version to version. (Working with Lua APIs and trying to track current is like working with certain open-source kernels -- you really can't maintain your investment.) And so forth. All of these decisions are actually fine, in context of a scripting language that's a world unto itself. But as a tool to give C programmers, they collectively were a real barrier to adoption. To address this, and to make it more palatable to our developers, after two years or so of experience (in 2002) MCCI forked the Lua interpreter and created a C-like variant (with renamed, stable APIs to address #4, and C-like syntax). There was a paper on this at the Lua conference in 2008: http://www.lua.org/wshop08.html#moore With that, we've been happily using something based on Lua, and I can agree with all the Lua enthusiasts: it's a great tool. If we weren't an entirely C-based company, we would not have bothered with the language changes (though we might have created stable APIs with non-clashing names). I'm not promoting our version as a solution. I am, however, pointing out that items #1 and #2 above merit careful consideration before putting Lua into a security-critical environment; C programmers won't like reviewing the code (and will tend to miss things). Best regards, --Terry -Original Message- From: tech-userlevel-ow...@netbsd.org [mailto:tech-userlevel- ow...@netbsd.org] On Behalf Of Mouse Sent: Wednesday, October 16, 2013 13:18 To: tech-userle...@netbsd.org Subject: Re: Lua in-kernel (lbuf library) [...0-origin vs 1-origin arrays...] It is hard to tell what is the least astonishing here. Well, least astonishing to whom, is really the question, it seems to me. Certainly I, as a C coder with no Lua experience, would find 0-origin arrays less astonishing. Someone with the converse experience would presumably have the opposite reaction. You propose Lua as a language embedded into C rather than separate one. I'd say that Lua designers made wrong decision here. Only if you think of Lua as being designed for embedding in C. It's just as coherent to think of the mistake as being trying to wed a language with 1-origin arrays with a language with 0-origin arrays. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTML mo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Lua in-kernel (lbuf library)
Am 15.10.13 23:01, schrieb Lourival Vieira Neto: [...] Also, having to switch mentally between zero-based arrays in the kernel C code and 1-based arrays in the Lua code make my head ache. It's something that doesn't bug me so much.. But, if necessary it could be changed to 0-based in this userdata. In C an array index is actually an offset from the top, so 0 is the natural way to denote element nr. 1 in C. In Lua, a numeric array index is not an offset, but the ordinal array position. So 1 is the natural way to denote the first element. Strictly speaking, it's actually C that is weird: Index n denotes array element n + 1... Following the principle of least astonishment, I would not recommend starting to do 0 based stuff in Lua, a Lua programmer certainly expects things to start at 1. [...]
Re: Lua in-kernel (lbuf library)
Hi Justin, On Tue, Oct 15, 2013 at 7:38 PM, Justin Cormack jus...@specialbusservice.com wrote: On Thu, Oct 10, 2013 at 7:15 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: Hi folks, It has been a long time since my GSoC project and though I have tried to come back, I've experienced some personal issues. However, now I'm coding again. I'm developing a library to handle buffers in Lua, named lbuf. It is been developed as part of my efforts to perform experimentation in kernel network stack using Lua. Initially, I intended to bind mbuf to allow, for example, to write protocols dissectors in Lua. For example, calling a Lua function to inspect network packets: function filter(packet) if packet.field == value then return DROP end return PASS end Thus, I started to design a Lua binding to mbuf inspired by '#pragma pack' and bitfields of C lang. Then, I realized that this Lua library could be useful to other kernel (and user-space) areas, such as device drivers and user-level protocols. So, I started to develop this binding generically as a independent library to give random access to bits in a buffer. It is just in the early beginning, but I want to share some thoughts. I have been using the luajit ffi and luaffi, which let you directly use C structs (with bitfields) in Lua to do this. It makes it easier to reuse stuff that is already defined in C. (luaffi is not in its current state portable but my plan is to strip out the non portable bits, which are the function call support). Justin I never used luaffi. It sounds very interesting and I think it could be very useful to bind already defined C structs, but my purpose is to dynamically define data layouts using Lua syntax (without parsing C code). Regards, -- Lourival Vieira Neto
Re: Lua in-kernel (lbuf library)
On Wed, Oct 16, 2013 at 3:50 AM, Marc Balmer m...@msys.ch wrote: Am 15.10.13 23:01, schrieb Lourival Vieira Neto: [...] Also, having to switch mentally between zero-based arrays in the kernel C code and 1-based arrays in the Lua code make my head ache. It's something that doesn't bug me so much.. But, if necessary it could be changed to 0-based in this userdata. In C an array index is actually an offset from the top, so 0 is the natural way to denote element nr. 1 in C. In Lua, a numeric array index is not an offset, but the ordinal array position. So 1 is the natural way to denote the first element. Strictly speaking, it's actually C that is weird: Index n denotes array element n + 1... Following the principle of least astonishment, I would not recommend starting to do 0 based stuff in Lua, a Lua programmer certainly expects things to start at 1. [...] Indeed. -- Lourival Vieira Neto
Re: Lua in-kernel (lbuf library)
On 16 Oct 2013 15:41, Lourival Vieira Neto lourival.n...@gmail.com wrote: Hi Justin, On Tue, Oct 15, 2013 at 7:38 PM, Justin Cormack jus...@specialbusservice.com wrote: On Thu, Oct 10, 2013 at 7:15 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: Hi folks, It has been a long time since my GSoC project and though I have tried to come back, I've experienced some personal issues. However, now I'm coding again. I'm developing a library to handle buffers in Lua, named lbuf. It is been developed as part of my efforts to perform experimentation in kernel network stack using Lua. Initially, I intended to bind mbuf to allow, for example, to write protocols dissectors in Lua. For example, calling a Lua function to inspect network packets: function filter(packet) if packet.field == value then return DROP end return PASS end Thus, I started to design a Lua binding to mbuf inspired by '#pragma pack' and bitfields of C lang. Then, I realized that this Lua library could be useful to other kernel (and user-space) areas, such as device drivers and user-level protocols. So, I started to develop this binding generically as a independent library to give random access to bits in a buffer. It is just in the early beginning, but I want to share some thoughts. I have been using the luajit ffi and luaffi, which let you directly use C structs (with bitfields) in Lua to do this. It makes it easier to reuse stuff that is already defined in C. (luaffi is not in its current state portable but my plan is to strip out the non portable bits, which are the function call support). Justin I never used luaffi. It sounds very interesting and I think it could be very useful to bind already defined C structs, but my purpose is to dynamically define data layouts using Lua syntax (without parsing C code). Yes absolutely it makes more sense if already defined in C. For parsing binary stuff I would look at Erlang for inspiration too, it is one of the nicer designs. Justin
Re: Lua in-kernel (lbuf library)
Am 16.10.13 16:41, schrieb Lourival Vieira Neto: Hi Justin, On Tue, Oct 15, 2013 at 7:38 PM, Justin Cormack jus...@specialbusservice.com wrote: On Thu, Oct 10, 2013 at 7:15 PM, Lourival Vieira Neto lourival.n...@gmail.com wrote: Hi folks, It has been a long time since my GSoC project and though I have tried to come back, I've experienced some personal issues. However, now I'm coding again. I'm developing a library to handle buffers in Lua, named lbuf. It is been developed as part of my efforts to perform experimentation in kernel network stack using Lua. Initially, I intended to bind mbuf to allow, for example, to write protocols dissectors in Lua. For example, calling a Lua function to inspect network packets: function filter(packet) if packet.field == value then return DROP end return PASS end Thus, I started to design a Lua binding to mbuf inspired by '#pragma pack' and bitfields of C lang. Then, I realized that this Lua library could be useful to other kernel (and user-space) areas, such as device drivers and user-level protocols. So, I started to develop this binding generically as a independent library to give random access to bits in a buffer. It is just in the early beginning, but I want to share some thoughts. I have been using the luajit ffi and luaffi, which let you directly use C structs (with bitfields) in Lua to do this. It makes it easier to reuse stuff that is already defined in C. (luaffi is not in its current state portable but my plan is to strip out the non portable bits, which are the function call support). Justin I never used luaffi. It sounds very interesting and I think it could be very useful to bind already defined C structs, but my purpose is to dynamically define data layouts using Lua syntax (without parsing C code). FFI in the kernel can be dangerous. Pure Lua is a perfect confinment for code, but with an FFI a Lua script can access almost anything in the kernel. One has to think twice if one wants that. Well, assuming it would be module, so I would not have to load it if I don't want to.
Re: Lua in-kernel (lbuf library)
Marc Balmer m...@msys.ch writes: Am 15.10.13 23:01, schrieb Lourival Vieira Neto: Also, having to switch mentally between zero-based arrays in the kernel C code and 1-based arrays in the Lua code make my head ache. It's something that doesn't bug me so much.. But, if necessary it could be changed to 0-based in this userdata. In C an array index is actually an offset from the top, so 0 is the natural way to denote element nr. 1 in C. In Lua, a numeric array index is not an offset, but the ordinal array position. So 1 is the natural way to denote the first element. Strictly speaking, it's actually C that is weird: Index n denotes array element n + 1... This depends on your background. If you studied or dealt with mathematical logic, set theory, or foundations of mathematics, then you start counting natural numbers (ordinals) from 0. In this respect C is more logical (pun intended) than Lua. Following the principle of least astonishment, I would not recommend starting to do 0 based stuff in Lua, a Lua programmer certainly expects things to start at 1. It is hard to tell what is the least astonishing here. You propose Lua as a language embedded into C rather than separate one. I'd say that Lua designers made wrong decision here. -- BCE HA MOPE!
Re: Lua in-kernel (lbuf library)
On Wed, Oct 16, 2013 at 11:45 AM, Justin Cormack jus...@specialbusservice.com wrote: (...) Yes absolutely it makes more sense if already defined in C. For parsing binary stuff I would look at Erlang for inspiration too, it is one of the nicer designs. Justin I never gone that far in Erlang. It looks really interesting [1]. I'll take a deeper look later. Thanks! Regards, -- Lourival Vieira Neto
Re: Lua in-kernel (lbuf library)
Hi Christoph,
On Mon, Oct 14, 2013 at 10:02 AM, Christoph Badura b...@bsd.de wrote:
First, I find the usage of the buf terminology confusing. In kernel
context I associate buf with the file system buffe cache buf structure.
Packet buffers a called mbufs. I would appreciate it if the terminology
was consistent with the kernel or at least not confusing.
This is due my lack of creativeness =).. I'm quite open for naming suggestions.
Also, having to switch mentally between zero-based arrays in the kernel C
code and 1-based arrays in the Lua code make my head ache.
It's something that doesn't bug me so much.. But, if necessary it
could be changed to 0-based in this userdata.
On Thu, Oct 10, 2013 at 03:15:54PM -0300, Lourival Vieira Neto wrote:
C API:
lbuf_new(lua_State L, void * buffer, size_t length, lua_Alloc free, bool
net);
* creates a new lbuf userdatum and pushes it on the Lua stack. The net
flag indicates if it is necessary to perform endianness conversion.
I what is buffer and how does it relate to mbufs? How do I create a new
lbuf from an mbuf? Or from an array of bytes?
Note, non-contiguous buffer still an open problem in lbuf. I don't
know if should use a ptrdiff_t to pass the distance to 'next' field,
a 'next()' to return 'next' field or something else.
However, you could create a lbuf from a mbuf header as follows:
lbuf_new(L, mbuf-m_data, mbuf-m_len, NULL, true);
or from an array:
uint8_t array[ N ];
lbuf_new(L, (void *) array, N, NULL, false); // 'false' means 'use
the platform endianess'
Then, you could call a Lua function passing this lbuf, for example:
lua_getglobal(L, handler);
lbuf_new(L, mbuf-m_data, mbuf-m_len, NULL, true);
lua_pcall(L, 1, 0, 0);
In order to indicate that endianness conversion is necessary I need to
know the future uses of the buffer. Clairvoyance excepted, that is kinda
hard.
It's a generic data structure that could be used to handle bit fields
or nonaligned data.
If you are going to make the buffers endianness aware, why not record the
endianness that the packet is encoded in. And byteswapping can be
performed automatically depending on the consumers endianness. I think
this way a lot of redundant code can be avoided.
And you don't describe under what circumstances endianness convresion is
performed.
Yes, mea culpa =(. I wasn't clear about that. 'net' flag was the way I
found to 'record' the buffer endianness. What means, true if the
buffer uses BE and false if it uses HE. It has the same semantics of
hton* and ntoh* functions. Don't know if it is better to pass the
endianness itself as a flag (e.g., enum { BIG_ENDIAN, LITTLE_ENDIAN,
HOST_ENDIAN }). What do you think?
So, if you set net flag true when you access a bit field, the
conversion to and from big endian, if needed, is done automatically
taking the smaller aligned set of bits. For example:
buf:rawget(0, 9) ~ if net flag is *true*: takes 16 bits from
beginning of the buffer (as is); convert these 2 bytes from BE to HE
(if necessary); and returns these 2 bytes masked to preserve only the
most significant 9 bits (zeroing the remaining bits) and shifted to
LSB. If net is *false*: just returns the first 2 bytes masked and
shifted (without conversion). Then these 2 bytes are expanded to
lua_Number type (int64_t in kernel)
That is:
a) If net flag is _true_ and the platform is LE:
1- Takes 16 bits:
[ b0 | b1 | b2 | b3 | b4 | b5 | b6 | b7 ][ b8 | b9 | b10 |
b11 | b12 | b13 | b14 | b15 ]
2- Convert it to LE:
[ b8 | b9 | b10 | b11 | b12 | b13 | b14 | b15 ][ b0 | b1 | b2 |
b3 | b4 | b5 | b6 | b7 ]
3- Returns the first 2 bytes masked and shifted:
[ b1 | b2 | b3 | b4 | b5 | b6 | b7 | b8 ][ 0 | 0 | 0 | 0 | 0 | 0 | 0 | b0 ]
b) If net flag is _false_ and the platform is LE:
1- Takes 16 bits:
[ b0 | b1 | b2 | b3 | b4 | b5 | b6 | b7 ][ b8 | b9 | b10 |
b11 | b12 | b13 | b14 | b15 ]
2- Returns the first 2 bytes masked and shifted:
[ b9 | b10 | b11 | b12 | b13 | b14 | b15 | b0 ][ 0 | 0 | 0 | 0
| 0 | 0 | 0 | b8 ]
c) If net flag is _true or false_ and platform is BE:
1- Takes 16 bits:
[ b0 | b1 | b2 | b3 | b4 | b5 | b6 | b7 ][ b8 | b9 | b10 |
b11 | b12 | b13 | b14 | b15 ]
2- Returns the first 2 bytes masked and shifted:
[ 0 | 0 | 0 | 0 | 0 | 0 | 0 | b0 ][ b1 | b2 | b3 | b4 |
b5 | b6 | b7 | b8 ]
Lua API:
- array access (1)
lbuf:mask(alignment [, offset, length])
buf[ix] ~ accesses 'alignment' bits from 'alignment*(ix -1)+offset' position
e.g.:
buf:mask(3)
buf[3] ~ accesses 3 bits from bit-6 position
What does that mean? Does it return the top-most 2 bits from the first
byte plus the least significant bit fom the second byte of the buffer?
It means the least-most 2 bits from the first byte and the LSB from the second.
What is 'length' for?
Offset and length could be used to impose boundaries to the mask. For
example, if you want to analyse a segment of the buffer that is
organized
Re: Lua in-kernel (lbuf library)
Christoph Badura wrote: Also, having to switch mentally between zero-based arrays in the kernel C code and 1-based arrays in the Lua code make my head ache. Yeah, I totally agree here. There are several other reasons why Lua will not become same league player with C in the kernel. But for some projects, the classical module (in C) and scripting (in Lua) separation works extremely well. This includes complex configurations where you need to orchestrate many calls to C code or some complex tasks like generating code for bpf or now defunct npf opcode. Alex
Re: Lua in-kernel (lbuf library)
Lourival Vieira Neto wrote: I'm developing a library to handle buffers in Lua, named lbuf. It is been developed as part of my efforts to perform experimentation in kernel network stack using Lua. Initially, I intended to bind mbuf to allow, for example, to write protocols dissectors in Lua. For example, calling a Lua function to inspect network packets: function filter(packet) if packet.field == value then return DROP end return PASS end Thus, I started to design a Lua binding to mbuf inspired by '#pragma pack' and bitfields of C lang. Then, I realized that this Lua library could be useful to other kernel (and user-space) areas, such as device drivers and user-level protocols. So, I started to develop this binding generically as a independent library to give random access to bits in a buffer. It is just in the early beginning, but I want to share some thoughts. I wonder if you looked at Lua support in Wireshark [1]? Unfortunately, it's GPL and they even have a special section 'Beware the GPL' on wiki. [1] http://wiki.wireshark.org/Lua Alex
Re: Lua in-kernel (lbuf library)
On Tue, Oct 15, 2013 at 7:22 PM, Alexander Nasonov al...@yandex.ru wrote: Lourival Vieira Neto wrote: I'm developing a library to handle buffers in Lua, named lbuf. It is been developed as part of my efforts to perform experimentation in kernel network stack using Lua. Initially, I intended to bind mbuf to allow, for example, to write protocols dissectors in Lua. For example, calling a Lua function to inspect network packets: function filter(packet) if packet.field == value then return DROP end return PASS end Thus, I started to design a Lua binding to mbuf inspired by '#pragma pack' and bitfields of C lang. Then, I realized that this Lua library could be useful to other kernel (and user-space) areas, such as device drivers and user-level protocols. So, I started to develop this binding generically as a independent library to give random access to bits in a buffer. It is just in the early beginning, but I want to share some thoughts. I wonder if you looked at Lua support in Wireshark [1]? Unfortunately, it's GPL and they even have a special section 'Beware the GPL' on wiki. [1] http://wiki.wireshark.org/Lua Alex Yes. In fact, I have already implemented a Wireshark dissector in Lua for a proprietary protocol that I was designing, inspired in ERP, to detect network loops. WS Lua dissectors also served as inspiration. However, I just used the API; I never looked at the binding implementation. Wireshark Lua dissectors is a good example of what can be done with Lua in that sense. But I'm looking for a more generic API that could allow random bit access in a buffer using Lua table notation, that could also be used to communicate with devices, for example. I think (IMHO) that lbuf masks is more straight forward. Regards, -- Lourival Vieira Neto
Re: Lua in-kernel (lbuf library)
First, I find the usage of the buf terminology confusing. In kernel
context I associate buf with the file system buffe cache buf structure.
Packet buffers a called mbufs. I would appreciate it if the terminology
was consistent with the kernel or at least not confusing.
Also, having to switch mentally between zero-based arrays in the kernel C
code and 1-based arrays in the Lua code make my head ache.
On Thu, Oct 10, 2013 at 03:15:54PM -0300, Lourival Vieira Neto wrote:
C API:
lbuf_new(lua_State L, void * buffer, size_t length, lua_Alloc free, bool net);
* creates a new lbuf userdatum and pushes it on the Lua stack. The net
flag indicates if it is necessary to perform endianness conversion.
I what is buffer and how does it relate to mbufs? How do I create a new
lbuf from an mbuf? Or from an array of bytes?
In order to indicate that endianness conversion is necessary I need to
know the future uses of the buffer. Clairvoyance excepted, that is kinda
hard.
If you are going to make the buffers endianness aware, why not record the
endianness that the packet is encoded in. And byteswapping can be
performed automatically depending on the consumers endianness. I think
this way a lot of redundant code can be avoided.
And you don't describe under what circumstances endianness convresion is
performed.
Lua API:
- array access (1)
lbuf:mask(alignment [, offset, length])
buf[ix] ~ accesses 'alignment' bits from 'alignment*(ix -1)+offset' position
e.g.:
buf:mask(3)
buf[3] ~ accesses 3 bits from bit-6 position
What does that mean? Does it return the top-most 2 bits from the first
byte plus the least significant bit fom the second byte of the buffer?
What is 'length' for?
How does endianness conversion fit in?
- array access (2)
buf:mask{ length_pos1, length_pos2, ... }
buf[ix] ~ accesses 'length_pos(ix)' bits from 'length_pos1 + ...
length_pos(ix-1)' position
e.g.:
buf:mask{ 2, 2, 32, 9 }
buf[2] ~ accesses 2 bits from bit-2 position
What exactly would buf[3] return. Please be explicit in whether you are
counting byte offsets or bit offsets. I can't figure that out from your
description.
Personally, the idea of making array access to the buffer depend on
state stored in the buffer does not look appealing to me. It prevents
buffers to be passed around because consumers don't know what they will
get back on array access.
buf:mask{ field = { offset, length }, ... }
buf.field ~ 'field.length' bits from 'offset' position
This actually makes some sense to me.
buf:segment(offset [, length])
returns a new lbuf corresponding a 'buf' segment.
What is a a 'segment' actually?
- mask reusing
lbuf.mask{ ... }
This makes sense again...
function filter(packet)
packet:mask(ethernet_mask)
if packet.type == 0x88CC then
lldp_pdu = packet.segment(payload_offset):mask(lldp_mask)
if packet.version 1 return DROP end
end
return PASS
end
... except the code seems to be not runnable. Where does 'payload_offset'
come from? And don't you mean lldp_pdu.version?
I find it not helpful when the examples do not actually work.
--chris
