[SSSD] Embedding Lua into SSSD
Hi everyone, I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? Lua is a language specifically designed to be embedded. It grew into popularity after it was embedded into World Of Warcraft. It was also embedded and is still being embedded into a bunch of other games. However, games is far from the only application for Lua. Lua is a simple and lightweight language, yet with lots of flexibility - grasping the essence of the whole reference manual [2] takes one evening and it's an entertaining read. The VM is small and fast. In fact some people use it on microcontrollers [3]. I personally embedded it into two C-based projects, one of which was already quite big, and another was a rewrite of a smallish, but complicated C project. In both cases, where before things were tedious and hard, they have become easy and fun. The first project was on an embedded platform. I also had some other, but smaller stuff done with it. Generally, Lua gets put into the middle with outside interfaces and performance-critical parts implemented in C. With a very simple interface to and from C one can put boundaries wherever necessary. Yes, the interface is simpler than Python's. The process is this: you implement some C libraries (shared/static) which are loadable from Lua (Lua->C interface) and then create a VM or several, load some Lua code and call into it (C->Lua interface). Some of the benefits are: Clearer logic (higher level language) Much less memory leaks and very little memory management (mostly in Lua->C interface). Faster development cycle (no compilation, edit and retry on a live daemon) Potentially easy plugin interface (yay for admin scripts!) Some of the drawbacks are: More difficult to do low level stuff (so we just don't do that) Lower performance than C (still good enough for games, and we can have the critical parts in C easily) Needs learning and development investment. Regarding the development investment, due to low cost of C/Lua interface implementation, it is practical to replace the relevant C code piece-by-piece, provided pieces are not too small (say, 2KLOC and up). Lastly, don't take this too seriously. I realise that this is disruptive and hard. However if you ever feel like SSSD has grown too big and complex, ping me, or just try Lua yourself :) Nick [1] http://www.lua.org/ [2] http://www.lua.org/manual/5.3/manual.html [3] http://www.eluaproject.net/ ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On Fri, 2015-08-21 at 20:01 +0300, Nikolai Kondrashov wrote: > Hi everyone, > > I might be in a strange and careless mood today, but here is something I > wanted to suggest since the time I saw the amount of logic that goes into SSSD > and is implemented in C. > > What if we implement some of the complicated logic inside SSSD in Lua [1]? > > Lua is a language specifically designed to be embedded. It grew into > popularity after it was embedded into World Of Warcraft. It was also embedded > and is still being embedded into a bunch of other games. However, games is far > from the only application for Lua. > > Lua is a simple and lightweight language, yet with lots of flexibility - > grasping the essence of the whole reference manual [2] takes one evening and > it's an entertaining read. The VM is small and fast. In fact some people use > it on microcontrollers [3]. > > I personally embedded it into two C-based projects, one of which was already > quite big, and another was a rewrite of a smallish, but complicated C project. > In both cases, where before things were tedious and hard, they have become > easy and fun. The first project was on an embedded platform. I also had some > other, but smaller stuff done with it. > > Generally, Lua gets put into the middle with outside interfaces and > performance-critical parts implemented in C. With a very simple interface to > and from C one can put boundaries wherever necessary. Yes, the interface is > simpler than Python's. > > The process is this: you implement some C libraries (shared/static) which are > loadable from Lua (Lua->C interface) and then create a VM or several, load > some Lua code and call into it (C->Lua interface). > > Some of the benefits are: > > Clearer logic (higher level language) > Much less memory leaks and very little memory management (mostly in > Lua->C > interface). > Faster development cycle (no compilation, edit and retry on a live > daemon) > Potentially easy plugin interface (yay for admin scripts!) > > Some of the drawbacks are: > > More difficult to do low level stuff (so we just don't do that) > Lower performance than C (still good enough for games, and we can have > the > critical parts in C easily) > Needs learning and development investment. > > Regarding the development investment, due to low cost of C/Lua interface > implementation, it is practical to replace the relevant C code piece-by-piece, > provided pieces are not too small (say, 2KLOC and up). > > Lastly, don't take this too seriously. I realise that this is disruptive and > hard. However if you ever feel like SSSD has grown too big and complex, ping > me, or just try Lua yourself :) > > Nick > > [1] http://www.lua.org/ > [2] http://www.lua.org/manual/5.3/manual.html > [3] http://www.eluaproject.net/ Hi Nick, can you provide an example of a piece of SSSD youd replace with Lua ? I am not asking for an implementation but a high level view of what a function looks like to day and what it would look like if we were going to use Lua (ideally with a list of primitives we'd still need to provide, to understand how much code Lua replaces for real. Simo. -- Simo Sorce * Red Hat, Inc * New York ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: Hi everyone, I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? Lua is a language specifically designed to be embedded. It grew into popularity after it was embedded into World Of Warcraft. It was also embedded and is still being embedded into a bunch of other games. However, games is far from the only application for Lua. Lua is a simple and lightweight language, yet with lots of flexibility - grasping the essence of the whole reference manual [2] takes one evening and it's an entertaining read. The VM is small and fast. In fact some people use it on microcontrollers [3]. I personally embedded it into two C-based projects, one of which was already quite big, and another was a rewrite of a smallish, but complicated C project. In both cases, where before things were tedious and hard, they have become easy and fun. The first project was on an embedded platform. I also had some other, but smaller stuff done with it. Generally, Lua gets put into the middle with outside interfaces and performance-critical parts implemented in C. With a very simple interface to and from C one can put boundaries wherever necessary. Yes, the interface is simpler than Python's. The process is this: you implement some C libraries (shared/static) which are loadable from Lua (Lua->C interface) and then create a VM or several, load some Lua code and call into it (C->Lua interface). Some of the benefits are: Clearer logic (higher level language) Much less memory leaks and very little memory management (mostly in Lua->C interface). Faster development cycle (no compilation, edit and retry on a live daemon) Potentially easy plugin interface (yay for admin scripts!) Some of the drawbacks are: More difficult to do low level stuff (so we just don't do that) Lower performance than C (still good enough for games, and we can have the critical parts in C easily) Needs learning and development investment. Regarding the development investment, due to low cost of C/Lua interface implementation, it is practical to replace the relevant C code piece-by-piece, provided pieces are not too small (say, 2KLOC and up). Lastly, don't take this too seriously. I realise that this is disruptive and hard. However if you ever feel like SSSD has grown too big and complex, ping me, or just try Lua yourself :) Nick [1] http://www.lua.org/ [2] http://www.lua.org/manual/5.3/manual.html [3] http://www.eluaproject.net/ Hi, can you tell us what features of Lua do you like and might help simplify SSSD? I don't know the language, just fast scrolled over the manual and seen few examples. It seems to be similar to javascript. ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: Hi everyone, I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? Lua is a language specifically designed to be embedded. It grew into popularity after it was embedded into World Of Warcraft. It was also embedded and is still being embedded into a bunch of other games. However, games is far from the only application for Lua. Lua is a simple and lightweight language, yet with lots of flexibility - grasping the essence of the whole reference manual [2] takes one evening and it's an entertaining read. The VM is small and fast. In fact some people use it on microcontrollers [3]. I personally embedded it into two C-based projects, one of which was already quite big, and another was a rewrite of a smallish, but complicated C project. In both cases, where before things were tedious and hard, they have become easy and fun. The first project was on an embedded platform. I also had some other, but smaller stuff done with it. Generally, Lua gets put into the middle with outside interfaces and performance-critical parts implemented in C. With a very simple interface to and from C one can put boundaries wherever necessary. Yes, the interface is simpler than Python's. The process is this: you implement some C libraries (shared/static) which are loadable from Lua (Lua->C interface) and then create a VM or several, load some Lua code and call into it (C->Lua interface). Some of the benefits are: Clearer logic (higher level language) Much less memory leaks and very little memory management (mostly in Lua->C interface). Faster development cycle (no compilation, edit and retry on a live daemon) Potentially easy plugin interface (yay for admin scripts!) Some of the drawbacks are: More difficult to do low level stuff (so we just don't do that) Lower performance than C (still good enough for games, and we can have the critical parts in C easily) Needs learning and development investment. Regarding the development investment, due to low cost of C/Lua interface implementation, it is practical to replace the relevant C code piece-by-piece, provided pieces are not too small (say, 2KLOC and up). Lastly, don't take this too seriously. I realise that this is disruptive and hard. However if you ever feel like SSSD has grown too big and complex, ping me, or just try Lua yourself :) Nick I may be too conservative here, but I believe SSSD should be written in language with static type-checking at least as strong as C. Michal ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
Hi Simo, On 08/21/2015 09:37 PM, Simo Sorce wrote: On Fri, 2015-08-21 at 20:01 +0300, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? can you provide an example of a piece of SSSD youd replace with Lua ? I am not asking for an implementation but a high level view of what a function looks like to day and what it would look like if we were going to use Lua (ideally with a list of primitives we'd still need to provide, to understand how much code Lua replaces for real. I don't have enough knowledge of SSSD to provide really meaningful examples. I have to know developer pain points to really suggest an interesting solution, and I have to really know the code to show good examples. I don't have the time now to really dive into the code and give a good answer, I'm sorry. If there's interest and support for this idea, I will be glad to dig in, give you more, and help with coding and design if it comes to that. Still, judging from the code distribution and overall architecture, I would say the first candidates will be providers. They are very complicated and have a relatively well defined interface to SSSD core. Most of them talk over network and any slowdown from Lua will likely be swamped by network latency. I would make a (partial?) adaptor from the current interface to Lua, allowing implementing a (limited?) provider in Lua, plus low-level communication libraries underneath. This will allow testing the feasibility without disrupting the existing providers. If the idea proves useful, existing providers can be re-implemented in Lua, perhaps section-by-section, and starting from the smallest. If all the providers are converted, the adaptor can be removed and the core will talk directly to high-level Lua provider code. One side-effect of such work will be refactoring and simplifying of the provider interface, which will have a positive effect on the overall architecture. I expect main savings to come from exception support, automatic memory management, simpler data massaging, and brevity of a scripting language (e.g. implicit, dynamic typing). Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
Hi Pavel, On 08/24/2015 11:54 AM, Pavel Březina wrote: On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? can you tell us what features of Lua do you like and might help simplify SSSD? I don't know the language, just fast scrolled over the manual and seen few examples. It seems to be similar to javascript. Most of all, I like its simplicity, then flexibility. And then how easy it is to embed. The latter comes from simplicity, first of all. I listed the features I think will help us in the other message in this thread, but here they are again and then some: * exception handling (somewhat rudimentary, but workable) * automatic memory management (incremental mark-and-sweep with good controls, customizable allocator) * easier data massaging (string operations, built-in (limited) regexes, etc.) * brevity (implicit dynamic typing, syntax sugar) * OOP support (there are several libraries to choose from) I don't know enough JavaScript to really compare, but to me knowing that it has two pairs of equality operators [1] is quite off-putting. There is, however, a seemingly good comparison, which should give the general idea: http://stackoverflow.com/a/1022683/1161045 Nick [1] https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference#Equality_operators ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
Hi Michal, On 08/24/2015 05:06 PM, Michal Židek wrote: On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? I may be too conservative here, but I believe SSSD should be written in language with static type-checking at least as strong as C. I agree that C's static typing is very useful, even though C also requires implicit typing bringing verbosity. Still, using Lua is a tradeoff. We can trade static typing for the benefits Lua brings, which mostly boil down to smaller, clearer code. BTW, a while ago I read a good treatise on the topic, which I'm happy to share: https://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 03:57 PM, Nikolai Kondrashov wrote: I expect main savings to come from exception support, automatic memory management, simpler data massaging, and brevity of a scripting language (e.g. implicit, dynamic typing). Oh, and another thing which scripting languages bring and which I'm not tiring of mentioning is live, hands-on development. We can make SSSD reload Lua code without recompilation and restarting, speeding up development/debugging/testing cycle. Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 04:21 PM, Nikolai Kondrashov wrote: Hi Michal, On 08/24/2015 05:06 PM, Michal Židek wrote: On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? I may be too conservative here, but I believe SSSD should be written in language with static type-checking at least as strong as C. I agree that C's static typing is very useful, even though C also requires implicit typing bringing verbosity. I meant "explicit" here, of course. Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 03:21 PM, Nikolai Kondrashov wrote: Hi Michal, On 08/24/2015 05:06 PM, Michal Židek wrote: On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? I may be too conservative here, but I believe SSSD should be written in language with static type-checking at least as strong as C. I agree that C's static typing is very useful, even though C also requires implicit typing bringing verbosity. Still, using Lua is a tradeoff. We can trade static typing for the benefits Lua brings, which mostly boil down to smaller, clearer code. BTW, a while ago I read a good treatise on the topic, which I'm happy to share: https://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ Nick Before we use Lua (or any additional language) we need to do some major refactoring inside SSSD. Create internal libraries with clean API that operates on defined levels of abstraction for every bigger component in SSSD (like responders, providers, sysdb etc.) and than re-implement these components using the language of choice and these libraries (libraries would still be in C). Even if the language of choice for the higher level components remains to be C, such refactoring will IMO sooner or later be needed. However it will be a *very* big task and we would need to plan how to deliver such change into SSSD. Internally in SSSD we are far from clean APIs. It partially may be because we use C on all levels of abstractions which sometimes leads to their unwanted merging. We sometimes use structures from lower levels too high in the stack or mix the parts in all directions to simply make stuff work somehow and do not always think about clean design. Using more abstract language for upper (or top) layers may help to fight this issue a little, but the main issue is still bad API of the internal libraries and their unclear position in the stack. Once we fix these issues we can start thinking about adopting new language into the project but the possibility is that it will no longer be such an issue at that point. Anyway, I think you raised a good topic to think about. Michal ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 06:05 PM, Michal Židek wrote: On 08/25/2015 03:21 PM, Nikolai Kondrashov wrote: On 08/24/2015 05:06 PM, Michal Židek wrote: On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? I may be too conservative here, but I believe SSSD should be written in language with static type-checking at least as strong as C. I agree that C's static typing is very useful, even though C also requires implicit typing bringing verbosity. Still, using Lua is a tradeoff. We can trade static typing for the benefits Lua brings, which mostly boil down to smaller, clearer code. BTW, a while ago I read a good treatise on the topic, which I'm happy to share: https://cdsmith.wordpress.com/2011/01/09/an-old-article-i-wrote/ Before we use Lua (or any additional language) we need to do some major refactoring inside SSSD. Create internal libraries with clean API that operates on defined levels of abstraction for every bigger component in SSSD (like responders, providers, sysdb etc.) and than re-implement these components using the language of choice and these libraries (libraries would still be in C). Even if the language of choice for the higher level components remains to be C, such refactoring will IMO sooner or later be needed. However it will be a *very* big task and we would need to plan how to deliver such change into SSSD. Internally in SSSD we are far from clean APIs. It partially may be because we use C on all levels of abstractions which sometimes leads to their unwanted merging. We sometimes use structures from lower levels too high in the stack or mix the parts in all directions to simply make stuff work somehow and do not always think about clean design. Using more abstract language for upper (or top) layers may help to fight this issue a little, but the main issue is still bad API of the internal libraries and their unclear position in the stack. Once we fix these issues we can start thinking about adopting new language into the project but the possibility is that it will no longer be such an issue at that point. Anyway, I think you raised a good topic to think about. Thank you, Michal. I agree that sssd is very much in need of refactoring and once we'll have it done, the code will be easier to read and understand. However, it will still be C, with all the associated overhead for the high level code. Also, it will stay C. Refactoring never ends, it's like weeding out a garden. So, we'll have to refactor C again and again. If the target of refactoring is written in a more suitable language, it will be that bit more pleasant and effective, and subsequent refactoring will be easier as well. Although, I suspect once/if we get a more powerful tool, we'll simply use it to build a more complex system and will be back to square one, but at least we'll get more features done :) Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On (25/08/15 15:57), Nikolai Kondrashov wrote: >Hi Simo, > >On 08/21/2015 09:37 PM, Simo Sorce wrote: >>On Fri, 2015-08-21 at 20:01 +0300, Nikolai Kondrashov wrote: >>>I might be in a strange and careless mood today, but here is something I >>>wanted to suggest since the time I saw the amount of logic that goes into >>>SSSD >>>and is implemented in C. >>> >>>What if we implement some of the complicated logic inside SSSD in Lua [1]? >> >>can you provide an example of a piece of SSSD youd replace with Lua ? >>I am not asking for an implementation but a high level view of what a >>function looks like to day and what it would look like if we were going >>to use Lua (ideally with a list of primitives we'd still need to >>provide, to understand how much code Lua replaces for real. > >I don't have enough knowledge of SSSD to provide really meaningful examples. >I have to know developer pain points to really suggest an interesting >solution, and I have to really know the code to show good examples. > SSSD is written in asynchronous way and tightly coupled with tevent[1] and talloc. This is not use case for using scripting language. The ideal use case for scripting languages is to write higly optimized library in C + bindings. The scipting language would be used for replacing boilerplate code in C. However there are still few ways. You can write simple example of using lua scripts with tevent[1]. Then we can evaluate pros and cons of using Lua. The another way would be to replace tevent[1] with lua implementation of event loop. In another words, rewrite sssd from scratch. >Still, judging from the code distribution and overall architecture, I would >say the first candidates will be providers. They are very complicated and have >a relatively well defined interface to SSSD core. Most of them talk over >network and any slowdown from Lua will likely be swamped by network latency. > Providers are very complicated due to compicated servers (LDAP, IPA, AD) We will not be able to replace high level iplementation with scripting language in short time (Lua -> C) . In short time, there still will need to be two way comunication between C and Lua (Lua <-> C). Which would not be simplify code as the first way. >I would make a (partial?) adaptor from the current interface to Lua, allowing >implementing a (limited?) provider in Lua, plus low-level communication >libraries underneath. This will allow testing the feasibility without >disrupting the existing providers. > Will it be asynchronous? In theory, you could use threads from scripting language but we there could be synchronisation related issues. So the ideal would be to use Erlang and message passing instead of shared variables. >If the idea proves useful, existing providers can be re-implemented in Lua, >perhaps section-by-section, and starting from the smallest. If all the >providers are converted, the adaptor can be removed and the core will talk >directly to high-level Lua provider code. One side-effect of such work will >be refactoring and simplifying of the provider interface, which will have a >positive effect on the overall architecture. > There is a missing step between a POC and re-implementing existing providers. We need to write functional test to assure we will not create a regressions. As I already wrote servers are complicated (LDAP, IPA, AD); So code coverage will need to be very high. >I expect main savings to come from > exception support, scripting language might help > automatic memory management, talloc helps a lot even in C. > simpler data massaging, What do you mean here? > and brevity of a scripting language (e.g. implicit, dynamic typing). > This can be pros and cons in the same time. Summary: This is an interesting idea; but currently I cannot see many benefits for using any scripting language in SSSD. Moreover sssd is a security service. Such step could cause problems with security certifications. LS [1] https://tevent.samba.org/ ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 04:14 PM, Nikolai Kondrashov wrote: Hi Pavel, On 08/24/2015 11:54 AM, Pavel Březina wrote: On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? can you tell us what features of Lua do you like and might help simplify SSSD? I don't know the language, just fast scrolled over the manual and seen few examples. It seems to be similar to javascript. Most of all, I like its simplicity, then flexibility. And then how easy it is to embed. The latter comes from simplicity, first of all. I listed the features I think will help us in the other message in this thread, but here they are again and then some: * exception handling (somewhat rudimentary, but workable) * automatic memory management (incremental mark-and-sweep with good controls, customizable allocator) * easier data massaging (string operations, built-in (limited) regexes, etc.) * brevity (implicit dynamic typing, syntax sugar) * OOP support (there are several libraries to choose from) I don't know enough JavaScript to really compare, but to me knowing that it has two pairs of equality operators [1] is quite off-putting. There is, however, a seemingly good comparison, which should give the general idea: http://stackoverflow.com/a/1022683/1161045 Some things about that comparison: * Lua supports bitwise operators now. * There are no typeof and instanceof operators, but functions doing that are usually implemented in OOP libraries * It doesn't mention that Lua uses 1-based array indices by default [1]. It feels awkward at first, but you can get used to it. Nick [1] A long discussion on the official wiki :) http://lua-users.org/wiki/CountingFromOne ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 06:59 PM, Lukas Slebodnik wrote: On (25/08/15 15:57), Nikolai Kondrashov wrote: Hi Simo, On 08/21/2015 09:37 PM, Simo Sorce wrote: On Fri, 2015-08-21 at 20:01 +0300, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? can you provide an example of a piece of SSSD youd replace with Lua ? I am not asking for an implementation but a high level view of what a function looks like to day and what it would look like if we were going to use Lua (ideally with a list of primitives we'd still need to provide, to understand how much code Lua replaces for real. I don't have enough knowledge of SSSD to provide really meaningful examples. I have to know developer pain points to really suggest an interesting solution, and I have to really know the code to show good examples. SSSD is written in asynchronous way and tightly coupled with tevent[1] and talloc. This is not use case for using scripting language. The ideal use case for scripting languages is to write higly optimized library in C + bindings. The scipting language would be used for replacing boilerplate code in C. However there are still few ways. You can write simple example of using lua scripts with tevent[1]. Then we can evaluate pros and cons of using Lua. Now we're talking specifics, awesome! Alright, perhaps I'll do that. Care to give a use case I should try to implement? The another way would be to replace tevent[1] with lua implementation of event loop. In another words, rewrite sssd from scratch. Whoa, no way :) Still, judging from the code distribution and overall architecture, I would say the first candidates will be providers. They are very complicated and have a relatively well defined interface to SSSD core. Most of them talk over network and any slowdown from Lua will likely be swamped by network latency. Providers are very complicated due to compicated servers (LDAP, IPA, AD) We will not be able to replace high level iplementation with scripting language in short time (Lua -> C) . In short time, there still will need to be two way comunication between C and Lua (Lua <-> C). Which would not be simplify code as the first way. Sure. It's complicated. That's why a higher level language might be beneficial. If we can cut off some parts of a provider cleanly (which is a good idea for refactoring anyway), we can reimplement them in Lua with relatively low overhead. Thus, going by pieces we can reduce the disruption. I would make a (partial?) adaptor from the current interface to Lua, allowing implementing a (limited?) provider in Lua, plus low-level communication libraries underneath. This will allow testing the feasibility without disrupting the existing providers. Will it be asynchronous? In theory, you could use threads from scripting language but we there could be synchronisation related issues. Well, we can try it? I assume a lot of people did such things already. There is a bunch of multithreading libraries for Lua, and then there are always the built-in coroutines. So the ideal would be to use Erlang and message passing instead of shared variables. Sadly, I don't know Erlang yet, can't say anything about that. I bet it's awesome. However, I expect it is harder to interface with as well. If the idea proves useful, existing providers can be re-implemented in Lua, perhaps section-by-section, and starting from the smallest. If all the providers are converted, the adaptor can be removed and the core will talk directly to high-level Lua provider code. One side-effect of such work will be refactoring and simplifying of the provider interface, which will have a positive effect on the overall architecture. There is a missing step between a POC and re-implementing existing providers. We need to write functional test to assure we will not create a regressions. As I already wrote servers are complicated (LDAP, IPA, AD); So code coverage will need to be very high. Yeah, this bothers me too, it is dangerous, I agree. Perhaps if we do it slowly, piece-by-piece, it will be safer and easier to manage, i.e. write tests for, or test manually? I expect main savings to come from exception support, scripting language might help automatic memory management, talloc helps a lot even in C. I agree. Still, it's explicit and needs taking care of, constantly. simpler data massaging, What do you mean here? The messy stuff. Parsing and gluing strings together, maintaining complicated data structures, converting from one representation to another, etc. and brevity of a scripting language (e.g. implicit, dynamic typing). This can be pros and cons in the same time. Implicit, dynamic typing - yes. Brevity - not likely. You can always make it more verbose, after all, but it is hard th
Re: [SSSD] Embedding Lua into SSSD
On Tue, 2015-08-25 at 16:14 +0300, Nikolai Kondrashov wrote: > Hi Pavel, > > On 08/24/2015 11:54 AM, Pavel Březina wrote: > > On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: > >> I might be in a strange and careless mood today, but here is something I > >> wanted to suggest since the time I saw the amount of logic that goes into > >> SSSD and is implemented in C. > >> > >> What if we implement some of the complicated logic inside SSSD in Lua [1]? > > > > can you tell us what features of Lua do you like and might help simplify > > SSSD? I don't know the language, just fast scrolled over the manual and seen > > few examples. It seems to be similar to javascript. > > Most of all, I like its simplicity, then flexibility. And then how easy it is > to embed. The latter comes from simplicity, first of all. > > I listed the features I think will help us in the other message in this > thread, but here they are again and then some: > > * exception handling >(somewhat rudimentary, but workable) This may be interesting, but we are pretty diligent I think, thanks to the tevent_req design pattern, at surfacing error > * automatic memory management >(incremental mark-and-sweep with good controls, customizable allocator) This is not really interesting as we already have talloc that does a great job at keeping track of memory allocations and cleanups (esp if we stop using the anti-pattern of allocating on NULL as discussed a few times already). > * easier data massaging >(string operations, built-in (limited) regexes, etc.) We do some limited string manipulation but not really much, what other kind of data massaging does Lua provide that makes it worth ? > * brevity >(implicit dynamic typing, syntax sugar) brevity is good for long-winded functions indeed. > * OOP support >(there are several libraries to choose from) This may be good or bad, it all depends on what you need to do. How does Lua work with asynchronous operations ? We have a ton of that and unless Lua has native support it would be a huge mismatch. > I don't know enough JavaScript to really compare, but to me knowing that it > has two pairs of equality operators [1] is quite off-putting. I hope it doesn't compare much because javascript is awful, especially for anything security oriented. > There is, however, a seemingly good comparison, which should give the general > idea: > > http://stackoverflow.com/a/1022683/1161045 To be honest a Javascript to Lua comparison doesn't really help much understanding if Lua could be an asset for SSSD or not. See above questions. Simo. > Nick > > [1] > https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference#Equality_operators > ___ > sssd-devel mailing list > sssd-devel@lists.fedorahosted.org > https://lists.fedorahosted.org/mailman/listinfo/sssd-devel -- Simo Sorce * Red Hat, Inc * New York ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On Tue, 2015-08-25 at 17:59 +0200, Lukas Slebodnik wrote: > So the ideal would be to use > Erlang and message passing instead of shared variables. Say Erlang one more time! ... (cit.) >:-| Simo. -- Simo Sorce * Red Hat, Inc * New York ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 08:48 PM, Simo Sorce wrote: On Tue, 2015-08-25 at 16:14 +0300, Nikolai Kondrashov wrote: On 08/24/2015 11:54 AM, Pavel Březina wrote: On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: I might be in a strange and careless mood today, but here is something I wanted to suggest since the time I saw the amount of logic that goes into SSSD and is implemented in C. What if we implement some of the complicated logic inside SSSD in Lua [1]? can you tell us what features of Lua do you like and might help simplify SSSD? I don't know the language, just fast scrolled over the manual and seen few examples. It seems to be similar to javascript. Most of all, I like its simplicity, then flexibility. And then how easy it is to embed. The latter comes from simplicity, first of all. I listed the features I think will help us in the other message in this thread, but here they are again and then some: * exception handling (somewhat rudimentary, but workable) This may be interesting, but we are pretty diligent I think, thanks to the tevent_req design pattern, at surfacing error Main benefit of exceptions to me is not to have to propagate errors explicitly. * automatic memory management (incremental mark-and-sweep with good controls, customizable allocator) This is not really interesting as we already have talloc that does a great job at keeping track of memory allocations and cleanups (esp if we stop using the anti-pattern of allocating on NULL as discussed a few times already). You still need to do it explicitly and think about it. Not that much, of course, and it's not that you don't have to think about it with a mark-and-sweep GC at all. Still, in the majority of cases you don't have to think about it, and you certainly don't have to assign memory to contexts and keep them, nor pass them around. * easier data massaging (string operations, built-in (limited) regexes, etc.) We do some limited string manipulation but not really much, what other kind of data massaging does Lua provide that makes it worth ? Well, can't think of many specifics, ATM, but let's say easier structure creation for example. Just stuff data into tables in any place with a succinct syntax. Especially useful for ad-hoc single-use structuring, where writing constructor functions is not worth it, and allocating a structure and then assigning fields in place is a pain. * brevity (implicit dynamic typing, syntax sugar) brevity is good for long-winded functions indeed. Well, it kind of works all through the code. * OOP support (there are several libraries to choose from) This may be good or bad, it all depends on what you need to do. Sure. Although having something is better than not having, theoretically. Except diseases, of course :) How does Lua work with asynchronous operations ? We have a ton of that and unless Lua has native support it would be a huge mismatch. Now this is a good question. It depends on what you mean. If you mean central loop with state machines underneath, then coroutines is an awesome solution. If you mean OS threads, then there are several libraries that support it, AFAIK. I haven't looked into them though. We'll have to try it to really know. If you can give me an example, I can try thinking about it. I don't know enough JavaScript to really compare, but to me knowing that it has two pairs of equality operators [1] is quite off-putting. I hope it doesn't compare much because javascript is awful, especially for anything security oriented. Now, this is interesting. Care to elaborate? Is there anything really in the language itself and not the browser parts? Not that I like JavaScript myself, just curious. There is, however, a seemingly good comparison, which should give the general idea: http://stackoverflow.com/a/1022683/1161045 To be honest a Javascript to Lua comparison doesn't really help much understanding if Lua could be an asset for SSSD or not. Sure, I was answering the specific question from Pavel of how it compares to JavaScript. I think a lightweight embeddable language could be an asset to SSSD, in general. It doesn't have to be Lua. I think Lua is a good candidate, but there are other similar languages around, although probably not of such extended vintage. See above questions. Thanks! Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 09:26 PM, Nikolai Kondrashov wrote: I think a lightweight embeddable language could be an asset to SSSD, in general. It doesn't have to be Lua. I think Lua is a good candidate, but there are other similar languages around, although probably not of such extended vintage. There is one question on StackOverflow about what to use instead of Lua, which ends with a glowing feedback, I like it :) http://stackoverflow.com/questions/4448835/alternatives-to-lua-as-an-embedded-language Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On Tue, 2015-08-25 at 21:26 +0300, Nikolai Kondrashov wrote: > On 08/25/2015 08:48 PM, Simo Sorce wrote: > > On Tue, 2015-08-25 at 16:14 +0300, Nikolai Kondrashov wrote: > >> On 08/24/2015 11:54 AM, Pavel Březina wrote: > >>> On 08/21/2015 07:01 PM, Nikolai Kondrashov wrote: > I might be in a strange and careless mood today, but here is something I > wanted to suggest since the time I saw the amount of logic that goes into > SSSD and is implemented in C. > > What if we implement some of the complicated logic inside SSSD in Lua > [1]? > >>> > >>> can you tell us what features of Lua do you like and might help simplify > >>> SSSD? I don't know the language, just fast scrolled over the manual and > >>> seen > >>> few examples. It seems to be similar to javascript. > >> > >> Most of all, I like its simplicity, then flexibility. And then how easy it > >> is > >> to embed. The latter comes from simplicity, first of all. > >> > >> I listed the features I think will help us in the other message in this > >> thread, but here they are again and then some: > >> > >> * exception handling > >> (somewhat rudimentary, but workable) > > > > This may be interesting, but we are pretty diligent I think, thanks to > > the tevent_req design pattern, at surfacing error > > Main benefit of exceptions to me is not to have to propagate errors > explicitly. > > >> * automatic memory management > >> (incremental mark-and-sweep with good controls, customizable > >> allocator) > > > > This is not really interesting as we already have talloc that does a > > great job at keeping track of memory allocations and cleanups (esp if we > > stop using the anti-pattern of allocating on NULL as discussed a few > > times already). > > You still need to do it explicitly and think about it. Not that much, of > course, and it's not that you don't have to think about it with a > mark-and-sweep GC at all. I think a GC would be a regression to be honest,for the usage pattern of SSSD a hierarchical allocator makes better use of memory. > Still, in the majority of cases you don't have > to think about it, and you certainly don't have to assign memory to contexts > and keep them, nor pass them around. True, but then you do not ever think of memory allocation with usually comical situations where you leak tremendous amounts of memory because you keep some reference mistakenly. Once that happen finding out what is happening and reviewing every single variable allocation to find out what is going on is generally extremely hard. It's a trade off really. > >> * easier data massaging > >> (string operations, built-in (limited) regexes, etc.) > > > > We do some limited string manipulation but not really much, what other > > kind of data massaging does Lua provide that makes it worth ? > > Well, can't think of many specifics, ATM, but let's say easier structure > creation for example. Just stuff data into tables in any place with a succinct > syntax. Especially useful for ad-hoc single-use structuring, where writing > constructor functions is not worth it, and allocating a structure and then > assigning fields in place is a pain. Yes explicit memory management is a pain point in C, but in and on itself does not justify changing sssd, so I'll discount this point under the general "memory management is easier" argument. > >> * brevity > >> (implicit dynamic typing, syntax sugar) > > > > brevity is good for long-winded functions indeed. > > Well, it kind of works all through the code. Sometimes the brevity of scripting languages is obtained through extremely inefficient code, with potentially unwanted side-effect (esp wrt memory management). So it is not better in general, it is better for the specific goal of making long functions more readable. > >> * OOP support > >> (there are several libraries to choose from) > > > > This may be good or bad, it all depends on what you need to do. > > Sure. Although having something is better than not having, theoretically. > Except diseases, of course :) As you see, sometimes having "something" is worse. For software that works at lower levels OOP usually makes for software that is harder to debug, so it may be an anti-feature. > > How does Lua work with asynchronous operations ? We have a ton of that > > and unless Lua has native support it would be a huge mismatch. > > Now this is a good question. It depends on what you mean. If you mean central > loop with state machines underneath, then coroutines is an awesome solution. It is not clear to me that Lua's coroutines (http://www.lua.org/pil/9.1.html) would do what asynchronous tevents do. > If you mean OS threads, then there are several libraries that support it, > AFAIK. I haven't looked into them though. No, I do not mean threads, in fact sssd is not thread safe. I mean I/O based events where a socket file becoming readable makes the chain of nested ev
Re: [SSSD] Embedding Lua into SSSD
On (25/08/15 13:53), Simo Sorce wrote: >On Tue, 2015-08-25 at 17:59 +0200, Lukas Slebodnik wrote: >> So the ideal would be to use >> Erlang and message passing instead of shared variables. > >Say Erlang one more time! ... (cit.) >:-| > As you wish :-) Erlang LS ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On (25/08/15 19:45), Nikolai Kondrashov wrote: >On 08/25/2015 06:59 PM, Lukas Slebodnik wrote: >>On (25/08/15 15:57), Nikolai Kondrashov wrote: >>>Hi Simo, >>> >>>On 08/21/2015 09:37 PM, Simo Sorce wrote: On Fri, 2015-08-21 at 20:01 +0300, Nikolai Kondrashov wrote: >I might be in a strange and careless mood today, but here is something I >wanted to suggest since the time I saw the amount of logic that goes into >SSSD >and is implemented in C. > >What if we implement some of the complicated logic inside SSSD in Lua [1]? can you provide an example of a piece of SSSD youd replace with Lua ? I am not asking for an implementation but a high level view of what a function looks like to day and what it would look like if we were going to use Lua (ideally with a list of primitives we'd still need to provide, to understand how much code Lua replaces for real. >>> >>>I don't have enough knowledge of SSSD to provide really meaningful examples. >>>I have to know developer pain points to really suggest an interesting >>>solution, and I have to really know the code to show good examples. >>> >>SSSD is written in asynchronous way and tightly coupled with tevent[1] and >>talloc. This is not use case for using scripting language. The ideal use >>case for scripting languages is to write higly optimized library in C + >>bindings. The scipting language would be used for replacing boilerplate code >>in C. However there are still few ways. You can write simple example of >>using lua scripts with tevent[1]. Then we can evaluate pros and cons of >>using Lua. > >Now we're talking specifics, awesome! Alright, perhaps I'll do that. Care to >give a use case I should try to implement? > >>The another way would be to replace tevent[1] with lua implementation of >>event loop. >>In another words, rewrite sssd from scratch. > >Whoa, no way :) > >>>Still, judging from the code distribution and overall architecture, I would >>>say the first candidates will be providers. They are very complicated and >>>have >>>a relatively well defined interface to SSSD core. Most of them talk over >>>network and any slowdown from Lua will likely be swamped by network latency. >>> >>Providers are very complicated due to compicated servers (LDAP, IPA, AD) >>We will not be able to replace high level iplementation with scripting >>language >>in short time (Lua -> C) . In short time, there still will need to be two way >>comunication between C and Lua (Lua <-> C). Which would not be simplify code >>as >>the first way. > >Sure. It's complicated. That's why a higher level language might be >beneficial. If we can cut off some parts of a provider cleanly (which is a >good idea for refactoring anyway), we can reimplement them in Lua with >relatively low overhead. Thus, going by pieces we can reduce the disruption. > We separated logic to small parts with tevent style. * create request * do staff * register callback * call callback after finishing asynchronous operation. If we want to use tevent + lua there might not be a huge saving in LOC; You safe few lines with lua code, but you add extra lines for bingins to wrap tevent request. Maybe it would help to write some example. BTW you might find simple example of tevent style in libtevent (samba) itself. It might be faster way then extracting LDAP example from sssd. >>>I expect main savings to come from >>>exception support, >>scripting language might help >> >>>automatic memory management, >>talloc helps a lot even in C. > >I agree. Still, it's explicit and needs taking care of, constantly. > Yes, but there is a huge simplification with hierachical allocator + registering destructors for complicated stuff. So memory management is not a benefit. >>>simpler data massaging, >>What do you mean here? > >The messy stuff. Parsing and gluing strings together, maintaining complicated >data structures, converting from one representation to another, etc. > Agree. BTW gluing strings together is not very compicated with talloc_asprintf; however parsing strings would be a much nicer. >>>and brevity of a scripting language (e.g. implicit, dynamic typing). >>> >>This can be pros and cons in the same time. > >Implicit, dynamic typing - yes. Brevity - not likely. You can always make it >more verbose, after all, but it is hard the other way around. > >>Summary: >>This is an interesting idea; but currently I cannot see many benefits >>for using any scripting language in SSSD. > >I'm careful about this as well. Yet, it is exciting :) > >>Moreover sssd is a security service. Such step could cause problems with >>security certifications. > >Well, I know little of certification, and I'm worried about security too, >however, would C be safer, really? > >Do you anticipate any specific security problems? I do not have anything specific; It's just key factor which shoudl be considered for such step as using scripting language in sssd. You can write secure code even in C. h
Re: [SSSD] Embedding Lua into SSSD
On 08/26/2015 09:50 AM, Lukas Slebodnik wrote: On (25/08/15 19:45), Nikolai Kondrashov wrote: On 08/25/2015 06:59 PM, Lukas Slebodnik wrote: On (25/08/15 15:57), Nikolai Kondrashov wrote: Still, judging from the code distribution and overall architecture, I would say the first candidates will be providers. They are very complicated and have a relatively well defined interface to SSSD core. Most of them talk over network and any slowdown from Lua will likely be swamped by network latency. Providers are very complicated due to compicated servers (LDAP, IPA, AD) We will not be able to replace high level iplementation with scripting language in short time (Lua -> C) . In short time, there still will need to be two way comunication between C and Lua (Lua <-> C). Which would not be simplify code as the first way. Sure. It's complicated. That's why a higher level language might be beneficial. If we can cut off some parts of a provider cleanly (which is a good idea for refactoring anyway), we can reimplement them in Lua with relatively low overhead. Thus, going by pieces we can reduce the disruption. We separated logic to small parts with tevent style. * create request * do staff * register callback * call callback after finishing asynchronous operation. If we want to use tevent + lua there might not be a huge saving in LOC; You safe few lines with lua code, but you add extra lines for bingins to wrap tevent request. Maybe it would help to write some example. BTW you might find simple example of tevent style in libtevent (samba) itself. It might be faster way then extracting LDAP example from sssd. Thanks a lot, Lukas. I'll see what can be done, once I have time. I have some ideas already. Well, I know little of certification, and I'm worried about security too, however, would C be safer, really? Do you anticipate any specific security problems? I do not have anything specific; It's just key factor which shoudl be considered for such step as using scripting language in sssd. Alright, I'll do some research on this. You can write secure code even in C. https://docs.fedoraproject.org/en-US/Fedora_Security_Team/1/html/Defensive_Coding/index.html Sure. The question is effort. I should read that though. Thanks! Nick ___ sssd-devel mailing list sssd-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
Re: [SSSD] Embedding Lua into SSSD
On 08/25/2015 10:19 PM, Simo Sorce wrote: On Tue, 2015-08-25 at 21:26 +0300, Nikolai Kondrashov wrote: On 08/25/2015 08:48 PM, Simo Sorce wrote: On Tue, 2015-08-25 at 16:14 +0300, Nikolai Kondrashov wrote: * automatic memory management (incremental mark-and-sweep with good controls, customizable allocator) This is not really interesting as we already have talloc that does a great job at keeping track of memory allocations and cleanups (esp if we stop using the anti-pattern of allocating on NULL as discussed a few times already). You still need to do it explicitly and think about it. Not that much, of course, and it's not that you don't have to think about it with a mark-and-sweep GC at all. I think a GC would be a regression to be honest,for the usage pattern of SSSD a hierarchical allocator makes better use of memory. I agree, memory can be used more efficiently with a hierarchical allocator. However, is saving memory a priority over lowering complexity for providers? Still, in the majority of cases you don't have to think about it, and you certainly don't have to assign memory to contexts and keep them, nor pass them around. True, but then you do not ever think of memory allocation with usually comical situations where you leak tremendous amounts of memory because you keep some reference mistakenly. Once that happen finding out what is happening and reviewing every single variable allocation to find out what is going on is generally extremely hard. It's a trade off really. Agree. Doesn't happen often, though. In fact, I can't remember it happening to me. I had some difficulties bridging Lua and a C library (ClearSilver) memory management, but it wasn't Lua's fault and it worked in the end. * brevity (implicit dynamic typing, syntax sugar) brevity is good for long-winded functions indeed. Well, it kind of works all through the code. Sometimes the brevity of scripting languages is obtained through extremely inefficient code, with potentially unwanted side-effect (esp wrt memory management). So it is not better in general, it is better for the specific goal of making long functions more readable. Sure. We can only consider it better or worse in isolation from other factors. * OOP support (there are several libraries to choose from) This may be good or bad, it all depends on what you need to do. Sure. Although having something is better than not having, theoretically. Except diseases, of course :) As you see, sometimes having "something" is worse. For software that works at lower levels OOP usually makes for software that is harder to debug, so it may be an anti-feature. Sure. OOP might only be useful for higher-level logic, but I don't suggest we use it otherwise. It was just a point that might help us. How does Lua work with asynchronous operations ? We have a ton of that and unless Lua has native support it would be a huge mismatch. Now this is a good question. It depends on what you mean. If you mean central loop with state machines underneath, then coroutines is an awesome solution. It is not clear to me that Lua's coroutines (http://www.lua.org/pil/9.1.html) would do what asynchronous tevents do. I'll try marrying them when I have time and we'll see. If you mean OS threads, then there are several libraries that support it, AFAIK. I haven't looked into them though. No, I do not mean threads, in fact sssd is not thread safe. I mean I/O based events where a socket file becoming readable makes the chain of nested events keep going on until the innnermost completes it's work and data is returned (potentially massaged all the way back) to the outermost event. With control returning to the main loop every time an I/O operation would block. Ah, good, I was worried we might need threads. We'll have to try it to really know. If you can give me an example, I can try thinking about it. Look at how tevent_req stuff works / see above. I will, thank you. I don't know enough JavaScript to really compare, but to me knowing that it has two pairs of equality operators [1] is quite off-putting. I hope it doesn't compare much because javascript is awful, especially for anything security oriented. Now, this is interesting. Care to elaborate? Is there anything really in the language itself and not the browser parts? Not that I like JavaScript myself, just curious. The browsers just implement the language, which is quite awful. Well the fact that a browser would implement a scripting language is not that bad, IMO, but the huge mess of the DOM and the APIs is what makes it very hard to contain. I would never suggest we put a browser, or DOM into sssd :) There is, however, a seemingly good comparison, which should give the general idea: http://stackoverflow.com/a/1022683/1161045 To be honest a Javascript to Lua comparison doesn't really help much understanding if Lua could be an asset for SSSD or n
Re: [SSSD] Embedding Lua into SSSD
On Wed, 2015-08-26 at 16:10 +0300, Nikolai Kondrashov wrote: > On 08/25/2015 10:19 PM, Simo Sorce wrote: > > On Tue, 2015-08-25 at 21:26 +0300, Nikolai Kondrashov wrote: > >> On 08/25/2015 08:48 PM, Simo Sorce wrote: > >>> On Tue, 2015-08-25 at 16:14 +0300, Nikolai Kondrashov wrote: > * automatic memory management > (incremental mark-and-sweep with good controls, customizable > allocator) > >>> > >>> This is not really interesting as we already have talloc that does a > >>> great job at keeping track of memory allocations and cleanups (esp if we > >>> stop using the anti-pattern of allocating on NULL as discussed a few > >>> times already). > >> > >> You still need to do it explicitly and think about it. Not that much, of > >> course, and it's not that you don't have to think about it with a > >> mark-and-sweep GC at all. > > > > I think a GC would be a regression to be honest,for the usage pattern of > > SSSD a hierarchical allocator makes better use of memory. > > I agree, memory can be used more efficiently with a hierarchical allocator. > However, is saving memory a priority over lowering complexity for providers? It is important, yes, because this is a system service that always runs on the machine. We cannot explode the memory requirements or it will negatively affect the whole system, we are already borderline for some use cases. > >> Still, in the majority of cases you don't have > >> to think about it, and you certainly don't have to assign memory to > >> contexts > >> and keep them, nor pass them around. > > > > True, but then you do not ever think of memory allocation with usually > > comical situations where you leak tremendous amounts of memory because > > you keep some reference mistakenly. Once that happen finding out what is > > happening and reviewing every single variable allocation to find out > > what is going on is generally extremely hard. It's a trade off really. > > Agree. Doesn't happen often, though. In fact, I can't remember it happening to > me. I had some difficulties bridging Lua and a C library (ClearSilver) memory > management, but it wasn't Lua's fault and it worked in the end. > > * brevity > (implicit dynamic typing, syntax sugar) > >>> > >>> brevity is good for long-winded functions indeed. > >> > >> Well, it kind of works all through the code. > > > > Sometimes the brevity of scripting languages is obtained through > > extremely inefficient code, with potentially unwanted side-effect (esp > > wrt memory management). So it is not better in general, it is better for > > the specific goal of making long functions more readable. > > Sure. We can only consider it better or worse in isolation from other factors. > > * OOP support > (there are several libraries to choose from) > >>> > >>> This may be good or bad, it all depends on what you need to do. > >> > >> Sure. Although having something is better than not having, theoretically. > >> Except diseases, of course :) > > > > As you see, sometimes having "something" is worse. For software that > > works at lower levels OOP usually makes for software that is harder to > > debug, so it may be an anti-feature. > > Sure. OOP might only be useful for higher-level logic, but I don't suggest we > use it otherwise. It was just a point that might help us. > > >>> How does Lua work with asynchronous operations ? We have a ton of that > >>> and unless Lua has native support it would be a huge mismatch. > >> > >> Now this is a good question. It depends on what you mean. If you mean > >> central > >> loop with state machines underneath, then coroutines is an awesome > >> solution. > > > > It is not clear to me that Lua's coroutines > > (http://www.lua.org/pil/9.1.html) would do what asynchronous tevents do. > > I'll try marrying them when I have time and we'll see. > > >> If you mean OS threads, then there are several libraries that support it, > >> AFAIK. I haven't looked into them though. > > > > No, I do not mean threads, in fact sssd is not thread safe. I mean I/O > > based events where a socket file becoming readable makes the chain of > > nested events keep going on until the innnermost completes it's work and > > data is returned (potentially massaged all the way back) to the > > outermost event. With control returning to the main loop every time an > > I/O operation would block. > > Ah, good, I was worried we might need threads. > > >> We'll have to try it to really know. If you can give me an example, I can > >> try > >> thinking about it. > > > > Look at how tevent_req stuff works / see above. > > I will, thank you. > > I don't know enough JavaScript to really compare, but to me knowing that > it > has two pairs of equality operators [1] is quite off-putting. > >>> > >>> I hope it doesn't compare much because javascript is awful, especially > >>> for anything security oriented. > >> > >> Now, this is interestin
