Re: Introducing GNUPaste (and guile-wiredtiger future)
Le 17/12/2017 à 18:35, Kristofer Buffington a écrit : Amirouche Boubekkiwrites: Héllo, On 2017-12-14 19:37, Kristofer Buffington wrote: Hello! I am excited to share GNUPaste! This is a really simple web app similar to paste.lisp.org built with Guile. I have a linode running it from git on GuixSD. https://paste.freshbakedyams.com (Please use it!) Source: https://github.com/kristoferbuffington/gnupaste.git Currently the frontend uses twitter bootstrap + jquery and highlightjs from a CDN. It really doesn't need all that boilerplate. It will definitely change in the future. GNUPaste depends on guile-wiredtiger and guile-fibers to compile. Thanks for considering guile-wiredtiger. Basically, guile-wiredtiger is not compatible yet with fiber in the general case, because fiber will spawn several threads and several fibers in each thread (and I think that fibers can be stolen by other threads but I am not sure). The way wiredtiger works is that there is one *connexion* (called *environment* in guile wiredtiger) per database and one *session* per "thread" of execution (called *context* in guile wiredtiger). I changed the naming because they are different from the original things. Both environment and context are backed by fluids. In a pre-fork thread model, one must use with-context [1] after the fork to populate the current fuild with a specific context. [1] https://framagit.org/a-guile-mind/guile-wiredtiger/blob/master/wiredtiger/extra.scm#L328 The thing is that when using fibers, the thread of execution is a fiber, hence simply said the wiredtiger extra abstraction called context fails. Because context/session must not be shared between different fibers even if they are executed in the same thread. I have noticed that I get "Resource busy" issues if multiple requests are handled by the server simultaneously. I tried running (with-env ...) in the request-handler instead, but it is the same problem. Multiple environments can't open the database simultaneously either. I will try to make the 0.8 before the end of the year with a fix. Otherwise said, guile-wiredtiger extra abstraction context and environment are handy in single thread context because it avoids passing the context around. It's also handy in simple multithread settings where you don't need to pass environment around (to create a new session per thread). But it fails in the advanced use case of guile fibers. This won't trigger a crash under low traffic, but will fail under load and advanced use of guile wiredtiger, like multiple statments transactions. This can be mitigated by turning off preemption in guile-fibers but again it's not perfect solution. I failed to create an API that makes simple things simple and complex things possible. There might be an escape if fibers implemented fluids for fibers something like PEP 550 [2]. But anyway, it won't be perfect, so I will rework the current databases in guile wiredtiger to completly avoid the use of fluids and instead pass around database cnx and session. [2] https://www.python.org/dev/peps/pep-0550/ I started doing this in culturia [3] but it's far from being complete. Since I need to convert all databases grf3, feature-space and ix to that style. Also, it's a backward incompatible change. [3] https://github.com/a-guile-mind/culturia.one/commit/15328e53fcb51d43f14de2f9f21d0b309237969a At the end of the day, I don't think I want to maintain grf3, feature-space and ix inside guile-wiredtiger because they are much more than wiredtiger bindings. So I am pondering the fact that I will drop those databases abstraction from guile wiredtiger and focus on improving the core bindings (like actionable exceptions) and maybe improve bindings coverage. IMHO I think feature-space, ix and grf3 don't really belong in guile-wiredtiger. agree For the next release 0.8 of guile-wiredtiger, the abstractions grf3, feature-space and ix will be deprecated. And after for 0.9 release they will be moved to the example folder (or better, forked by other people to be maintained separatly). I really like the advanced abstractions. I'm not sure I'm knowledgeable enough to fork and improve them yet. This is not happening before I have a replacement for them. When I think replacement, I think about a database (server?) that is hopefully both easy to use (in the spirit of mongodb) but that is also ACID across documents (aka. that cares about your data). Ultimately I want to be able to deploy with something like: $ guix system disk-image gnupaste-system.scm Then boot it up in a VPS. That will be neat! Thanks! Kris I really don't want to rely on a database server. It would be easy enough to store pastes on the filesystem, maybe even with (guix store) or use git like tekuti and get the benefit of revision history with paste modifications. I don't like either the idea of the database server. That said without one, you have to code a secure API to allow third
Re: Introducing GNUPaste (and guile-wiredtiger future)
On Sun 17 Dec 2017 15:39, Amirouche Boubekkiwrites: > Basically, guile-wiredtiger is not compatible yet with fiber > in the general case, because fiber will spawn several threads > and several fibers in each thread (and I think that fibers > can be stolen by other threads but I am not sure). Note that it's possible to run fibers with only one kernel thread. See the docs. Also note that in fibers (and indeed in Guile threads), a newly spawned fiber (or thread) inherits the fluid values that were current when the thread was spawned. Fluid values in other fibers or threads are unaffected. Anyway I reply to offer some more general notes :) If what you need is sequential access to a database, you can arrange to access the database from a single fiber. That fiber can communicate with others via channels (for example). If the fiber migrates to another threads, that usually doesn't matter -- it's as if a kernel thread migrated to a different CPU. The memory model of Guile and fibers ensures that there will be no problems. You do end up having to route database requests to that fiber, usually via messages over channels, but that can be OK -- see https://blog.acolyer.org/2017/12/04/ffwd-delegation-is-much-faster-than-you-think/. Sometimes though you need real thread affinity between some external resource and a fiber. In that case the usual solution is to spawn a thread instead of a fiber, and access the resource only in that thread. You can still use channels to communicate between that thread and other fibers running on your system, if that's what you want. Cheers, Andy
Re: Introducing GNUPaste (and guile-wiredtiger future)
Amirouche Boubekkiwrites: > Héllo, > > On 2017-12-14 19:37, Kristofer Buffington wrote: >> Hello! >> >> I am excited to share GNUPaste! This is a really simple web app >> similar to paste.lisp.org built with Guile. I have a linode running it >> from git on GuixSD. >> >> https://paste.freshbakedyams.com (Please use it!) >> Source: https://github.com/kristoferbuffington/gnupaste.git >> >> Currently the frontend uses twitter bootstrap + jquery and highlightjs >> from a CDN. It really doesn't need all that boilerplate. It will >> definitely change in the future. GNUPaste depends on guile-wiredtiger >> and guile-fibers to compile. > > Thanks for considering guile-wiredtiger. > > Basically, guile-wiredtiger is not compatible yet with fiber > in the general case, because fiber will spawn several threads > and several fibers in each thread (and I think that fibers > can be stolen by other threads but I am not sure). > > The way wiredtiger works is that there is one > *connexion* (called *environment* in guile wiredtiger) > per database and one *session* per "thread" of execution > (called *context* in guile wiredtiger). > > I changed the naming because they are different from the > original things. Both environment and context are backed > by fluids. > > In a pre-fork thread model, one must use with-context [1] > after the fork to populate the current fuild with a specific > context. > > [1] > https://framagit.org/a-guile-mind/guile-wiredtiger/blob/master/wiredtiger/extra.scm#L328 > > The thing is that when using fibers, the thread of execution > is a fiber, hence simply said the wiredtiger extra abstraction > called context fails. Because context/session must not be > shared between different fibers even if they are executed in > the same thread. I have noticed that I get "Resource busy" issues if multiple requests are handled by the server simultaneously. I tried running (with-env ...) in the request-handler instead, but it is the same problem. Multiple environments can't open the database simultaneously either. > Otherwise said, guile-wiredtiger extra abstraction context > and environment are handy in single thread context because > it avoids passing the context around. It's also handy in > simple multithread settings where you don't need to pass > environment around (to create a new session per thread). > But it fails in the advanced use case of guile fibers. > > This won't trigger a crash under low traffic, but will fail > under load and advanced use of guile wiredtiger, like multiple > statments transactions. This can be mitigated by turning off > preemption in guile-fibers but again it's not perfect solution. > > I failed to create an API that makes simple things simple and > complex things possible. There might be an escape if fibers > implemented fluids for fibers something like PEP 550 [2]. > But anyway, it won't be perfect, so I will rework the current > databases in guile wiredtiger to completly avoid the use of > fluids and instead pass around database cnx and session. > > [2] https://www.python.org/dev/peps/pep-0550/ > > I started doing this in culturia [3] but it's far from being > complete. Since I need to convert all databases grf3, feature-space > and ix to that style. > > Also, it's a backward incompatible change. > > [3] > https://github.com/a-guile-mind/culturia.one/commit/15328e53fcb51d43f14de2f9f21d0b309237969a > > At the end of the day, I don't think I want to maintain grf3, > feature-space and ix inside guile-wiredtiger because they are > much more than wiredtiger bindings. So I am pondering the fact > that I will drop those databases abstraction from guile wiredtiger > and focus on improving the core bindings (like actionable exceptions) > and maybe improve bindings coverage. IMHO I think feature-space, ix and grf3 don't really belong in guile-wiredtiger. > For the next release 0.8 of guile-wiredtiger, the abstractions grf3, > feature-space and ix will be deprecated. And after for 0.9 release > they will be moved to the example folder (or better, forked by > other people to be maintained separatly). I really like the advanced abstractions. I'm not sure I'm knowledgeable enough to fork and improve them yet. >> Ultimately I want to be able to deploy with something like: >> >> $ guix system disk-image gnupaste-system.scm >> >> Then boot it up in a VPS. > > That will be neat! > >> >> Thanks! >> Kris I really don't want to rely on a database server. It would be easy enough to store pastes on the filesystem, maybe even with (guix store) or use git like tekuti and get the benefit of revision history with paste modifications. Thanks for the update! Kris
Re: Introducing GNUPaste
Am 14.12.2017 um 19:37 schrieb Kristofer Buffington: > Currently the frontend uses twitter bootstrap + jquery and highlightjs > from a CDN. If the default is to this stull, please consider *including* it into the software or as part of the installation procedure. Using CDN is obeying user privacy, since it allows these CDNs to track users. Thanks. -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |
Re: Introducing GNUPaste (and guile-wiredtiger future)
Héllo, On 2017-12-14 19:37, Kristofer Buffington wrote: Hello! I am excited to share GNUPaste! This is a really simple web app similar to paste.lisp.org built with Guile. I have a linode running it from git on GuixSD. https://paste.freshbakedyams.com (Please use it!) Source: https://github.com/kristoferbuffington/gnupaste.git Currently the frontend uses twitter bootstrap + jquery and highlightjs from a CDN. It really doesn't need all that boilerplate. It will definitely change in the future. GNUPaste depends on guile-wiredtiger and guile-fibers to compile. Thanks for considering guile-wiredtiger. Basically, guile-wiredtiger is not compatible yet with fiber in the general case, because fiber will spawn several threads and several fibers in each thread (and I think that fibers can be stolen by other threads but I am not sure). The way wiredtiger works is that there is one *connexion* (called *environment* in guile wiredtiger) per database and one *session* per "thread" of execution (called *context* in guile wiredtiger). I changed the naming because they are different from the original things. Both environment and context are backed by fluids. In a pre-fork thread model, one must use with-context [1] after the fork to populate the current fuild with a specific context. [1] https://framagit.org/a-guile-mind/guile-wiredtiger/blob/master/wiredtiger/extra.scm#L328 The thing is that when using fibers, the thread of execution is a fiber, hence simply said the wiredtiger extra abstraction called context fails. Because context/session must not be shared between different fibers even if they are executed in the same thread. Otherwise said, guile-wiredtiger extra abstraction context and environment are handy in single thread context because it avoids passing the context around. It's also handy in simple multithread settings where you don't need to pass environment around (to create a new session per thread). But it fails in the advanced use case of guile fibers. This won't trigger a crash under low traffic, but will fail under load and advanced use of guile wiredtiger, like multiple statments transactions. This can be mitigated by turning off preemption in guile-fibers but again it's not perfect solution. I failed to create an API that makes simple things simple and complex things possible. There might be an escape if fibers implemented fluids for fibers something like PEP 550 [2]. But anyway, it won't be perfect, so I will rework the current databases in guile wiredtiger to completly avoid the use of fluids and instead pass around database cnx and session. [2] https://www.python.org/dev/peps/pep-0550/ I started doing this in culturia [3] but it's far from being complete. Since I need to convert all databases grf3, feature-space and ix to that style. Also, it's a backward incompatible change. [3] https://github.com/a-guile-mind/culturia.one/commit/15328e53fcb51d43f14de2f9f21d0b309237969a At the end of the day, I don't think I want to maintain grf3, feature-space and ix inside guile-wiredtiger because they are much more than wiredtiger bindings. So I am pondering the fact that I will drop those databases abstraction from guile wiredtiger and focus on improving the core bindings (like actionable exceptions) and maybe improve bindings coverage. For the next release 0.8 of guile-wiredtiger, the abstractions grf3, feature-space and ix will be deprecated. And after for 0.9 release they will be moved to the example folder (or better, forked by other people to be maintained separatly). Ultimately I want to be able to deploy with something like: $ guix system disk-image gnupaste-system.scm Then boot it up in a VPS. That will be neat! Thanks! Kris -- Amirouche ~ amz3 ~ http://www.hyperdev.fr
Re: Introducing GNUPaste
Key, Kristofer: On Thu, Dec 14, 2017 at 13:37:30 -0500, Kristofer Buffington wrote: > I am excited to share GNUPaste! This is a really simple web app > similar to paste.lisp.org built with Guile. I have a linode running it > from git on GuixSD. Please reconsider the use of "GNU" in the program name, since this isn't in any way affiliated with the GNU Project. I did notice one of your commits is titled "Initial commit prepped for being part of the GNU project". If you do wish to offer your software to GNU, please see: https://www.gnu.org/help/evaluation.html We'd be happy to review it. -- Mike Gerwitz Free Software Hacker+Activist | GNU Maintainer & Volunteer GPG: D6E9 B930 028A 6C38 F43B 2388 FEF6 3574 5E6F 6D05 https://mikegerwitz.com signature.asc Description: PGP signature
Re: Introducing GNUPaste
Hi Kristofer, Kristofer Buffingtonskribis: > I am excited to share GNUPaste! This is a really simple web app > similar to paste.lisp.org built with Guile. I have a linode running it > from git on GuixSD. > > https://paste.freshbakedyams.com (Please use it!) > Source: https://github.com/kristoferbuffington/gnupaste.git Looks nice! I suppose you could use ‘guile-syntax-highlight’ to add some color into the mix. :-) > Ultimately I want to be able to deploy with something like: > > $ guix system disk-image gnupaste-system.scm > > Then boot it up in a VPS. That’d be great. Thanks for sharing! Ludo’.
Introducing GNUPaste
Hello! I am excited to share GNUPaste! This is a really simple web app similar to paste.lisp.org built with Guile. I have a linode running it from git on GuixSD. https://paste.freshbakedyams.com (Please use it!) Source: https://github.com/kristoferbuffington/gnupaste.git Currently the frontend uses twitter bootstrap + jquery and highlightjs from a CDN. It really doesn't need all that boilerplate. It will definitely change in the future. GNUPaste depends on guile-wiredtiger and guile-fibers to compile. Ultimately I want to be able to deploy with something like: $ guix system disk-image gnupaste-system.scm Then boot it up in a VPS. I need to fix some issues with autotools before it can be installed with make. My issue is creating a consistent test-env because configure substitutes @localstatedir@ (root/gnupaste/config.scm.in) with a path/to/wt that is not in the build tree. I'm planning to imitate guix.git/build-aux/test-env in that regard. Otherwise, I would really appreciate some feature suggestions, especially if it involves using pubstrate! Thanks! Kris