Re: [Chicken-users] using mmap files as strings?
From: Alan Post Subject: [Chicken-users] using mmap files as strings? Date: Thu, 21 Oct 2010 15:01:10 -0600 > I have the following file: > > <++> mmap.scm > (use posix) > (use lolevel) > > (let* ((fd (file-open "mmap.scm" (+ open/rdonly open/nonblock))) >(size (file-size fd)) >(mmap (map-file-to-memory #f size prot/read (+ map/file map/shared) > fd)) >(buf (memory-mapped-file-pointer mmap))) > (write fd) > (newline) > (write size) > (newline) > (write mmap) > (newline) > (write buf) > (newline) > (write (pointer-s8-ref buf)) > (newline) > (write (integer->char (pointer-s8-ref buf))) > (newline)) > <--> > > Which opens itself as an mmap and reads the first byte. The output > is: > > <++> csc -o mmap mmap.scm && ./mmap > 3 > 458 > # > # > 40 > #\( > <--> > > So far so good, that is what I would expect. I'd like to work with > an mmap buffer like a string. Is it possible to create an object > that will treat the mmap area as a string that I can run regular > string operations on without copying the mmap buffer? I can't think of a way in the moment. Strings require a header-word, and the data pointed to by a pointer doesn't have that header, so it isn't possible to "sneak in" a pointer as a replacement of a string. > I'm > specifically interested in running regular expressions across the > mmap space. irregex provides a "chunk" based API, perhaps it is possible to define chunks over the data represented by the mmap pointer. Alex or Peter might be able to tell more about this. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Installing data files for eggs
From: Jim Ursetto Subject: Re: [Chicken-users] Installing data files for eggs Date: Thu, 21 Oct 2010 17:38:44 -0500 > On Thu, Oct 21, 2010 at 01:11, Felix > wrote: > >> IIRC, debian only allows post-installation changes to VARDIR. Does this >> apply to DATADIR as well? Where do you (Peter, Jim) see this data files >> to be copied to on egg-installation? > > I have no idea about either VARDIR or DATADIR. I have just been > installing stuff > into directories under (chicken-home), which on my system is e.g. > "/usr/local/chicken-4/share/chicken". So chicken-doc stuff goes into > (chicken-home)/chicken-doc. That's not done by the installer of > course, but I'd like > to get chickadee installing stuff in the same area via the installer. > > I see Peter is using (installation-prefix), like (list > (installation-prefix) "share" "chicken" "slatex"). I am not sure if > (list (chicken-home) "slatex") is always correct but it seems to > produce the same results in this case. > Well, the question is whether it is allowed on systems that use a strict policy on what directories can be written to on installation (my terminology may be wrong). So on debian systems the egg repository ends up somewhere in /var. What I'd like to know whether the same policy applies to $PREFIX/share, or more precisely: should the data be installed there or in /var ? Installing in the normal repository path ($PREFIX/lib/chicken/$BINVERSION or $PREFIX/var/) is no big deal, and any extension can already install subdirectories by simple putting the directory name in the files-list in `install-extension'. cheers, felix ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] using mmap files as strings?
On Fri, Oct 22, 2010 at 06:20:01AM -0400, Felix wrote: > I can't think of a way in the moment. Strings require a header-word, > and the data pointed to by a pointer doesn't have that header, so it > isn't possible to "sneak in" a pointer as a replacement of a string. > > > I'm > > specifically interested in running regular expressions across the > > mmap space. > > irregex provides a "chunk" based API, perhaps it is possible to > define chunks over the data represented by the mmap pointer. > Alex or Peter might be able to tell more about this. The chunked API requires being able to extract strings from the mmapped memory. Chunks also only move forward, so you can't backtrack. If the SRE gets compiled to a backtracking matcher you probably want to include a few "commit"s because otherwise all chunks stick around in memory, I think. I'm not exactly sure how it handles that though. Cheers, Peter -- http://sjamaan.ath.cx -- "The process of preparing programs for a digital computer is especially attractive, not only because it can be economically and scientifically rewarding, but also because it can be an aesthetic experience much like composing poetry or music." -- Donald Knuth ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] Installing data files for eggs
On Fri, Oct 22, 2010 at 6:24 AM, Felix wrote: > Well, the question is whether it is allowed on systems that use a > strict policy on what directories can be written to on installation > (my terminology may be wrong). So on debian systems the egg repository > ends up somewhere in /var. What I'd like to know whether the same > policy applies to $PREFIX/share, or more precisely: should the data be > installed there or in /var ? Installing in the normal repository path > ($PREFIX/lib/chicken/$BINVERSION or $PREFIX/var/) is > no big deal, and any extension can already install subdirectories > by simple putting the directory name in the files-list in > `install-extension'. The constraint on share directories (or at least on /usr/share and /usr/local/share) is that files placed there must be read-only (in normal operation, obviously not during installation) and architecture-independent. For example, /usr/share/dict/words is the current system dictionary, which meets both requirements. Man pages also meet both requirements and go in /usr(/local)/share/man (for historical reasons there is a symlink at /usr/local/man to /usr/local/share/man). Preformatted man pages in plain text or HTML, being writable, are supposed to go in /var/cache/man, but not every distro does this. Stuff in /var is intended to be mutated in normal operation, like /var/log (syslogs) or /var/mail. In principle, if /usr and /var are different partitions, you can mount /usr read-only except when installing software (it can even be shared between homogeneous computers using NFS), whereas /var must be mounted read-write and is specific to a particular computer. ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] using mmap files as strings?
On Fri, Oct 22, 2010 at 06:20:01AM -0400, Felix wrote: > From: Alan Post > Subject: [Chicken-users] using mmap files as strings? > Date: Thu, 21 Oct 2010 15:01:10 -0600 > > > I have the following file: > > > > <++> mmap.scm > > (use posix) > > (use lolevel) > > > > (let* ((fd (file-open "mmap.scm" (+ open/rdonly open/nonblock))) > >(size (file-size fd)) > >(mmap (map-file-to-memory #f size prot/read (+ map/file map/shared) > > fd)) > >(buf (memory-mapped-file-pointer mmap))) > > (write fd) > > (newline) > > (write size) > > (newline) > > (write mmap) > > (newline) > > (write buf) > > (newline) > > (write (pointer-s8-ref buf)) > > (newline) > > (write (integer->char (pointer-s8-ref buf))) > > (newline)) > > <--> > > > > Which opens itself as an mmap and reads the first byte. The output > > is: > > > > <++> csc -o mmap mmap.scm && ./mmap > > 3 > > 458 > > # > > # > > 40 > > #\( > > <--> > > > > So far so good, that is what I would expect. I'd like to work with > > an mmap buffer like a string. Is it possible to create an object > > that will treat the mmap area as a string that I can run regular > > string operations on without copying the mmap buffer? > > I can't think of a way in the moment. Strings require a header-word, > and the data pointed to by a pointer doesn't have that header, so it > isn't possible to "sneak in" a pointer as a replacement of a string. > > > I'm > > specifically interested in running regular expressions across the > > mmap space. > > irregex provides a "chunk" based API, perhaps it is possible to > define chunks over the data represented by the mmap pointer. > Alex or Peter might be able to tell more about this. > This is excellent, I've learned a lot more about how all of this works together now. What method would I use to convert a pointer and a length to a string? Even though I'm copying the memory, how do I do it at all? I think I can accomplish what I have in mind by thunking to C, though in that case I'd need to create scheme strings of the results. Thank you! -Alan -- .i ko djuno fi le do sevzi ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
[Chicken-users] New directory during extension install
Hi, Probably better to use the 'setup-api' routine 'create-directory/ parents' instead of the 'posix' 'create-directory' since the former will obey "chicken-install -sudo". Best Wishes, Kon ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
[Chicken-users] Re: New directory during extension install
Oh, thank you. That's the first good explanation I got regarding the difference between (posix create-directory) and (chicken eggs create-directory/parents). That means I can't create directories without creating their parents then, right? Say out of some misguided safety precaution I don't want to accidentally create a large directory tree if I make a typo. On Fri, Oct 22, 2010 at 19:21, Kon Lovett wrote: > Hi, > > Probably better to use the 'setup-api' routine 'create-directory/parents' > instead of the 'posix' 'create-directory' since the former will obey > "chicken-install -sudo". > > Best Wishes, > Kon > > > ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] using mmap files as strings?
Peter Bex writes: > On Fri, Oct 22, 2010 at 06:20:01AM -0400, Felix wrote: >> >> > I'm specifically interested in running regular >> > expressions across the mmap space. >> >> irregex provides a "chunk" based API, perhaps it is possible to >> define chunks over the data represented by the mmap pointer. >> Alex or Peter might be able to tell more about this. > > The chunked API requires being able to extract strings from the mmapped > memory. Chunks also only move forward, so you can't backtrack. > If the SRE gets compiled to a backtracking matcher you probably > want to include a few "commit"s because otherwise all chunks stick > around in memory, I think. I'm not exactly sure how it handles that > though. Actually, currently even non-backtracking regexps will hold onto all chunks. The intention of the chunked API is to make it easy to work with non-consecutive strings, not strings that can't fit into memory. Specifically, I wanted to be able to handle any kind of text-buffer (gap-buffer, piece-table, etc.) easily. -- Alex ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
[Chicken-users] Re: New directory during extension install
On Oct 22, 2010, at 6:54 PM, Jim Ursetto wrote: Oh, thank you. That's the first good explanation I got regarding the difference between (posix create-directory) and (chicken eggs create-directory/parents). That means I can't create directories without creating their parents then, right? Say out of some misguided safety precaution I don't want to accidentally create a large directory tree if I make a typo. Yes, that is very real problem. Probably should be entered as a some sort of bug. On Fri, Oct 22, 2010 at 19:21, Kon Lovett wrote: Hi, Probably better to use the 'setup-api' routine 'create-directory/ parents' instead of the 'posix' 'create-directory' since the former will obey "chicken-install -sudo". Best Wishes, Kon Best Wishes, Kon ___ Chicken-users mailing list Chicken-users@nongnu.org http://lists.nongnu.org/mailman/listinfo/chicken-users
