On Mon, 24 Feb 2014 15:19:06 -0500, Hugo Florentino wrote: > On Mon, 24 Feb 2014 19:08:16 +0000 (UTC), Justin Whear wrote: >> >> Specifically std.regex.splitter[1] creates a lazy range over the input. >> You can couple this with lazy file reading (e.g. >> `File("mailbox").byChunk (1024).joiner`). >> >> > Would something like this work? (I cannot test it right now) > > auto themailbox = args[1]; > immutable uint chunksize = 1024 * 64; > static auto re = regex(`\n\nFrom .+@.+$`); > auto mailbox; > auto mail; > while (mailbox = File(themailbox).byChunk(chunksize).joiner) != EOF) { > mail = splitter(mailbox, re); > } > > If so, I have a couple of furter doubts: > > Using splitter actually removes the expression from the string, how > could I reinsert it to the beginning of each resulting string in an > efficient way (i.e. avoiding copying something which is already loaded > in memory)? > > I am seeing the splitter fuction returns a struct, how could I > progressively dump to disk each resulting string, removing it from the > struct, so that so that it does not end up having the full mailbox > loaded into memory, in this case as a struct? > > Regards, Hugo
The code you've posted won't work, primarily because you don't need to loop over the file-reading range, nor will it ever "return" EOF. Also, if you don't actually want to remove the regex matches, you can just use the matchAll function. Here's some _untested_ sample code to set you on the right track. -------------------------------------------------------- import std.algorithm, std.range, std.stdio, std.regex; void main(string[] args) { auto mailboxPath = args[1]; immutable size_t chunksize = 1024 * 64; auto re = regex(`\n\nFrom .+@.+$`); // you might want to try using ctRegex auto mailStarts = File(mailboxPath).byChunk(chunksize).joiner .matchAll(re); } -------------------------------------------------------- This code won't actually do any work--no data will be loaded from the file (caveat: the first chunk might be prefetched, not sure), no matches will actually be performed. If you use `take(10)` on the mailStarts variable, the code will load only as much of the file (down to the granularity of chunksize) as is needed to find the first 10 instances of the regular expression. The regex matches will not copy, but rather provide slices over the data that is in memory. And, thinking about this further, you don't want to use my code either-- partly because byChunk reuses its buffer, partly because the functions in std.regex provide slices over the input data. I think what you'll want to do is: load the data from File chunk-by-chunk lazily, scan each chunk with the regex, if you don't find a match, copy that data into an "overlap" buffer and repeat, if you do find a match then the contents of the overlap buffer + the slice up to the current match is one mail, rinse and repeat. You should be able to encapsulate all of this in clean, lazy range, but I don't have the time right now to work out if it can be done by simply compositing existing functions from Phobos. Justin