pause garbage collection in parallel code
Dear all, I have a parallel program, using std.parallelism (awesome!), but I recently noticed that I achieve very poor performance on many CPUs, and I identified the Garbage Collector to be the main cause of this. Because I have quite heavy memory usage, the Garbage collector interrupts all multi-threading while it runs, which reduces the total runtime of my program dramatically. This is so bad that I actually achieve poorer performance running on 20 cores than on 4 cores. I see several ways how to improve my code: 1.) Is there a way to tell the GC the maximum heap size allowed before it initiates a collection cycle? Cranking that up would cause fewer collection cycles and hence spend more time in my multithreaded code? 2.) Is there a way to pause the GC collection for the parallel part of my program, deliberately accepting higher memory usage? 3.) Most of the memory is used in one huge array, perhaps I should simply use malloc and free for that particular array to avoid the GC from running so often. Certainly, Option 1 and 2 are noninvasive, so preferred. Are the other ways? I am a bit surprised that there is no command line option for dmd to control GC maximum heap size. Who determines how often the GC is run? For example, in Haskell I can simply set the maximum heap size to 10Mb in GHC using -A10m, which I used in the past to help exactly the same problem and dramatically reduce the frequency of GC collection cycles. Thanks for help! Stephan
Re: pause garbage collection in parallel code
On Monday, 15 December 2014 at 11:54:44 UTC, Daniel Murphy wrote: Stephan Schiffels wrote in message news:wjeozpnitvhtxrkhu...@forum.dlang.org... I see several ways how to improve my code: 1.) Is there a way to tell the GC the maximum heap size allowed before it initiates a collection cycle? Cranking that up would cause fewer collection cycles and hence spend more time in my multithreaded code? Yes, sort of. You can use http://dlang.org/phobos/core_memory.html#.GC.reserve to have the GC grab a big chunk of memory, and collections won't run until that is exhausted. As it is only allocating virtual memory, it should be more or less equivalent to setting the max heap size. 2.) Is there a way to pause the GC collection for the parallel part of my program, deliberately accepting higher memory usage? Yes, with GC.enable and GC.diable 3.) Most of the memory is used in one huge array, perhaps I should simply use malloc and free for that particular array to avoid the GC from running so often. Yes, this may work if it's a reasonable design for your application. Other options like re-using one GC buffer per thread might work too. I am a bit surprised that there is no command line option for dmd to control GC maximum heap size. Who determines how often the GC is run? For example, in Haskell I can simply set the maximum heap size to 10Mb in GHC using -A10m, which I used in the past to help exactly the same problem and dramatically reduce the frequency of GC collection cycles. Being able to configure the GC at run-time is something that's currently being worked on. I assume this will be possible rather soon, probably in the next release. Excellent, thanks everyone, problem solved. I use GC.disable and GC.enable for now, works like a charm. I new about these functions, but I thought they prevent GC-allocation, not just collection. Using ThreadLocal storage with std.parallelism is also interesting, I won't need it now, as the memory is still within manageable bounds, but certainly an option to reduce the memory footprint... nice! And yes, I saw GC.reserve just after I wrote this thread, it seems to do what I wanted. Stephan
Re: pause garbage collection in parallel code
On Monday, 15 December 2014 at 11:54:44 UTC, Daniel Murphy wrote: Stephan Schiffels wrote in message news:wjeozpnitvhtxrkhu...@forum.dlang.org... I see several ways how to improve my code: 1.) Is there a way to tell the GC the maximum heap size allowed before it initiates a collection cycle? Cranking that up would cause fewer collection cycles and hence spend more time in my multithreaded code? Yes, sort of. You can use http://dlang.org/phobos/core_memory.html#.GC.reserve to have the GC grab a big chunk of memory, and collections won't run until that is exhausted. As it is only allocating virtual memory, it should be more or less equivalent to setting the max heap size. This doesn't work for me, for some reason. I reserve via GC.reserve(4_000_000_000), ensured that it does return at least that amount, but the Garbage collector will still collect like crazy, long before that reserved memory is exhausted...
stack trace output on exception
Hi, I am using dmd with version: DMD64 D Compiler v2.065-devel-db2a73d My program throws a custom exception with a custom error message at some point. The stack trace (below) is very uninformative. Is there a way to output the function names of each position in the stack? I already compile using -g and I also tried -gc and -gs and all three of them together. I also tried -debug. The stack trace looks always like this: popGenFunc.IllegalParametersException@popGenFunc.d(556): gamma out of range /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x42752b] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x423ee7] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x404bab] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x402d7e] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x4028d6] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433cc4] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433c1e] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433c84] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433c1e] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433b9f] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x421121] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7ffe8440e76d] Thanks for your help, Stephan
Re: stack trace output on exception
Ah nice. That worked. Thanks! 2014-07-08 9:25 GMT+02:00 JR via Digitalmars-d-learn digitalmars-d-learn@puremagic.com: On Tuesday, 8 July 2014 at 07:11:26 UTC, Stephan Schiffels wrote: Hi, I am using dmd with version: DMD64 D Compiler v2.065-devel-db2a73d My program throws a custom exception with a custom error message at some point. The stack trace (below) is very uninformative. Is there a way to output the function names of each position in the stack? I already compile using -g and I also tried -gc and -gs and all three of them together. I also tried -debug. The stack trace looks always like this: popGenFunc.IllegalParametersException@popGenFunc.d(556): gamma out of range /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x42752b] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x423ee7] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x404bab] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x402d7e] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x4028d6] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433cc4] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433c1e] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433c84] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433c1e] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x433b9f] /nfs/users/nfs_s/ss27/Projects/hfit/build/hfit() [0x421121] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7ffe8440e76d] Thanks for your help, Stephan Tried building with -L--export-dynamic?
Re: Read and write gzip files easily.
On Wednesday, 19 February 2014 at 15:51:53 UTC, Kamil Slowikowski
wrote:
Hi there, I'm new to D and have a lot of learning ahead of me.
It would
be extremely helpful to me if someone with D experience could
show me
some code examples.
I'd like to neatly read and write gzipped files for my work. I
have read
several threads on these forums on the topic of std.zlib or
std.zip and I haven't been able to figure it out.
Hi Kamil,
I am glad someone has the exact same problem as I had. I actually
solved this, inspired by the python API you quoted above. I wrote
these classes:
GzipInputRange, GzipByLine, and GzipOut.
Here is how I can now use them:
_
import gzip;
import std.stdio;
void main() {
auto byLine = new GzipByLine(test.gz);
foreach(line; byLine)
writeln(line);
auto gzipOutFile = new GzipOut(testout.gz);
gzipOutFile.compress(bla bla bla);
gzipOutFile.finish();
}
That is all quite convenient and I was wondering whether
something like that would be useful even in Phobos. But it's
clear that for phobos things would involve a lot more work to
comply with the requirements. This so far simply served my needs
and is not as generic as it could be:
Here is the code:
___gzip.d__
import std.zlib;
import std.stdio;
import std.range;
import std.traits;
class GzipInputRange {
UnCompress uncompressObj;
File f;
auto CHUNKSIZE = 0x4000;
ReturnType!(f.byChunk) chunkRange;
bool exhausted;
char[] uncompressedBuffer;
size_t bufferIndex;
this(string filename) {
f = File(filename, r);
chunkRange = f.byChunk(CHUNKSIZE);
uncompressObj = new UnCompress();
load();
}
void load() {
if(!chunkRange.empty) {
auto raw = chunkRange.front.dup;
chunkRange.popFront();
uncompressedBuffer =
cast(char[])uncompressObj.uncompress(raw);
bufferIndex = 0;
}
else {
if(!exhausted) {
uncompressedBuffer = cast(char[])uncompressObj.flush();
exhausted = true;
bufferIndex = 0;
}
else
uncompressedBuffer.length = 0;
}
}
@property char front() {
return uncompressedBuffer[bufferIndex];
}
void popFront() {
bufferIndex += 1;
if(bufferIndex = uncompressedBuffer.length) {
load();
bufferIndex = 0;
}
}
@property bool empty() {
return uncompressedBuffer.length == 0;
}
}
class GzipByLine {
GzipInputRange range;
char[] buf;
this(string filename) {
this.range = new GzipInputRange(filename);
popFront();
}
@property bool empty() {
return buf.length == 0;
}
void popFront() {
buf.length = 0;
while(!range.empty range.front != '\n') {
buf ~= range.front;
range.popFront();
}
range.popFront();
}
string front() {
return buf.idup;
}
}
class GzipOut {
Compress compressObj;
File f;
this(string filename) {
f = File(filename, w);
compressObj = new Compress(HeaderFormat.gzip);
}
void compress(string s) {
auto compressed = compressObj.compress(s.dup);
f.rawWrite(compressed);
}
void finish() {
auto compressed = compressObj.flush();
f.rawWrite(compressed);
}
}
Re: Read and write gzip files easily.
On Thursday, 20 February 2014 at 17:05:37 UTC, Kamil Slowikowski wrote: On Thursday, 20 February 2014 at 10:35:50 UTC, Stephan Schiffels wrote: Hi Kamil, I am glad someone has the exact same problem as I had. I actually solved this, inspired by the python API you quoted above. I wrote these classes: GzipInputRange, GzipByLine, and GzipOut. Stephan, awesome! Thank you very much for sharing your classes. It's nice to see how you've approached this problem. Your code is very clear and easy to understand (for me). Also, I now see the error in my code: I believe I should use rawWrite to write compressed data and not writeExact. You're welcome. If you manage to put GzipOut.finish() into the destructor of the class to automatically flush the file upon destruction of the object, let me know. I tried this and it gives a SegFault… I was too lazy to try to understand it but I am sure it must be in principle possible. Stephan
Re: std.range.chunk without length
On Thursday, 13 February 2014 at 17:41:37 UTC, monarch_dodra wrote: On Thursday, 13 February 2014 at 14:45:44 UTC, bearophile wrote: Stephan Schiffels: It would be actually easy to implement chunks without the save function, by using an internal buffer, which would however make this algorithm's memory burden linear in the chunk size. Would that be acceptable? I think it's acceptable. But perhaps you need to add one more optional argument for the buffer :-) Bye, bearophile Users andralex: https://github.com/D-Programming-Language/phobos/pull/1186 And quickfur: https://github.com/D-Programming-Language/phobos/pull/1453 Have submitted different algorithms for a similar problem: Basically, bu being 2-dimensional lazy (each subrange is itself a lazy range). However, both come with their own pitfalls. Andrei's still requires forward ranges. quickfur's doesn't, and, arguably, has a simpler design. However, if I remember correctly, it is also less efficient (it does double work). Implementing Quickfur's solution in Chunks for input ranges only could be a good idea. It *is* extra work, more code, more code to cover (that is difficult to cover). I'm not sure we have the man power to support such complexity: I was able to make chunks work with forward ranges, but I still haven't even fixed Splitter yet! I think that should take precedence. Yeah, nevermind, I won't do it. I realised that you had good reasons to require a ForwardRange. Chunking really needs some sort of save implemented. And what I had in mind to make it work on File.byLine with a buffer is actually a hack that effectively adds save functionality to the InputRange… so I agree it's logically not reasonable to do it here. Thanks anyway. Stephan
Re: std.range.chunk without length
On Thursday, 31 October 2013 at 10:35:54 UTC, Stephan Schiffels
wrote:
On Wednesday, 30 October 2013 at 20:43:54 UTC, qznc wrote:
On Wednesday, 30 October 2013 at 00:20:12 UTC, Stephan
Schiffels wrote:
Hi,
I'd like a version of std.range.chunk that does not require
the range to have the length property.
As an example, consider a file that you would like parse by
lines and always lump together four lines, i.e.
import std.stdio;
void main() {
auto range = File(test.txt, r).byLine();
foreach(c; range.chunks(4)) { //doesn't compile
writefln(%s %s, c[0], c[1]);
}
}
Your wish was granted. Monarchdodra was sent back in time [0],
so it is already fixed in HEAD. You could try the dmd beta [1].
[0] https://github.com/D-Programming-Language/phobos/pull/992
[1]
http://forum.dlang.org/thread/526dd8c5.2040...@digitalmars.com
Ah, awesome! Should have updated my github clone then.
Thanks,
Stephan
Sorry for the late follow up, but it turns out that
std.range.chunks needs a ForwardRange, and hence does not work on
File.byLine(). The referenced pull request claims that it does in
the comments, but of course the current implementation needs a
save() function which doesn't exist for the byLine range.
It would be actually easy to implement chunks without the save
function, by using an internal buffer, which would however make
this algorithm's memory burden linear in the chunk size. Would
that be acceptable? If so, I'd be happy to make that change and
push it.
Stephan
Re: std.range.chunk without length
On Wednesday, 30 October 2013 at 20:43:54 UTC, qznc wrote:
On Wednesday, 30 October 2013 at 00:20:12 UTC, Stephan
Schiffels wrote:
Hi,
I'd like a version of std.range.chunk that does not require
the range to have the length property.
As an example, consider a file that you would like parse by
lines and always lump together four lines, i.e.
import std.stdio;
void main() {
auto range = File(test.txt, r).byLine();
foreach(c; range.chunks(4)) { //doesn't compile
writefln(%s %s, c[0], c[1]);
}
}
Your wish was granted. Monarchdodra was sent back in time [0],
so it is already fixed in HEAD. You could try the dmd beta [1].
[0] https://github.com/D-Programming-Language/phobos/pull/992
[1]
http://forum.dlang.org/thread/526dd8c5.2040...@digitalmars.com
Ah, awesome! Should have updated my github clone then.
Thanks,
Stephan
std.range.chunk without length
Hi,
I'd like a version of std.range.chunk that does not require the
range to have the length property.
As an example, consider a file that you would like parse by lines
and always lump together four lines, i.e.
import std.stdio;
void main() {
auto range = File(test.txt, r).byLine();
foreach(c; range.chunks(4)) { //doesn't compile
writefln(%s %s, c[0], c[1]);
}
}
Thanks,
Stephan
Re: Why I chose D over Ada and Eiffel
Nice! I cannot anymore go through all the over 100 replies to this, sorry if someone else has suggested this: You should write this article (tidied up a bit) in a blog or somewhere more public on the web! Here in this forum, things are not as public as they could be! But thanks for sharing anyway! Stephan On Monday, 19 August 2013 at 20:18:06 UTC, Ramon wrote: Sorry, this is a long and big post. But then, so too is my way that led me here; long, big, troublesome. And I thought that my (probably not everyday) set of needs and experiences might be interesting or useful for some others, too. And, of course, I confess it, I just feel like throwing a very big THANK YOU at D's creators and makers. Thank you!
this() immutable
Hi,
I have some problems with adopting my code to a breaking change
introduced in version 2.063. Apparently, now it's not anymore
possible to instantiate an immutable object via:
auto object = new immutable(SomeClass)(contructor_args...);
without also defining either
this(constructor_args...) immutable {...}
or
this(constructor_args...) pure {...}
As much as I think that this change will make the system more
consistent, I don't know how to fix my code. Often, in a
constructor you pass other objects (via constructor dependency
injection), and it seems that I have to fix a hell of a lot of
dependency code that way. Is there a simple way of emulating the
way it just magically worked in version 2.062?
For example, is there a way of instantiating an object normally
(i.e. mutable), and then later freeze it to immutable via a
simple cast or so?
Thanks,
Stephan
Re: this() immutable
On Thursday, 13 June 2013 at 12:29:57 UTC, Simen Kjaeraas wrote: On Thu, 13 Jun 2013 14:17:22 +0200, Stephan Schiffels stephan_schiff...@mac.com wrote: For example, is there a way of instantiating an object normally (i.e. mutable), and then later freeze it to immutable via a simple cast or so? In std.exception there is assumeUnique. It's basically just a cast, but might be good enough for you. I see, will look at that. I actually need it immutable since I am using it across multiple threads.
Re: builtin sort
On Saturday, 8 June 2013 at 22:51:06 UTC, bearophile wrote: Peter Williams: Rather than deprecate it why not fix it? Shouldn't have to import std.algorithm just to sort an array. Generally you want to keep the compiler (all its layers) as simpler as possible, to make it simpler to compile, debug and develop. A sort is implementable very well in library code. Other things, like tuples with a good syntax maybe can't be implemented well in library code, so they should be in the compiler :) I suggest to kill the built-in sort ASAP. Bye, bearophile I agree. Do people have the same opinion on the builtin reverse? I don't remember whether there was a discussion about this. I suggest to kill that as well. sort and reverse are perfectly well implemented in the standard library rather than builtin. Is anyone actually on this? I could try to dig into it, I guess I could start looking for spurious places in phobos and druntime where these builtin functions are used and replace them with the library ones. If we deprecate it in the end we don't want to see it being used anywhere in the standard implementations. Stephan
Re: builtin sort
On Monday, 10 June 2013 at 11:10:06 UTC, Jacob Carlborg wrote: On 2013-06-10 11:03, Stephan Schiffels wrote: I agree. Do people have the same opinion on the builtin reverse? I don't remember whether there was a discussion about this. I suggest to kill that as well. sort and reverse are perfectly well implemented in the standard library rather than builtin. reverse is implemented with the stupid name retro. That's not correct, it's called reverse and is a builtin property of dynamic arrays, see bearophiles answer... retro is lazy! Is anyone actually on this? I could try to dig into it, I guess I could start looking for spurious places in phobos and druntime where these builtin functions are used and replace them with the library ones. If we deprecate it in the end we don't want to see it being used anywhere in the standard implementations. Perhaps start with modifying the compiler to indicate the sort and reverse functions are deprecated. Then it will be easier to find in Phobos and druntime. Also, if used in druntime they need to be reimplemented there. Right, I thought so, too. Indeed, bearophiles issue addresses this in this order. I will try this on a local branch first and possibly open a pull request to start a more concrete discussion on this... Stephan
builtin sort
Hi,
I know it has been discussed previously to deprecate the builtin
sort. I don't know the status of this, but I observed the
following problem.
With dmd, druntime and phobos all on 2.063, this program runs
successfully on a mac:
#!/usr/bin/env rdmd
import std.stdio;
void main() {
int[] a = [5, 4, 3, 2, 1];
a.sort;
writeln(a);
}
But it fails on linux, with the line:
/nfs/users/nfs_s/ss27/Software/dlang/phobos/generated/linux/release/64/libphobos2.a(qsort_4c4_2cc.o):
In function `_adSort':
src/rt/qsort.d:(.text._adSort+0x47): undefined reference to
`qsort_r'
collect2: ld returned 1 exit status
--- errorlevel 1
When I change a.sort - a.sort() and add import std.algorithm
everything works fine.
Does this mean that the builtin sort on Linux is broken with
2.063?
Stephan
Re: builtin sort
On Saturday, 8 June 2013 at 08:52:22 UTC, Jonathan M Davis wrote: Hmm, it works just fine on my system (64-bit Linux), so I don't know why you're seeing the problem that you're seeing. Hmm, that's bizarre, but I guess there's no need to understand this further since things work fine with std.algorithm.sort. However, we really, really need to deprecate the built-in sort - especially when a pair of parens can make the difference between whether you call the built-in sort or std.algorithm's sort - and it's particularly broken with regards to Unicode. Yes, I completely agree! Stephan
Re: builtin sort
On Saturday, 8 June 2013 at 09:16:23 UTC, monarch_dodra wrote:
On Saturday, 8 June 2013 at 08:52:22 UTC, Jonathan M Davis
wrote:
However, we really, really need to deprecate the built-in sort
- especially
when a pair of parens can make the difference between whether
you call the
built-in sort or std.algorithm's sort - and it's particularly
broken with
regards to Unicode.
- Jonathan M Davis
Or anything that uses non-POD comparison (opCmp) for that
matter:
//
import std.stdio;
import std.algorithm;
struct S
{
int i;
int opCmp(S o)
{
return (i o.i) - (i o.i); //inverse order
}
}
void main()
{
auto a = [S(1), S(2), S(3)];
writeln(a); //[S(1), S(2), S(3)]
a.sort;
writeln(a); //[S(1), S(2), S(3)]
a.sort();
writeln(a); //[S(3), S(2), S(1)]
}
//
I had pretty much forgotten .sort even existed (it really
never even crossed my mind that it could be built-in), and
given the push for optional parenthesis in general (especially
in UFCS call chains), I can say I am surprised by this behavior.
Wow, that's much worse than in my example... indeed very
unexpected behaviour.
Re: What happened to next DConf 2013 talk: Shared libraries in D by Martin Nowak?
On Wednesday, 29 May 2013 at 15:33:36 UTC, Ettienne Gilbert wrote: BTW, thanks for all the hard work in getting it all out! Rgds +1
Re: any and all not in the doc-index
On Tuesday, 28 May 2013 at 22:31:13 UTC, Andrei Alexandrescu wrote: On 5/28/13 6:19 PM, Stephan Schiffels wrote: Hi, is there a reason why any and all from std.algorithm are neither linked in the function index nor in the cheat sheet list at the top of std.algorithm's documentation page? I am happy to change this and make a pull request, just weren't sure whether it's on purpose. Stephan It's an omission, thanks for offering to make a pull request. Go for it! Andrei https://github.com/D-Programming-Language/phobos/pull/1320
any and all not in the doc-index
Hi, is there a reason why any and all from std.algorithm are neither linked in the function index nor in the cheat sheet list at the top of std.algorithm's documentation page? I am happy to change this and make a pull request, just weren't sure whether it's on purpose. Stephan
open a range of files - segfault
Hi,
this code crashes with a segfault. I need help to understand what
might be wrong with it.
import std.array;
import std.algorithm;
void main() {
auto names = [file1.txt, file2.txt, file3.txt]; // let
these files exist
auto files = names.map!(f = File(f, r))().array();
}
Thanks,
Stephan
Re: open a range of files - segfault
On Wednesday, 22 May 2013 at 11:09:49 UTC, John Colvin wrote:
On Wednesday, 22 May 2013 at 11:07:39 UTC, bearophile wrote:
Stephan Schiffels:
this code crashes with a segfault. I need help to understand
what might be wrong with it.
import std.array;
import std.algorithm;
void main() {
auto names = [file1.txt, file2.txt, file3.txt]; // let
these files exist
auto files = names.map!(f = File(f, r))().array();
}
If I use the latest 2.063beta5 compiler, and I also import
std.stdio, then I see no crash on Windows32.
Bye,
bearophile
With git master on linux x64 I get a segfault, see my other
comment.
OK, thanks for confirming. I updated to the latest git master and
still get a segfault. I'll file a bug report.
Stephan
Range for files by character
Hi,
I need an Input Range that iterates a file character by
character. In bioinformatics this is often important, and having
a D-range is of course preferable than any foreach-byLine
combination, since we can apply filters and other goodies from
std.algorithm. In this implementation, I am simply filtering out
new-lines, as an example.
import std.stdio;
import std.conv;
import std.algorithm;
void main() {
auto f = File(someFile.txt, r);
foreach(c; f.byChunk(1).filter!(a = to!char(a[0]) != '\n'))
write(to!char(c[0]));
}
Is this the right way to do it? I was a bit surprised that
std.stdio doesn't provide a byChar or byByte range. Is there
a reason for this, or is this a too special need?
Stephan
Re: Range for files by character
On Monday, 20 May 2013 at 21:40:51 UTC, w0rp wrote:
On Monday, 20 May 2013 at 21:36:41 UTC, Stephan Schiffels wrote:
Hi,
I need an Input Range that iterates a file character by
character. In bioinformatics this is often important, and
having a D-range is of course preferable than any
foreach-byLine combination, since we can apply filters and
other goodies from std.algorithm. In this implementation, I am
simply filtering out new-lines, as an example.
import std.stdio;
import std.conv;
import std.algorithm;
void main() {
auto f = File(someFile.txt, r);
foreach(c; f.byChunk(1).filter!(a = to!char(a[0]) != '\n'))
write(to!char(c[0]));
}
Is this the right way to do it? I was a bit surprised that
std.stdio doesn't provide a byChar or byByte range. Is
there a reason for this, or is this a too special need?
Stephan
I would try f.byChunk(n).joiner. joiner is from std.algorithm
and it produces a range which joins a range of ranges, quite
like your typical array to string join function.
Ah, wonderful. That's exactly what I needed. I think that pretty
much does what Jonathan suggested under the hood. I can also use
byLine then, indeed...
Thanks.
Are people using textmate for D programming?
Hi everyone, I would like to get an idea how many people are using Textmate as editor in Mac OS X. I am keen to improve the current bundle. Especially the syntax highlighting seems to be stuck somewhere in D1 or so... also, it'd be great to have some command to run unittests and code-coverage... I can just proceed and suggest improvements (the bundle is on github), but learning about other people using it would increase my motivation :-) Stephan
Re: Are people using textmate for D programming?
thanks for sharing. I am using TM2 (https://github.com/textmate), and I think it might make a difference. Anyway, I think I will try to reanimate the github bundle. You could think about committing your changes to it. Stephan On Saturday, 18 May 2013 at 13:22:11 UTC, Jacob Carlborg wrote: On 2013-05-18 13:41, Stephan Schiffels wrote: Hi everyone, I would like to get an idea how many people are using Textmate as editor in Mac OS X. I am keen to improve the current bundle. Especially the syntax highlighting seems to be stuck somewhere in D1 or so... also, it'd be great to have some command to run unittests and code-coverage... I can just proceed and suggest improvements (the bundle is on github), but learning about other people using it would increase my motivation :-) Stephan I'm using it. I have made some improvements to the bundle as well. I have updated the build and run commands. The build command will look for a shell script in the project with the name build.sh. The run command will do the same but look for run.sh. If no run.sh file exists it will run RDMD on the current file. Both commands will properly handle compile errors and runtime exceptions by creating links back to the source code. Unfortunately I never made the changes under source control. But I put the whole bundle here: https://dl.dropboxusercontent.com/u/18386187/D.tmbundle.zip Note, this bundle is for TextMate 1.5, if there's any difference.
Re: Are people using textmate for D programming?
On Saturday, 18 May 2013 at 16:16:50 UTC, TommiT wrote: On Saturday, 18 May 2013 at 11:41:22 UTC, Stephan Schiffels wrote: Hi everyone, I would like to get an idea how many people are using Textmate as editor in Mac OS X. I am keen to improve the current bundle. Especially the syntax highlighting seems to be stuck somewhere in D1 or so... also, it'd be great to have some command to run unittests and code-coverage... I can just proceed and suggest improvements (the bundle is on github), but learning about other people using it would increase my motivation :-) Stephan I've started using TextMate recently. I added some code snippets, but don't really know anything about the bundle stuff. Fumbling in the dark I managed to change the bundle indentation rules to make switch cases indent to the switch column, and I added some missing keywords for syntax highlighting though. Any improvements to the bundle highly appreciated. Ah that's cool. Why don't you fork the repo and send a pull request with these changes? It's pretty much these small changes (keywords, indentation) that I had in mind... they add up over time. Stephan
dlang/tools compilation failed?
Hi, is it just me or does changed.d need to be fixed? I get these compiler errors (I use dmd, druntime and phobos from github and up to date). $ make MODEL=64 -f posix.mak dmd -m64 rdmd.d dmd -m64 ddemangle.d dmd -m64 catdoc.d dmd -m64 detab.d dmd -m64 tolf.d dmd -c changed.d changed.d(132): Error: cannot append type string to type Appender!(string) changed.d(133): Error: cannot append type string to type Appender!(string) changed.d(135): Error: cannot append type string to type Appender!(string) changed.d(136): Error: cannot append type string to type Appender!(string) changed.d(138): Error: cannot append type string to type Appender!(string) changed.d(146): Error: cannot append type string to type Appender!(string) changed.d(148): Error: cannot append type string to type Appender!(string) changed.d(151): Error: cannot append type string to type Appender!(string) changed.d(154): Error: cannot append type string to type Appender!(string) changed.d(155): Error: cannot append type string to type Appender!(string) make: *** [changed] Error 1
Re: dlang/tools compilation failed?
Oh man, my git skills are so mediocre that I thought remote update does the job (I am a mercurial user , should really switch at some point). Thanks for clarifying, now it compiles. Stephan On Thursday, 14 March 2013 at 10:37:22 UTC, Jonathan M Davis wrote: On Thursday, March 14, 2013 11:06:47 Stephan Schiffels wrote: Hi, is it just me or does changed.d need to be fixed? I get these compiler errors (I use dmd, druntime and phobos from github and up to date). $ make MODEL=64 -f posix.mak dmd -m64 rdmd.d dmd -m64 ddemangle.d dmd -m64 catdoc.d dmd -m64 detab.d dmd -m64 tolf.d dmd -c changed.d changed.d(132): Error: cannot append type string to type Appender!(string) changed.d(133): Error: cannot append type string to type Appender!(string) changed.d(135): Error: cannot append type string to type Appender!(string) changed.d(136): Error: cannot append type string to type Appender!(string) changed.d(138): Error: cannot append type string to type Appender!(string) changed.d(146): Error: cannot append type string to type Appender!(string) changed.d(148): Error: cannot append type string to type Appender!(string) changed.d(151): Error: cannot append type string to type Appender!(string) changed.d(154): Error: cannot append type string to type Appender!(string) changed.d(155): Error: cannot append type string to type Appender!(string) make: *** [changed] Error 1 Are you building with the latest dmd, druntime, and Phobos from master on github? My guess would be that you're not and that that's the problem. - Jonathan M Davis
joiner and map, strange behavior
Hi,
I am struggling with understanding this behavior. In the code
below, the function getVec is called 8 times, but it should be
called only 4 times (once for each call inside of map).
Any explanations?
Stephan
import std.stdio;
import std.algorithm;
import std.array;
int[] getVec(size_t i) {
writeln(getVec is called);
auto vals = [
[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16]
];
return vals[i];
}
void main() {
auto result = [0, 1, 2, 3].map!(getVec).joiner.array;
writeln(result);
}
Re: joiner and map, strange behavior
On Tuesday, 12 March 2013 at 17:43:43 UTC, bearophile wrote:
Stephan Schiffels:
I am struggling with understanding this behavior. In the code
below, the function getVec is called 8 times, but it should
be called only 4 times (once for each call inside of map).
Any explanations?
Maybe it's a matter of calling front() more times, as in filter
in a recent thread. To be sure take a look inside phobos
sources.
By the way, your code is quite inefficient, because it
allocates a new matrix at each call to getVec.
Bye,
bearophile
Thanks, I had a brief look at std.algorithm.joiner but couldn't
find anything obvious, maybe I should look deeper into it.
And yes, the example is quite bad. This version of getVec would
have been better to illustrate:
import std.range;
auto getVec(size_t i) {
writeln(getVec is called);
return iota(i * 4, (i + 1) * 4);
}
Stephan
Re: joiner and map, strange behavior
Am 12.03.13 20:22, schrieb Timon Gehr:
On 03/12/2013 06:51 PM, Stephan Schiffels wrote:
...
Thanks, I had a brief look at std.algorithm.joiner but couldn't find
anything obvious, maybe I should look deeper into it.
...
I guess it is because of the following:
Eg (similar code occurs two times):
...
if (_sep.empty)
{
// Advance to the next range in the
// input
//_items.popFront();
for (;; _items.popFront())
{
if (_items.empty) return;
if (!_items.front.empty) break; // front called
}
_current = _items.front; // front called again
_items.popFront();
}
...
The code should only call front once for the first non-empty range.
Presumed example fix (though the implementation seems rather clumsy and
should probably be replaced completely):
...
if (_sep.empty)
{
// Advance to the next range in the
// input
//_items.popFront();
for (;; _items.popFront())
{
if (_items.empty) return;
_current = _items.front;
if (!_current.empty) break;
}
_items.popFront();
}
...
Thanks, that's very clear now. I stumbled over it because the function I
call loads a lot of data and a factor 2 makes all the difference in
runtime. Maybe I should file a bug report or at least a request to
improve this.
Stephan
Re: Best way to handle settings files (ini file syntax or similar)
Am 27.02.13 21:36, schrieb Dicebot: For anything even remotely complex I would have probably chosen JSON, either new std.json pending for review (not current std.json!) or vibe.data.json from vibed.org project. Which std.json are you referring to? There is no std.json2 or something in the review queue.
