fft and isPowerOf2?
Hi, this applications throws an error in std.numeric (Line 2826). => assert(isPowerOf2(range.length)); --- module std.numeric void fftImplPureReal(Ret, R)(R range, Ret buf) const in { assert(range.length >= 4); assert(isPowerOf2(range.length)); } --- The application is rewritten from python (numpy). How can I workaround this issue? Isn't it possible to give an arbitrary length of data to fft like in numpy? import std.numeric; import std.datetime.stopwatch; import std.stdio: writeln; import std.random; import std.array : array; import std.range : generate, takeExactly; void fourierTest(int samplingRate) { auto sw = StopWatch(AutoStart.yes); foreach(n; 0..60 * 24) { sw.stop(); double[] x = generate!(() => uniform01).takeExactly(samplingRate * 60).array; sw.start(); writeln(x); auto y = fft(x); } sw.stop(); writeln("microseconds: ", sw.peek.total!"usecs"); } void main() { fourierTest(50); } Kind regards André
Re: fft and isPowerOf2?
On Thursday, 17 May 2018 at 12:34:25 UTC, Andre Pany wrote: this applications throws an error in std.numeric (Line 2826). => assert(isPowerOf2(range.length)); Isn't it possible to give an arbitrary length of data to fft like in numpy? There is a mathematical background which is well explained here: https://math.stackexchange.com/questions/77118/non-power-of-2-ffts#77152
Creating a template mixin for explicit casts.
Given the following code `struct Foo(T) if(isNumeric!T) { T t; .. other code } struct Bar(T) if(isNumeric!T) { T t; .. other code } Foo!float foo_float; Foo!double foo_double; Bar!float bar_float; ` I want to make a template mixin that is able to cast one of these generic structs to the other explicitly. I have a bunch of these structs and therefore I thought it would make sense to do it by template mixin. I just don't know what language or library features I need to use and how to apply them. ` fooDouble = cast (Foo!double) foo_float; // case [1] if possible, done by // an implicit cast. Else by //explicit. fooFloat = cast (Foo!float) foo_double; // case [2] barFloat = cast (Bar!float) foo_float; // case [3] ` How would I do this in D?
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 15:25:37 UTC, Sjoerd Nijboer wrote: I want to make a template mixin that is able to cast one of these generic structs to the other explicitly. I have a bunch of these structs and therefore I thought it would make sense to do it by template mixin. I just don't know what language or library features I need to use and how to apply them. It sounds like you want to overload opCast: https://dlang.org/spec/operatoroverloading.html#cast
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 15:25:37 UTC, Sjoerd Nijboer wrote: Given the following code `struct Foo(T) if(isNumeric!T) { T t; .. other code } struct Bar(T) if(isNumeric!T) { T t; .. other code } Foo!float foo_float; Foo!double foo_double; Bar!float bar_float; ` I want to make a template mixin that is able to cast one of these generic structs to the other explicitly. I have a bunch of these structs and therefore I thought it would make sense to do it by template mixin. I just don't know what language or library features I need to use and how to apply them. ` fooDouble = cast (Foo!double) foo_float; // case [1] if possible, done by // an implicit cast. Else by //explicit. fooFloat = cast (Foo!float) foo_double; // case [2] barFloat = cast (Bar!float) foo_float; // case [3] ` How would I do this in D? Before you write anything yourself, check whether std.conv.castFrom does what you need: https://dlang.org/phobos/std_conv.html#castFrom
How to inform dub of generated source files?
Hi, Context: https://github.com/veelo/Pascal2D One of my source files is generated by executing `cd source && rdmd generate.d`, which creates the file `source/epparser.d`. (There is actually one step in between, calling `rdmd make.d`, which checks creation times, but that's not relevant here). I have added this step as preGenerateCommands in dub.json [1]. The problem is that the first time `dub build` is run, it does not seem to be aware of `source/epparser.d` and linking fails [2]. A second `dub build` succeeds. Isn't preGenerateCommands meant to cover this case? I can add `epparser.d` in an extra soureFile line in dub.json, which fixes the very first build, but that makes successive builds fail because it causes the file name to appear twice in the arguments to dmd once the file exists. Changing preGenerateCommands into preBuildCommands makes no difference. Thanks! Bastiaan. [1] https://github.com/veelo/Pascal2D/blob/master/dub.json#L22 [2] https://travis-ci.org/veelo/Pascal2D/builds/379096446#L511
Re: How to inform dub of generated source files?
On Thursday, 17 May 2018 at 17:42:21 UTC, Bastiaan Veelo wrote: Isn't preGenerateCommands meant to cover this case? Appears to be a bug. Filed https://github.com/dlang/dub/issues/1474
Re: Splitting up large dirty file
On Wednesday, 16 May 2018 at 15:47:29 UTC, Jon Degenhardt wrote: If you write it in the style of my earlier example and use counters and if-tests it will work. byLine by itself won't try to interpret the characters (won't auto-decode them), so it won't trigger an exception if there are invalid utf-8 characters. When printing to stdout it seems to skip any validation, but writing to a file does give an exception: ``` auto inputStream = (args.length < 2 || args[1] == "-") ? stdin : args[1].File; auto outputFile = new File("output.txt"); foreach (line; inputStream.byLine(KeepTerminator.yes)) outputFile.write(line); ``` std.exception.ErrnoException@C:\D\dmd2\windows\bin\..\..\src\phobos\std\stdio.d(2877): (No error) According to the documentation, byLine can throw an UTFException so relying on the fact that it doesn't in some cases doesn't seem like a good idea.
Re: there's no gdc for windows?
On Tuesday, 15 May 2018 at 14:25:31 UTC, Dr.No wrote: Has gdc been supported for Windows? if so, where can I find it? I've only find Linux versions so far... ldc has good Windows support: https://github.com/ldc-developers/ldc/releases
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 16:27:48 UTC, Paul Backus wrote: On Thursday, 17 May 2018 at 15:25:37 UTC, Sjoerd Nijboer wrote: I want to make a template mixin that is able to cast one of these generic structs to the other explicitly. I have a bunch of these structs and therefore I thought it would make sense to do it by template mixin. I just don't know what language or library features I need to use and how to apply them. It sounds like you want to overload opCast: https://dlang.org/spec/operatoroverloading.html#cast Something like: ` Foo!T opCast(U)(Foo!U old) if (isImplicitlyConvertible!(T, U)) { return Foo!T(old.t); } Foo!T opCast(UTemplate, U)(UTemplate!U old) if (isImplicitlyConvertible!(T, U) && isNumerical!U) { return Foo!T(old.t); } ` But then how do you put this into a mixin template so I can write something like ` struct Foo(T) if(isNumerical(T)) { mixin castingRules(typeof(this) T); } `
Re: Creating a template mixin for explicit casts.
On Thursday, 17 May 2018 at 20:38:13 UTC, Sjoerd Nijboer wrote: But then how do you put this into a mixin template so I can ... mixin castingRules(typeof(this) T); I guess I can refine my question to "How do you let a mixin template detect the template name it is instantiated with and return its type whitout the template arguments.
Re: Splitting up large dirty file
On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: For various reasons, that doesn't always hold true like it should, but pretty much all of Phobos is written with that assumption and will generally throw an exception if it isn't. It's unfortunate that Phobos tells you 'there's problems with the encoding' without providing any means to fix it or even diagnose it. The UTFException doesn't contain what the character in question was. You just have to abort whatever you were trying to do. On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: If you're ever dealing with a different encoding (or with invalid Unicode), you really need to use integral types like ubyte I tried something like byChunk(4096).joiner.splitter(cast(ubyte) '\n') but it turns out splitter wants at least a forward range, even when the separator is a single element.
Re: Splitting up large dirty file
On Tuesday, 15 May 2018 at 20:36:21 UTC, Dennis wrote: I have a file with two problems: - It's too big to fit in memory (apparently, I thought 1.5 Gb would fit but I get an out of memory error when using std.file.read) Memory mapping should work. That's in core.sys.posix.sys.mman for Posix systems, and Windows has some equivalent probably. (But nobody uses Windows, right?) - It is dirty (contains invalid Unicode characters, null bytes in the middle of lines) std.algorithm should generally work with sequences of anything, not just strings. So memory map, cast to ubyte[], and deal with it that way? - When you convert chunks to arrays, you have the risk of a split being in the middle of a character with multiple code units It's straightforward to scan for the start of a Unicode character; you just skip past characters where the highest bit is set and the next-highest is not. (0b1100_ through 0b_1110 is the start of a multibyte character; 0b_ through 0b0111_ is a single-byte character.) That said, you seem to only need to split based on a newline character, so you might be able to ignore this entirely, even if you go by chunks.
Template instantiation fails on Linux, succeeds on Windows
Hi! The code in [1] compiles and runs flawlessly on Windows, but not on Linux (neither run.dlang nor Travis docker image). Any idea what can be done? errors: Error: undefined identifier __va_list_tag onlineapp.d(282): Error: template instance `onlineapp.SetFactory!byte` error instantiating Error: undefined identifier __va_list_tag onlineapp.d(288): Error: template instance `onlineapp.SetFactory!(Count)` error instantiating Error: undefined identifier __va_list_tag onlineapp.d(290): Error: template instance `onlineapp.SetFactory!char` error instantiating SetFactory is defined on row 233, and it seems that the problem relates to the vararg opIndex. [1] https://run.dlang.io/gist/6dcbe4e82846cbeb3bf478e92a2b6e6f
Re: Splitting up large dirty file
On 05/17/2018 11:40 PM, Neia Neutuladh wrote: 0b1100_ through 0b_1110 is the start of a multibyte character Nitpick: It only goes up to 0b_0100. The highest code point is U+10. There are no sequences with more than four bytes.
Re: Template instantiation fails on Linux, succeeds on Windows
On Thursday, 17 May 2018 at 22:07:46 UTC, Bastiaan Veelo wrote: Hi! The code in [1] compiles and runs flawlessly on Windows, but not on Linux (neither run.dlang nor Travis docker image). Any idea what can be done? Hello. Yes, add `import core.stdc.stdarg;` in your module and it works. I don't know why it's not required on windows, this is strange.
Re: Splitting up large dirty file
On Thursday, 17 May 2018 at 20:08:09 UTC, Dennis wrote: On Wednesday, 16 May 2018 at 15:47:29 UTC, Jon Degenhardt wrote: If you write it in the style of my earlier example and use counters and if-tests it will work. byLine by itself won't try to interpret the characters (won't auto-decode them), so it won't trigger an exception if there are invalid utf-8 characters. When printing to stdout it seems to skip any validation, but writing to a file does give an exception: ``` auto inputStream = (args.length < 2 || args[1] == "-") ? stdin : args[1].File; auto outputFile = new File("output.txt"); foreach (line; inputStream.byLine(KeepTerminator.yes)) outputFile.write(line); ``` std.exception.ErrnoException@C:\D\dmd2\windows\bin\..\..\src\phobos\std\stdio.d(2877): (No error) According to the documentation, byLine can throw an UTFException so relying on the fact that it doesn't in some cases doesn't seem like a good idea. Instead of: auto outputFile = new File("output.txt"); try: auto outputFile = File("output.txt", "w"); That works for me. The second arg ("w") opens the file for write. When I omit it, I also get an exception, as the default open mode is for read: * If file does not exist: Cannot open file `output.txt' in mode `rb' (No such file or directory) * If file does exist: (Bad file descriptor) The second error presumably occurs when writing. As an aside - I agree with one of your bigger picture observations: It would be preferable to have more control over utf-8 error handling behavior at the application level.
Re: Template instantiation fails on Linux, succeeds on Windows
On Thursday, 17 May 2018 at 23:18:32 UTC, Basile B. wrote: On Thursday, 17 May 2018 at 22:07:46 UTC, Bastiaan Veelo wrote: Hi! The code in [1] compiles and runs flawlessly on Windows, but not on Linux (neither run.dlang nor Travis docker image). Any idea what can be done? Hello. Yes, add `import core.stdc.stdarg;` in your module and it works. I don't know why it's not required on windows, this is strange. Great! Thanks a lot, Basile! Seems there is room for improvement somewhere, a better error message at the least.
Re: Splitting up large dirty file
On Thursday, May 17, 2018 21:10:35 Dennis via Digitalmars-d-learn wrote: > On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: > > For various reasons, that doesn't always hold true like it > > should, but pretty much all of Phobos is written with that > > assumption and will generally throw an exception if it isn't. > > It's unfortunate that Phobos tells you 'there's problems with the > encoding' without providing any means to fix it or even diagnose > it. The UTFException doesn't contain what the character in > question was. You just have to abort whatever you were trying to > do. UTFException has a sequence member and a len member (which appear to be public but undocumented) which should contain the invalid sequence of code units. In general though, exceptions aren't a great way to deal with this problem. I think that you either want to be calling decode manually (in which case, you have direct access to where the invalid Unicode is and have the freedom to deal with it however is appropriate), or using the Unicode replacement character would be better (which std.utf.decode supports, but it's not what's used by default). Really, what's byting you here is the auto-decoding. With Phobos, you have to fight to have it not happen by doing stuff like special-casing your code for strings or using std.string.representation or std.utf.byCodeUnit. In principle, the way that Unicode would ideally be handled would be to validate all character data when it enters the program (soing whatever is appropriate with invalid Unicode at that point), and then the rest of the program then either is always dealing with valid Unicode, or it's dealing with integral values that it doesn't treat as Unicode (e.g. ubyte[]). But the way that Phobos is written, it ends up decoding and validating all over the place. > On Wednesday, 16 May 2018 at 10:30:34 UTC, Jonathan M Davis wrote: > > If you're ever dealing with a different encoding (or with > > invalid Unicode), you really need to use integral types like > > ubyte > > I tried something like byChunk(4096).joiner.splitter(cast(ubyte) > '\n') but it turns out splitter wants at least a forward range, > even when the separator is a single element. Actually, I'm pretty sure that splitter curently requires a random-access range (even though it should theoretically work with a forward range). I don't think that it can be made to work with an input range though given how the range API works - or at least, it were made to work with it, you'd have to deal with the fact that popping front on the spitter range would invalidate anything that had been returned from front. And it would be difficult to implement it @safely if what gets returned by front is not completely independent of the splitter range (which means that it needs save). Basic input ranges in general tend to be extremely limited in what they can do, which can get really annoying when you deal with stuff like files or sockets where making it a forward range likely means either reading it all into memory or having buffers that potentially have to be dup-ed by each call to save. - Jonathan M Davis
Re: fft and isPowerOf2?
On Thursday, 17 May 2018 at 13:36:39 UTC, kdevel wrote: On Thursday, 17 May 2018 at 12:34:25 UTC, Andre Pany wrote: this applications throws an error in std.numeric (Line 2826). => assert(isPowerOf2(range.length)); Isn't it possible to give an arbitrary length of data to fft like in numpy? There is a mathematical background which is well explained here: https://math.stackexchange.com/questions/77118/non-power-of-2-ffts#77152 Thanks a lot for the details link. Kind regards André