Re: Improving CSV parsing performance, Episode 2 (Was: Re: Speed of csvReader)
On Sunday, 24 January 2016 at 01:57:11 UTC, H. S. Teoh wrote: - Ummm... make it ready for integration with std.csv maybe? ;-) T My suggestion is to take the unittests used in std.csv and try to get your code working with them. As fastcsv limitations would prevent replacing the std.csv implementation the API may not need to match, but keeping close to the same would be best.
First project: questions on how-to, and on language features
Source code: https://alexvincent.us/d-language/samples/intervalmap-rev1.d.txt After reading Ali Çehreli's book, "Programming in D", I wrote this little sample up as a starting point for my explorations into D. I've long admired the D language, but I've never actually tried to write practical code in D. So I wrote this little sample up in dlangide, and as I was going, I realized I had quite a few questions. (1) It's not clear how to specify certain parts of a module or library as non-exportable. Is that possible? Is it desirable? (It's not that important, yet, but still...) (2) In the unittest code, I have a block that I want to rewrite using assertThrown, but I can't figure out from either the book or the website the correct usage. What's the right way to specify a StringException with a particular message I expect to receive? (3) How do I actually create a library in dub? How does dub determine what files to build? (4) How should the scope(exit) and scope(failure) guard statements intermix with preconditions and postconditions? (5) My append() function has a postcondition that currently depends on debug-only members of the class being set in the precondition. This seems artificial - not the part about setting these variables in the precondition, but having the variables defined on the class to begin with. If they were defined only for the lifetime of the function's execution, starting in the precondition, this would be more natural. Is there a Right Way to define function-only debug variables for use in postconditions? If not, would this be a valid use-case to consider amending the language specification to clarify? (6) Would someone please review my sample code and offer feedback? :-) (7) Naming: This code is the start of a port of a mini-library I wrote in JavaScript which I called "ObjectRange". However, in Phobos, the term "Range" has a very different meaning - in fact, almost the opposite meaning of what this code does. I asked for a better name in the D IRC channel, and someone suggested Interval, based on the mathematical definition of the word. IntervalMap seemed more accurate, since there is a payload to each interval. Does the name make sense in the context? (8) Is there a similar, more mature library out there which tries to achieve what I'm doing here? I'm rather surprised I didn't see anything like it in the standard library...
Re: alias template parameter
On Friday, 21 June 2013 at 14:08:43 UTC, Sergei Nosov wrote:
Hi!
I've been thinking about how alias template parameters work and
I'm really confused =)
It makes perfect sense for literals, names, etc. But what I
can't get is how does it work for delegates.
If I have a function
auto apply(alias fun, T...)(T args)
{
return fun(args);
}
And then I have
int y = 2;
apply!(x => y)(1);
How in the world does this work? Is the context address known
at compile-time?
y is allocated on the heap and the pointer is implicitly passed
to the apply, or is a field of a struct if you use map!(x => y)
instead.
Improving CSV parsing performance, Episode 2 (Was: Re: Speed of csvReader)
On Fri, Jan 22, 2016 at 10:04:58PM +, data pulverizer via Digitalmars-d-learn wrote: [...] > I guess the next step is allowing Tuple rows with mixed types. Alright. I threw together a new CSV parsing function that loads CSV data into an array of structs. Currently, the implementation is not quite polished yet (it blindly assumes the first row is a header row, which it discards), but it does work, and outperforms std.csv by about an order of magnitude. The initial implementation was very slow (albeit still somewhat fast than std.csv by about 10% or so) when given a struct with string fields. However, structs with POD fields are lightning fast (not significantly different from before, in spite of all the calls to std.conv.to!). This suggested that the slowdown was caused by excessive allocations of small strings, causing a heavy GC load. This suspicion was confirmed when I ran the same input data with a struct where all string fields were replaced with const(char)[] (so that std.conv.to simply returned slices to the data) -- the performance shot back up to about 1700 msecs, a little slower than the original version of reading into an array of array of const(char)[] slices, but about 58 times(!) the performance of std.csv. So I tried a simple optimization: instead of allocating a string per field, allocate 64KB string buffers and copy string field values into it, then taking slices from the buffer to assign to the struct's string fields. With this optimization, running times came down to about the 1900 msec range, which is only marginally slower than the const(char)[] case, about 51 times faster than std.csv. Here are the actual benchmark values: 1) std.csv: 2126883 records, 102136 msecs 2) fastcsv (struct with string fields): 2126883 records, 1978 msecs 3) fastcsv (struct with const(char)[] fields): 2126883 records, 1743 msecs The latest code is available on github: https://github.com/quickfur/fastcsv The benchmark driver now has 3 new targets: stdstruct - std.csv parsing of CSV into structs faststruct - fastcsv parsing of CSV into struct (string fields) faststruct2 - fastcsv parsing of CSV into struct (const(char)[] fields) Note that the structs are hard-coded into the code, so they will only work with the census.gov test file. Things still left to do: - Fix header parsing to have a consistent interface with std.csv, or at least allow the user to configure whether or not the first row should be discarded. - Support transcription to Tuples? - Refactor the code to have less copy-pasta. - Ummm... make it ready for integration with std.csv maybe? ;-) T -- Fact is stranger than fiction.
Re: Possible to get Class of Interface at runtime
On Saturday, 23 January 2016 at 21:03:21 UTC, Josh Phillips wrote: On Friday, 22 January 2016 at 23:44:34 UTC, Adam D. Ruppe wrote: There's a .classinfo property that works on Objects. If you have an interface, cast to Object first, and check for null, then get .classinfo off that. I tried this but it will return A not B http://dpaste.dzfl.pl/f1bcf74d8cab
Re: Possible to get Class of Interface at runtime
On Saturday, 23 January 2016 at 21:03:21 UTC, Josh Phillips wrote: I tried this but it will return A not B Are you sure you correctly casted first?
Re: Possible to get Class of Interface at runtime
On Friday, 22 January 2016 at 23:44:34 UTC, Adam D. Ruppe wrote: There's a .classinfo property that works on Objects. If you have an interface, cast to Object first, and check for null, then get .classinfo off that. I tried this but it will return A not B
Re: Define "createXXX" functions for the constructors of class XXX
On Sat, 23 Jan 2016 19:42:29 +, Johan Engelen wrote:
> // Somehow define these guys automatically, "genCreateCtors!(XXX)" ?
>XXX createXXX(int a, int b) { return new XXX(a, b); }
>XXX createXXX(bool flag) { return new XXX(flag); }
Check out http://dpaste.dzfl.pl/430dabf25935
I used string mixins to be absolutely, 100% sure that the right things
ended up in the symbol table.
__traits(getOverloads, Type, "name") gets function overloads for the
given name. It gets them as an AliasSeq, which presents an array-like
interface.
Constructors use the name "__ctor". So you can get all constructors for a
class named MyClass with:
__traits(getOverloads, MyClass, "__ctor")
This only includes explicitly defined constructors, not the default
constructor, so we handle that separately.
std.traits defines a Parameters template that takes a function and yields
a parameter type tuple. We can hand it a constructor and it does the
right thing. We can treat a bunch of values as one value if their type is
a type tuple, so that's exactly what we do.
Then it's just a bit of string manipulation to get the right class names
and overloads in the right places and we've generated a string of D code
that does what we want. Mix it in and Bob's your uncle.
In your code, the line
mixin(factory!MyClass);
should be inside an extern(C++) block, naturally.
The major caveat there: wherever you call "mixin(factory!MyClass);", you
need MyClass to be in scope directly. If you have to refer to it with a
qualified name, this will break. If that causes problems for you, you can
use:
extern(C++) {
{
import mypkg.mymodule : MyClass;
mixin(factory!MyClass);
}
}
And that should import just that class for just that one block of code.
Re: Define "createXXX" functions for the constructors of class XXX
On Saturday, 23 January 2016 at 19:42:29 UTC, Johan Engelen wrote:
Hi all,
While trying to interface C++ and D, I have to new a few D
objects in C++ code. I am doing this using a D function: "XXX
createXXX(...) { return new XXX(...); }".
I am sure there must be some great way to automatically
generate these creator functions, but I don't know how to do it.
In the C++-header I will write manually:
XXX* createXXX(int a, int b);
XXX* createXXX(bool flag);
In D source:
extern (C++) class XXX {
this(int a, int b) { /+...+/ }
this(bool flag) { /+...+/ }
}
// Somehow define these guys automatically,
"genCreateCtors!(XXX)" ?
XXX createXXX(int a, int b) { return new XXX(a, b); }
XXX createXXX(bool flag) { return new XXX(flag); }
Thanks a lot!
Johan
Wow! There are lots of XXX there.
Anyway, I did a similar thing to yours for automatic attribute
definition before. Three things:
1. Template
2. Mixin
3. Compile time function
You define a compile time function which generates a string that
is valid D code.
You define template that takes some parameters (Your XXX values),
and calls the function to merge them.
In your class, you use mixin and template to generate the string
and inject the generated code.
Not that complex once you do it.
Try to understand this code.
http://david.rothlis.net/d/templates/
Its in there.
Define "createXXX" functions for the constructors of class XXX
Hi all,
While trying to interface C++ and D, I have to new a few D
objects in C++ code. I am doing this using a D function: "XXX
createXXX(...) { return new XXX(...); }".
I am sure there must be some great way to automatically generate
these creator functions, but I don't know how to do it.
In the C++-header I will write manually:
XXX* createXXX(int a, int b);
XXX* createXXX(bool flag);
In D source:
extern (C++) class XXX {
this(int a, int b) { /+...+/ }
this(bool flag) { /+...+/ }
}
// Somehow define these guys automatically,
"genCreateCtors!(XXX)" ?
XXX createXXX(int a, int b) { return new XXX(a, b); }
XXX createXXX(bool flag) { return new XXX(flag); }
Thanks a lot!
Johan
Re: Adam D. Ruppe's Minigui using example
On Saturday, 16 January 2016 at 03:14:01 UTC, Adam D. Ruppe wrote: On Saturday, 16 January 2016 at 01:27:32 UTC, Andre Polykanine wrote: Does anyone have an actual example of, say, a simple form with a set of radio buttons or check boxes to start with? Thanks in advance! Sort of. I still haven't used it in a real world program so it isn't complete but here's one of my test programs: [...] On Windows, it uses the native controls, so it should work reasonably well, though on Linux it does its own thing and is far from complete. BTW simplewindow doesn't seem to compile on newest dmd because of the new core.sys.windows.windows so I'll have to update that if you're on the newest release... Thanks Adam, I have DMD 2.068.2 so far so it did work. However it seems to be not so accessible as I expected (tested with JAWS for Windows, latest release [1]). For instance, the focus lands nowhere when the program is launched. Then there is a (not too comfortable) possibility to make it show the elements to the screen reader upon pressing Tab. [1]:https://en.wikipedia.org/wiki/JAWS_(screen_reader)
Re: Is memory-safe IO possible?
You can try to write a trusted wrapper for one of curl functions and ask for a review on forum. Maybe it will be fruitful.
Re: Shared library question
On Saturday, 23 January 2016 at 00:38:45 UTC, Dibyendu Majumdar wrote: On Friday, 22 January 2016 at 22:06:35 UTC, Dibyendu Majumdar wrote: Hi I am trying to create a simple shared library that exports a D function, but when I try to link to it I get errors such as: error LNK2001: unresolved external symbol _D7xxx12__ModuleInfoZ I have uploaded my small test here so if anyone can tell me what I am doing wrong it will hugely appreciated. Thanks! https://github.com/dibyendumajumdar/d-lab/tree/master/sharedlib Dll support on windows is basically non existant at the moment and I strongly advice against trying to use it. Trust me I've been there. I'm currently working on propper Dll support but that is most likely going to take a few more months. Your best option currently is to simply link everything statically. Kind Regards Benjamin
Re: std.zip for Binary example
On Sunday, 17 January 2016 at 10:43:55 UTC, Sean Campbell wrote:
On Sunday, 17 January 2016 at 10:34:19 UTC, locco wrote:
Hi :)
I found this example:
==
import std.file: write;
import std.string: representation;
void main()
{
char[] data = "Test data.\n".dup;
// Create an ArchiveMember for the test file.
ArchiveMember am = new ArchiveMember();
am.name = "test.txt";
am.expandedData(data.representation);
// Create an archive and add the member.
ZipArchive zip = new ZipArchive();
zip.addMember(am);
// Build the archive
void[] compressed_data = zip.build();
// Write to a file
write("test.zip", compressed_data);
}
==
But i cound't find example code for binary file.
How can i make ArciveMember for binary file?
std.zip dosen't discriminate against text and binary files
files.
p.s. remember to import std.zip
import std.stdio;
import std.file:write,getSize;
import std.string: representation;
import std.zip: ArchiveMember, ZipArchive;
void main() {
auto stream = File("example.jpg", "r+");
auto inbytes = new ubyte[ cast(uint)getSize("example.jpg") ];
stream.rawRead(inbytes);
auto zip = new ZipArchive();
ArchiveMember ae = new ArchiveMember();
ae.name = "example_.jpg";
ae.expandedData( inbytes );
zip.addMember(ae);
void[] compressed_data = zip.build();
write("test.zip", compressed_data);
}
I made this, work fine. But i guess it wasn't nice solution.
Right?
Re: Templates, templates, templates...
On 23.01.2016 12:30, Voitech wrote:
Ok so i want to hold different types in LogicRule maybe Algebraic
implementation would do?
private alias ControllTemplate(T) =Rule!(T,ControllFlag);
private alias SymbolRule =ControllTemplate!(SymbolType);
private alias StringRule =ControllTemplate!(SymbolRule[]);
private alias LogicTemplate(T...)
=Rule!(Algebraic!(ControllTemplate(T))[],LogicFlag); <--error
You're missing an exclamation mark there, and you've got the order of
Algebraic and ControllTemplate wrong. This compiles:
private alias LogicTemplate(T...) =
Rule!(ControllTemplate!(Algebraic!T)[],LogicFlag);
private alias AlgebraicLogicRule = LogicTemplate!(SymbolRule,StringRule);
error:
Error: cannot pass type (Rule!(SymbolType, ControllFlag),
Rule!(Rule!(SymbolType, ControllFlag)[], ControllFlag)) as a function
argument
[...]
Is there any nicer way to handle this case ?
Instead of Algebraic you could use a common base class, or interface,
for the Rule instantiations:
abstract class RuleBase
{
... whatever common functionality rules have ...
}
class Rule(V,F) : RuleBase { ...}
But I have to say that I'm having trouble making sense of all that class
and template complexity, and how it helps in actually validating user input.
Since this is a parsing thing, you may want to look into writing parsers
an/or using a parse generator. I think Pegged is the most popular one
for D. http://code.dlang.org/packages/pegged
Re: core.thread.Thread.start is marked as "nothrow" but documentation says it throws
On 23.01.2016 12:44, tcak wrote: https://dlang.org/phobos/core_thread.html#.Thread final nothrow Thread.start() Looking at the code, no "throw new ..." is seen, but the function "onThreadError" is called which has "throw" in it. Most weird thing is that "onThreadError" function is marked as "nothrow" but it still throws. I would think that yes, maybe the compiler might not be able to see it because throw is found in another function, but how come "onThreadError" throws with nothrow. onThreadError [1] throws a ThreadError [2] which derives from Error, as the name suggests. nothrow functions may throw Errors, because Errors are considered a way of force-exiting the program. Errors are not supposed to be caught. So onThreadError is fine. And if Thread.start can actually only throw ThreadError, and not ThreadException, then that one is ok too, but the documentation is wrong and should be fixed. [1] https://github.com/D-Programming-Language/druntime/blob/33f1962747801be35a48f4875c909e16747fdcce/src/core/thread.d#L2972 [2] https://github.com/D-Programming-Language/druntime/blob/33f1962747801be35a48f4875c909e16747fdcce/src/core/thread.d#L88
core.thread.Thread.start is marked as "nothrow" but documentation says it throws
https://dlang.org/phobos/core_thread.html#.Thread final nothrow Thread.start() Looking at the code, no "throw new ..." is seen, but the function "onThreadError" is called which has "throw" in it. Most weird thing is that "onThreadError" function is marked as "nothrow" but it still throws. I would think that yes, maybe the compiler might not be able to see it because throw is found in another function, but how come "onThreadError" throws with nothrow.
Templates, templates, templates...
Hi, I have a problem with creating proper inheritance chain with
templates. First i will give some background about my problem.
I'm trying to create a validator for math calculation
expressions. I don't want to use regexps as this is approach
gives me headache and probably will not allow further extension i
want. For now expressions should look like this:
1+2+3/4, -(5*(25-5)/19), sqrt(-5/-6*(212)) etc.
So each of charactes is parsed to something like PrimaryElement
class PrimaryElement {
const dchar value;
SymbolType symbolType;
}
SymbolsType is an enum which contains:
EXPR_START,+,-,/,DIGIT,(,),.,EXPR_END
So now i want to create validator for input so user couldn't
insert something like:
/00..34-+/493 but only 0.34-493. I want to divide it into phases.
First phase is SymbolType validation so it will handles problems
like:
+-/345..3 but not .234+3-53
Second phase will take care about value validation so user cant
insert .0+3 but only 0.000+3
Third phase would be executed to check expression is completed
and not allow to calculate expression for situations like
0.000+3, 3-3+4.5+ or 364/4-5.3+(
Now i'm trying to implement phase one. So i create
PrimaryElementProcessor which will take some kind of rules and
check PrimaryElement[] that way if divided parts of array fits
into one of rules then it is valid. Rules will be trimmed to size
of expression if necessary.
Model and constants declaration looks like this:
private enum ControllFlag{
none,ommitable,repeatable
}
private enum LogicFlag{
none,or,and
}
private class Rule(V,F){
V value;
F flag;
this(){
}
this(V value){
this.value=value;
}
this(V value,F flag){
this.value=value;
this.flag=flag;
}
}
private alias ControllTemplate(T) =Rule!(T,ControllFlag);
private alias SymbolRule =ControllTemplate!(SymbolType);
private alias StringRule =ControllTemplate!(SymbolRule[]);
private alias LogicTemplate(T...)
=Rule!(ControllTemplate(T)[],LogicFlag);
private alias LogicRule=LogicTemplate!([SymbolRule,StringRule]);
<--error
So i first want to handle case like D. or DD where D
is Digit
instantiation code:
SymbolRule digitRule = new
SymbolRule(SymbolType.digit,ControllFlag.repeatable);
SymbolRule commaRule = new SymbolRule(SymbolType.comma);
StringRule numericRule = new
StringRule([digitRule,commaRule,digitRule]);
LogicRule decimalRule = new
LogicRule([digitRule,numericRule],LogicFlag.or);
error:
Error: type Rule!(Rule!(SymbolType, ControllFlag)[],
ControllFlag) has no value
Ok so i want to hold different types in LogicRule maybe Algebraic
implementation would do?
private alias ControllTemplate(T) =Rule!(T,ControllFlag);
private alias SymbolRule =ControllTemplate!(SymbolType);
private alias StringRule =ControllTemplate!(SymbolRule[]);
private alias LogicTemplate(T...)
=Rule!(Algebraic!(ControllTemplate(T))[],LogicFlag); <--error
private alias AlgebraicLogicRule =
LogicTemplate!(SymbolRule,StringRule);
error:
Error: cannot pass type (Rule!(SymbolType, ControllFlag),
Rule!(Rule!(SymbolType, ControllFlag)[], ControllFlag)) as a
function argument
So maybe something simpler:
private alias ControllTemplate(T) =Rule!(T,ControllFlag);
private alias SymbolRule =ControllTemplate!(SymbolType);
private alias StringRule =ControllTemplate!(SymbolRule[]);
private alias SimpleLogicRule
=Rule!(Algebraic!(SymbolRule,StringRule)[],LogicFlag);
Compiles but... when i try to instantiate SimpleLogicRule like
SymbolRule digitRule = new
SymbolRule(SymbolType.digit,ControllFlag.repeatable);
SymbolRule commaRule = new SymbolRule(SymbolType.comma);
StringRule numericRule = new
StringRule([digitRule,commaRule,digitRule]);
SimpleLogicRule decimalRule = new
SimpleLogicRule([digitRule,numericRule],LogicFlag.or); <--- error
i get error:
None of the overloads of '__ctor' are callable using argument
types (Object[], LogicFlag), candidates are: ...
So i understand compiler can't cast/extract array of different
types to known type.
I created a wrapper for this two types SymbolRule and StringRule
and init it somewhere before passing to LogicRule ctor. This
approach makes a lot of boilerplate code for example:
alias Wrapper = Algebraic!(SymbolRule,StringRule);
alias LogicRule =Rule!(Wrapper[],LogicFlag);
SymbolRule digitRule = new
SymbolRule(SymbolType.digit,ControllFlag.repeatable);
SymbolRule commaRule = new SymbolRule(SymbolType.comma);
StringRule numericRule = new
StringRule([digitRule,commaRule,digitRule]);
Wrapper digitRuleWrapper =digitRule; <-- how to ommit this ?
Wrapper numericRuleWrapper =numericRule; <-- how to ommit this ?
LogicRule decimalRule=new
LogicRule([digitRuleWrapper,numericRuleWrapper],LogicFlag.or);
Is there any nicer way to handle this case ?
Cheers Voitech.
Re: htod question
On Friday, 22 January 2016 at 01:04:50 UTC, Dibyendu Majumdar wrote: On Friday, 22 January 2016 at 01:03:09 UTC, Dibyendu Majumdar wrote: On Friday, 22 January 2016 at 00:52:59 UTC, W.J. wrote: Counter question: What's so bad about the D std library ? I am trying to create bindings for existing C library so I was trying to use htod for that. The library includes various C header files ... causing htod to fail Yeah, htod often requires preparing the files your trying to convert. Often removing macro's and the like. Its a manual process, and it can get dirty.
Re: Collapsing n-dimensional array to linear (1 dimensional)
On 01/22/2016 04:07 AM, abad wrote:
Let's say I have an array like this:
int[][][] array;
And I want to generate a linear int[] based on its data. Is there a
standard library method for achieving this, or must I iterate over the
array manually?
What I'm thinking of is something like this:
int[] onedim = std.array.collapse(array);
Something feels extra down there but it seems to work. :)
import std.stdio;
import std.range;
import std.algorithm;
auto collapse(R)(R r)
if (isArray!R) {
return r.joiner.collapse.joiner;
}
auto collapse(R)(R r)
if (!isArray!R) {
return r;
}
unittest {
auto r = 3.iota
.map!(i => iota(3 * i, 3 * i + 3)
.map!(j => iota(3 * j, 3 * j + 3)
.array)
.array)
.array;
writeln("Original:\n", r);
writeln("\nCollapsed:\n", r.collapse);
assert(r.collapse.equal(iota(27)));
}
void main() {
}
Original:
[[[0, 1, 2], [3, 4, 5], [6, 7, 8]], [[9, 10, 11], [12, 13, 14], [15, 16,
17]], [[18, 19, 20], [21, 22, 23], [24, 25, 26]]]
Collapsed:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23, 24, 25, 26]
Ali
