How do I get names of regex captures during iteration? Populate AAs with captures?
Is there a way to get the name of a named capture when iterating
over captures from a regular expression match? I've looked at
the std.regex code and it seems like "no" to my eyes, but I
wonder if others here have... a way.
My original problem is this: I need to populate an associative
array (AA) with all named captures that successfully matched
during a regex match (and none of the captures that failed). I
was wondering what the best way to do this might be.
Thanks!
Please see comments in the below program for details and my
current progress:
void main()
{
import std.compiler;
import std.regex;
import std.range;
import std.stdio;
writefln("Compiler name:%s", std.compiler.name);
writefln("Compiler version: %s.%s", version_major,
version_minor);
writeln("");
enum pattern = `(?P\w+)\s*=\s*(?P\d+)?;`;
writefln("Regular expression: `%s`", pattern);
writeln("");
auto re = regex(pattern);
auto c = matchFirst("a = 42;", re);
reportCaptures(re, c);
c = matchFirst("a = ;", re);
reportCaptures(re, c);
}
void reportCaptures(Regex, RegexCaptures)(Regex re, RegexCaptures
captures)
{
import std.range;
import std.regex;
import std.stdio;
writefln("Captures from matched string '%s'",
captures[0]);
string[string] captureList;
// I am trying to read the captures from a regular
expression match
// into the above AA.
//
// ...
//
// This kind of works, but requires a string lookup for
each capture
// and using it in practice relies on undocumented
behavior regarding
// the return value of std.regex.Capture's
opIndex[string] method
// when the string index is a valid named capture that
was not actually
// captured during the match (ex: the named capture was
qualified with
// the ? operator or the * operator in the regex and
never appeared in
// the matched string).
foreach( captureName; re.namedCaptures )
{
auto capture = captures[captureName];
if ( capture is null )
writefln(" captures[%s] is null",
captureName);
else if ( capture.empty )
writefln(" captures[%s] is empty",
captureName);
else
{
writefln(" captures[%s] is '%s'",
captureName, capture);
captureList[captureName] = capture;
}
}
writefln("Total captures: %s", captureList);
/+
// I really want to do something like this, instead:
foreach( capture; captures )
captureList[capture.name] = capture.value;
// And, in reality, it might need to be more like this:
foreach( capture; captures )
foreach ( valueIndex, value; capture.values )
captureList[format("%s-%s",capture.name,valueIndex)] = value;
// Because, logically, named captures qualified with the
// *, +, or {} operators in regular expressions may
capture
// multiple slices.
writefln("Total captures: %s", captureList);
+/
writeln("");
}
//Output, DMD64 D Compiler v2.073.1:
//---
//
//Compiler name:Digital Mars D
//Compiler version: 2.73
//
//Regular expression: `(?P\w+)\s*=\s*(?P\d+)?;`
//
//Captures from matched string 'a = 42;'
// captures[value] is '42'
// captures[var] is 'a'
//Total captures: ["value":"42", "var":"a"]
//
//Captures from matched string 'a = ;'
// captures[value] is empty
// captures[var] is 'a'
//Total captures: ["var":"a"]
Alignment of struct containing SIMD field - GDC
struct vec_struct {
alias field this;
bool b;
int8 field;
}
In this code when you look at the generated x64 code output by
GDC it seems to be doing a nice job, because it has got the
offset right for the 256-bit YMM 'field' correct.
Does D automatically propagate the alignment restrictions on the
field to the allocation of static structs or structs on the stack?
In this case:
struct vec_struct {
bool b2;
struct {
alias field this;
bool b;
int8 field;
}
}
it appears that the offset to 'field' is no longer aligned
correctly - offset is 40 bytes in GDC. I don't suppose the
compiler will use solely unaligned instructions? In any event, I
could take the address of field and then pass that to someone
expecting to pick up something with guaranteed correct alignment,
if I have understood the D docs.
Please don't bite. I'm both new to D and I hope I have understood
the x86 SIMD instructions' docs. (Very experienced professional
asm and C programmer, but v out-of-date.)
Noob q: I notice that the GDC opcodes look a bit odd, for example
the compiler generates a 256-bit unaligned fetch followed by an
aligned binary operation (I think), eg a movdqu followed by a
vpaddd r, ymm ptr blah - is the latter aligned-only? Apologies if
I have got this wrong, need to read up. Would someone
sanity-check me?
Re: Mixing libraries
On Tuesday, 28 February 2017 at 20:08:25 UTC, Jordan Wilson wrote: Hello, Been trying to learn the Simple Fast Multimedia Library (SFML) using the Derelict bindings, and noticed some functionality is offered by both SFML and the std library (for example, sfClock and sfMutex). Is there a general design principle of, say, use the std library whenever possible and only SFML when I have too? Or should I try to be consistent and use SFML library whenever possible? Thanks, Jordan No. Entirely up to you.
Re: How to enforce compile time evaluation (and test if it was done at compile time)
On Tuesday, 28 February 2017 at 00:22:28 UTC, sarn wrote: If you ever have doubts, you can always use something like this to check: assert (__ctfe); Sorry, "enforce" would more appropriate if you're really checking. if (!__ctfe) assert(false); ... might be the best option. That shouldn't be compiled out even in -release builds.
Re: Getting underlying struct for parseJSON
On Tuesday, 28 February 2017 at 20:48:33 UTC, Petar Kirov [ZombineDev] wrote: On Tuesday, 28 February 2017 at 20:27:25 UTC, Alexey H wrote: [...] If you really care about performance, have a look this: http://forum.dlang.org/post/20151014090114.60780ad6@marco-toshiba std.json is not tuned for performance, so don't expect good results from it. It's a bit OT, but asdf is even faster and has a simple API: https://github.com/tamediadigital/asdf In terms of performance: Reading JSON line separated values and parsing them to ASDF - 300+ MB per second (SSD). Writing ASDF range to JSON line separated values - 300+ MB per second (SSD). Another good library is std.data.json (Json parsing extracted from Vibe.d): https://github.com/s-ludwig/std_data_json
Re: Getting underlying struct for parseJSON
On Tuesday, 28 February 2017 at 20:27:25 UTC, Alexey H wrote:
So, my real question is: can i actually, by any change, get the
description of an underlying struct that the call to parseJSON
generates?
It doesn't actually generate one, it just returns a tagged union
(a kind of dynamic type).
But, my json inspector program (never finished btw) has something
that can kinda generate structs from json:
https://github.com/adamdruppe/inspector
I ran your data.json through my program, and it spat this out
(well, not directly, I did a few minor tweaks by hand since it
doesn't output 100% valid D, but it is a good start):
struct Json_t {
struct sports_t {
long regionId;
string name;
long id;
string sortOrder;
long parentId;
string kind;
}
sports_t[] sports;
long siteVersion;
struct eventBlocks_t {
long[] factors;
long eventId;
string state;
}
eventBlocks_t[] eventBlocks;
struct customFactors_t {
long lo;
long e;
bool isLive;
double v;
string pt;
long f;
long p;
long hi;
}
customFactors_t[] customFactors;
struct announcements_t {
long segmentId;
string place;
bool liveHalf;
long regionId;
string name;
long[] tv;
string segmentName;
string segmentSortOrder;
long id;
long num;
string namePrefix;
string team1;
long startTime;
long sportId;
string team2;
}
announcements_t[] announcements;
struct events_t {
long level;
string sortOrder;
string place;
string name;
long rootKind;
long parentId;
long id;
long num;
string namePrefix;
string team1;
long kind;
struct state_t {
bool inHotList;
bool willBeLive;
bool liveHalf;
}
state_t state;
long startTime;
long sportId;
long priority;
string team2;
}
events_t[] events;
long packetVersion;
struct eventMiscs_t {
long liveDelay;
long timerUpdateTimestamp;
long[] tv;
string comment;
long score2;
long servingTeam;
long id;
long timerSeconds;
long timerDirection;
long score1;
}
eventMiscs_t[] eventMiscs;
long fromVersion;
long factorsVersion;
}
And I *think* my jsvar.d has magic methods to load up one of
those but since my jsvar.d just uses std.json and then builds
up junk on top of it, it will necessarily be even slower than
what you already have :S so meh.
Re: code D'ish enough? - ignore previous post with same subject
On Tuesday, 28 February 2017 at 20:49:39 UTC, crimaniak wrote: On Sunday, 26 February 2017 at 21:50:38 UTC, Jordan Wilson wrote: .map!(a => a.to!double) If lambda just calls another function you can pass it directly: == .map!(to!double) Learn something new everyday, thanks :-)
Re: Getting underlying struct for parseJSON
On Tuesday, 28 February 2017 at 20:48:33 UTC, Petar Kirov [ZombineDev] wrote: On Tuesday, 28 February 2017 at 20:27:25 UTC, Alexey H wrote: Hello, guys! I'm working on a project that involves parsing of huge JSON datasets in real-time. Just an example of what i'm dealing with is here: https://gist.githubusercontent.com/gdmka/125014058bb7d7f01b867fac56300a61/raw/f0c6b5be5fb01b16dd83f07c577b72f76f72c855/data.json Can't think of any tools other that D or Go to solve this problem. My experience of solving the problem with Go has led me to Stackoverflow and the community out there seemed too reluctant to help so i assumed that the language cannot handle such a set of operations on complex datastructures. My experience with D was like a charm. Where i have had ~100 lines of Go code i did the same with 12 in with D. But, nevertheless, i did some profiling (unfortunately on OS X) and saw much heavier CPU usage with D than Go. Probably because the Go solution was unpacking all the data strictly to struct. So, my real question is: can i actually, by any change, get the description of an underlying struct that the call to parseJSON generates? The goers have this thing https://mholt.github.io/json-to-go/ to generate structs from JSON automatically. Since D easily parses JSON by type inference, i assume it builds a JSONValue struct which holds all the fields and the data. If it is possible, then i can build a similar JSON to D tool just for the sake of saving people's time and patience. If you really care about performance, have a look this: http://forum.dlang.org/post/20151014090114.60780ad6@marco-toshiba std.json is not tuned for performance, so don't expect good results from it. I am not expecting good results from stdlib's json. As for now i just need a concise way to get 1.2-1.5 MB JSON and dump it into a struct to perform numeric computations. Since it's not the only data source i will be parsing, i need a straightforward way to generate D structs right out of the predefined JSON schema. So i am willing to sacrifice some speed for convenience at this point. Fastjson might be good when dealing with trusted input, but this is not my case.
Re: code D'ish enough? - ignore previous post with same subject
On Sunday, 26 February 2017 at 21:50:38 UTC, Jordan Wilson wrote: .map!(a => a.to!double) If lambda just calls another function you can pass it directly: == .map!(to!double)
Re: Getting underlying struct for parseJSON
On Tuesday, 28 February 2017 at 20:27:25 UTC, Alexey H wrote: Hello, guys! I'm working on a project that involves parsing of huge JSON datasets in real-time. Just an example of what i'm dealing with is here: https://gist.githubusercontent.com/gdmka/125014058bb7d7f01b867fac56300a61/raw/f0c6b5be5fb01b16dd83f07c577b72f76f72c855/data.json Can't think of any tools other that D or Go to solve this problem. My experience of solving the problem with Go has led me to Stackoverflow and the community out there seemed too reluctant to help so i assumed that the language cannot handle such a set of operations on complex datastructures. My experience with D was like a charm. Where i have had ~100 lines of Go code i did the same with 12 in with D. But, nevertheless, i did some profiling (unfortunately on OS X) and saw much heavier CPU usage with D than Go. Probably because the Go solution was unpacking all the data strictly to struct. So, my real question is: can i actually, by any change, get the description of an underlying struct that the call to parseJSON generates? The goers have this thing https://mholt.github.io/json-to-go/ to generate structs from JSON automatically. Since D easily parses JSON by type inference, i assume it builds a JSONValue struct which holds all the fields and the data. If it is possible, then i can build a similar JSON to D tool just for the sake of saving people's time and patience. If you really care about performance, have a look this: http://forum.dlang.org/post/20151014090114.60780ad6@marco-toshiba std.json is not tuned for performance, so don't expect good results from it.
Getting underlying struct for parseJSON
Hello, guys! I'm working on a project that involves parsing of huge JSON datasets in real-time. Just an example of what i'm dealing with is here: https://gist.githubusercontent.com/gdmka/125014058bb7d7f01b867fac56300a61/raw/f0c6b5be5fb01b16dd83f07c577b72f76f72c855/data.json Can't think of any tools other that D or Go to solve this problem. My experience of solving the problem with Go has led me to Stackoverflow and the community out there seemed too reluctant to help so i assumed that the language cannot handle such a set of operations on complex datastructures. My experience with D was like a charm. Where i have had ~100 lines of Go code i did the same with 12 in with D. But, nevertheless, i did some profiling (unfortunately on OS X) and saw much heavier CPU usage with D than Go. Probably because the Go solution was unpacking all the data strictly to struct. So, my real question is: can i actually, by any change, get the description of an underlying struct that the call to parseJSON generates? The goers have this thing https://mholt.github.io/json-to-go/ to generate structs from JSON automatically. Since D easily parses JSON by type inference, i assume it builds a JSONValue struct which holds all the fields and the data. If it is possible, then i can build a similar JSON to D tool just for the sake of saving people's time and patience.
Mixing libraries
Hello, Been trying to learn the Simple Fast Multimedia Library (SFML) using the Derelict bindings, and noticed some functionality is offered by both SFML and the std library (for example, sfClock and sfMutex). Is there a general design principle of, say, use the std library whenever possible and only SFML when I have too? Or should I try to be consistent and use SFML library whenever possible? Thanks, Jordan
Re: foreach for string[string]AA
On Tuesday, 28 February 2017 at 17:16:43 UTC, Daniel Kozák wrote: V Tue, 28 Feb 2017 15:15:00 + Anton Pastukhov via Digitalmars-d-learn napsáno: I can't see the logic in AA foreach order. Consider this code: ... Output: three two one four I was sure output should be one two three four https://forum.dlang.org/post/xbanhtkvrizyqjcib...@forum.dlang.org Thank you for the link, it was informative reading. It's a pity that still there is no ordered AA at least as a library type.
Re: foreach for string[string]AA
V Tue, 28 Feb 2017 15:15:00 + Anton Pastukhov via Digitalmars-d-learn napsáno: > I can't see the logic in AA foreach order. Consider this code: > ... > Output: > three > two > one > four > > I was sure output should be > one > two > three > four https://forum.dlang.org/post/xbanhtkvrizyqjcib...@forum.dlang.org
Building a project with CMAKE
I'm using CMAKE to build my project. With https://github.com/dcarp/cmake-d this works almost. The only thing I do not manage to get working is running cmake in release mode. When I use -DCMAKE_BUILD_TYPE=Release I get some linker errors, which I do not get, when compiling manually. (In both cases gdc/linux is used.) I allready could figure out, that it's due to the -O3 compiler flag used in release mode. It looks like cmake compiles all *.d files separately and while doing so, removes some functions when optimizing them. Later, when linking everything together, they are missing. (But I'm not sure on this.) Here is my project: https://github.com/crocopaw/croco/tree/devel Anybody here who has experience with CMAKE and D and could help?
Re: foreach for string[string]AA
On Tuesday, 28 February 2017 at 15:44:46 UTC, bachmeier wrote:
On Tuesday, 28 February 2017 at 15:33:46 UTC, ikod wrote:
AA implemented as hash table, so it doesn't preserve insertion
order. You have to sort keys when you need:
import std.algorithm;
import std.stdio;
void main() {
auto aa = ["one":1,
"two":2
];
foreach(k; sort(aa.keys)) {
writeln(k);
}
}
That will only work if sorting recovers the insertion order. An
easy way to save the insertion order would be to use an array
of structs. If an associate array is really needed, you can
create a struct that contains the associative array and a
string[] holding the keys.
Thank you for quick replies. I'm used to arrays in PHP that are
actually ordered maps, so I was expected the same from D's AAs.
For now I'm fine with array of structs.
Re: foreach for string[string]AA
On Tuesday, 28 February 2017 at 15:33:46 UTC, ikod wrote:
AA implemented as hash table, so it doesn't preserve insertion
order. You have to sort keys when you need:
import std.algorithm;
import std.stdio;
void main() {
auto aa = ["one":1,
"two":2
];
foreach(k; sort(aa.keys)) {
writeln(k);
}
}
That will only work if sorting recovers the insertion order. An
easy way to save the insertion order would be to use an array of
structs. If an associate array is really needed, you can create a
struct that contains the associative array and a string[] holding
the keys.
Re: foreach for string[string]AA
On Tuesday, 28 February 2017 at 15:15:00 UTC, Anton Pastukhov
wrote:
I can't see the logic in AA foreach order. Consider this code:
```
void main() {
string[string] test = [
"one": "1",
"two": "2",
"three": "3",
"four": "4"
];
import std.stdio:writeln;
foreach(k, v; test) {
writeln(k);
}
}
Output:
three
two
one
four
I was sure output should be
one
two
three
four
AA implemented as hash table, so it doesn't preserve insertion
order. You have to sort keys when you need:
import std.algorithm;
import std.stdio;
void main() {
auto aa = ["one":1,
"two":2
];
foreach(k; sort(aa.keys)) {
writeln(k);
}
}
foreach for string[string]AA
I can't see the logic in AA foreach order. Consider this code:
```
void main() {
string[string] test = [
"one": "1",
"two": "2",
"three": "3",
"four": "4"
];
import std.stdio:writeln;
foreach(k, v; test) {
writeln(k);
}
}
Output:
three
two
one
four
I was sure output should be
one
two
three
four
Re: Calling destroy on struct pointer
On Saturday, 25 February 2017 at 16:39:18 UTC, Moritz Maxeiner wrote: On Saturday, 25 February 2017 at 15:13:27 UTC, Radu wrote: [...] Thanks for the example. [...] Hm, that's an issue you'd best take up to the bugtracker, I think. Maybe there's a way around that, but I don't know. [...] Well, yes, but that is still backwards-incompatible and breaking user code is something I was under the impression was a big NO currently. Made a bug report on the fwd reference error. https://issues.dlang.org/show_bug.cgi?id=17230
Re: How to enforce compile time evaluation (and test if it was done at compile time)
On Tuesday, 28 February 2017 at 07:41:36 UTC, Christian Köstlin wrote: As I understand the only difference between assert and enforce is, that assert is not compiled into releases? Thanks! Christian Pretty much so. The intention is that assert means something that's supposed to be true (and can be assumed in release) while enforce means something you want to be true (but can't guarantee). E.g., you can assert that a list is sorted after running heapsort on it, but you need to enforce that a file is in the correct format.
