Re: How do I use CTFE to generate an immutable associative array at compile time?
On Wed, Feb 22, 2017 at 06:17:14PM +, Stefan Koch via Digitalmars-d-learn wrote: > On Wednesday, 22 February 2017 at 17:05:17 UTC, H. S. Teoh wrote: > > On Wed, Feb 22, 2017 at 04:08:45PM +, Stefan Koch via > > Digitalmars-d-learn wrote: > > > [...] > > > > I'm not sure it's that simple. Just because AA's become CTFEable > > doesn't mean they will automatically be convertible to object code > > in the executable that allows runtime AA lookups. For instance, a > > CTFE pointer will have a value that has no correspondence with the > > object code in the executable, and neither will memory allocated > > during CTFE have any correspondence with the emitted object code > > (because this memory is allocated in the compiler, not in the object > > code). So if the CTFE AA implementation allocates nodes for storing > > key/value pairs, they will only exist in the CTFE engine, and > > pointers to them will only make sense within the CTFE engine. In > > order to make them work in the executable (so that you can do AA > > lookups to these computed nodes at runtime), you will need to > > somehow map them to object code. Just because AA's become CTFEable > > will not automatically solve this for you. > > > > [...] > > The CTFE allocated memory can be copied into the object file verbatim > because the structure is the same. Pointers are easy to fix up as > well, because by definition the have to point to static data that will > be in the object file. This will fail for cross-compilation. Unless you mean that the new CTFE engine is emulating the target machine directly? T -- Computers shouldn't beep through the keyhole.
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Wednesday, 22 February 2017 at 17:05:17 UTC, H. S. Teoh wrote: On Wed, Feb 22, 2017 at 04:08:45PM +, Stefan Koch via Digitalmars-d-learn wrote: [...] I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you. [...] The CTFE allocated memory can be copied into the object file verbatim because the structure is the same. Pointers are easy to fix up as well, because by definition the have to point to static data that will be in the object file.
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Wed, Feb 22, 2017 at 04:08:45PM +, Stefan Koch via Digitalmars-d-learn wrote: > On Wednesday, 22 February 2017 at 15:27:22 UTC, H. S. Teoh wrote: > > (In fact, now I'm wondering if we could just hack dmd to emit the > > equivalent of this code as a lowering, whenever the user tries to > > declare a compile-time initialized AA.) > > All the problems disappear if the AA's are compiler visible and > CTFEable code. I'm not sure it's that simple. Just because AA's become CTFEable doesn't mean they will automatically be convertible to object code in the executable that allows runtime AA lookups. For instance, a CTFE pointer will have a value that has no correspondence with the object code in the executable, and neither will memory allocated during CTFE have any correspondence with the emitted object code (because this memory is allocated in the compiler, not in the object code). So if the CTFE AA implementation allocates nodes for storing key/value pairs, they will only exist in the CTFE engine, and pointers to them will only make sense within the CTFE engine. In order to make them work in the executable (so that you can do AA lookups to these computed nodes at runtime), you will need to somehow map them to object code. Just because AA's become CTFEable will not automatically solve this for you. > There is as far as I know a project by Martin Nowak that does that. > And uses templates for AA's rather then whacky TypeInfos. > > So in the future this problem will disappear. Oh I know, the problem is that this "future" has been taking a long time arriving. I was involved in an early (but unfortunately unsuccessful) effort to rewrite AA's as library code. But there was simply too much compiler magic involved with AA's that it couldn't work at the time. Much of this magic has been dispelled over the past few years, though, so we should be in better shape now for moving AA's completely into the library. I'm very much looking forward to Martin's implementation when it's ready. But in the meantime, lowering compile-time initialized AA's could have the above hack as a temporary workaround until we can get AA literals to be embeddable in object code. T -- GEEK = Gatherer of Extremely Enlightening Knowledge
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Wednesday, 22 February 2017 at 15:27:22 UTC, H. S. Teoh wrote: (In fact, now I'm wondering if we could just hack dmd to emit the equivalent of this code as a lowering, whenever the user tries to declare a compile-time initialized AA.) All the problems disappear if the AA's are compiler visible and CTFEable code. There is as far as I know a project by Martin Nowak that does that. And uses templates for AA's rather then whacky TypeInfos. So in the future this problem will disappear.
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Wed, Feb 22, 2017 at 09:00:36AM +0100, Jacob Carlborg via
Digitalmars-d-learn wrote:
[...]
> You can use an enum to declare the AA and then assign it to an
> immutable variable using "static this". The you would only use to the
> immutable variable and never the enum.
>
> enum aa = [1 : 2];
>
> immutable int[int] iaa;
>
> static this()
> {
> iaa = aa;
> }
[...]
Wow, this is miles better than the hacks that I've come up with!
Also makes the no-static-AA issue much less of a problem that I thought
it was. (In fact, now I'm wondering if we could just hack dmd to emit
the equivalent of this code as a lowering, whenever the user tries to
declare a compile-time initialized AA.)
T
--
Computers are like a jungle: they have monitor lizards, rams, mice, c-moss,
binary trees... and bugs.
Re: How do I use CTFE to generate an immutable associative array at compile time?
Yes this is how I mean it.
Dne 22. 2. 2017 9:05 napsal uživatel "Jacob Carlborg via
Digitalmars-d-learn" :
> On 2017-02-21 23:49, H. S. Teoh via Digitalmars-d-learn wrote:
>
> That may appear to work, but I would *strongly* recommend against it,
>> because what happens when you use enum with an AA, is that the AA will
>> be created *at runtime*, *every single time* it is referenced. (It is
>> as if you copy-n-pasted the entire AA into the code each time you
>> reference the enum.) Which will introduce ridiculous amounts of
>> redundant work at runtime and cause a big performance penalty.
>>
>
> You can use an enum to declare the AA and then assign it to an immutable
> variable using "static this". The you would only use to the immutable
> variable and never the enum.
>
> enum aa = [1 : 2];
>
> immutable int[int] iaa;
>
> static this()
> {
> iaa = aa;
> }
>
> --
> /Jacob Carlborg
>
Re: How do I use CTFE to generate an immutable associative array at compile time?
On 2017-02-21 23:49, H. S. Teoh via Digitalmars-d-learn wrote:
That may appear to work, but I would *strongly* recommend against it,
because what happens when you use enum with an AA, is that the AA will
be created *at runtime*, *every single time* it is referenced. (It is
as if you copy-n-pasted the entire AA into the code each time you
reference the enum.) Which will introduce ridiculous amounts of
redundant work at runtime and cause a big performance penalty.
You can use an enum to declare the AA and then assign it to an immutable
variable using "static this". The you would only use to the immutable
variable and never the enum.
enum aa = [1 : 2];
immutable int[int] iaa;
static this()
{
iaa = aa;
}
--
/Jacob Carlborg
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tuesday, 21 February 2017 at 22:34:57 UTC, Chad Joan wrote:
In this case the AA isn't actually coded into the executable;
but at least the configuration from some_data.csv will be in
the executable as a string. The program will construct the AA
at startup. It's not as "cool", but it should get the job done.
I have a partial static AA implementation that seems like it
works, I mentioned this in a different thread.
https://github.com/rtcvb32/Side-Projects/blob/master/staticaa.d
Try it out, etc.
Usage:
Create your AA as an enum (for simplicity)
StaticAA!(KeyType, ValueType, getAALen(EnumAssosiativeArray),
EnumAssosiativeArray.length)(EnumAssosiativeArray);
Afterwards use it as you normally would for the same thing.
Unittest example:
enum AA = ["one":1, "two":2, "three":3, "four":4, "five":5,
"six":6, "seven":7, "eight":8, "nine":9, "zero":0];
auto SAA = StaticAA!(string, int, getAALen(AA), AA.length)(AA);
//just verifies the keys/values match.
foreach(k, v; AA) {
assert(SAA[k] == v);
}
Note: getAALen basically tests the array expanding it out until
none of the hashes overlap or causes problems. Trying to compile
these into a single call I've had issues, so if anyone has a
better solution I'd go for it.
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Wed, Feb 22, 2017 at 12:38:47AM +, Chad Joan via Digitalmars-d-learn
wrote:
[...]
Hmm. It's actually not necessary to manually write a foreach loop to
convert an array to an AA. We could, instead, change parseTwoColumnCsv()
to return an array of std.typecons.Tuple instead (which, incidentally,
means you can elide that foreach loop too!), then there's this handy
function std.array.assocArray that will do the transcription for us,
as follows.
(In fact, you can merge AA's without ever needing to write `foreach` at
all! See below.)
---
pure private auto parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
return csvReader!(Tuple!(string,string))(inputCsv);
}
immutable string[string] dataLookup;
immutable string[string] dataLookup1;
immutable string[string] dataLookup2;
static this()
{
import std.array : assocArray;
import std.range : chain;
// Force CTFE
immutable tuples1 = parseTwoColumnCsv("some_data1.csv");
immutable tuples2 = parseTwoColumnCsv("some_data2.csv");
dataLookup1 = tuples1.assocArray; // Bam! :-P
dataLookup2 = tuples2.assocArray; // Bam! :-P
// Bam, bam! - merge AA's in a single step!
dataLookup3 = chain(tuples1, tuples2).assocArray;
}
---
T
--
You have to expect the unexpected. -- RL
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tuesday, 21 February 2017 at 23:30:52 UTC, Chad Joan wrote:
On Tuesday, 21 February 2017 at 22:43:15 UTC, H. S. Teoh wrote:
Parsing strings at program startup is ugly and slow. What
about parsing at compile-time with CTFE into an array literal
and transforming that into an AA at startup, which should be a
lot faster?
// Warning: untested code
pure string[2][] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[2][] result;
foreach (record;
csvReader!(Tuple!(string,string))(inputCsv)) {
result ~= [record[0], record[1]];
}
return result;
}
immutable string[string] dataLookup;
static this()
{
enum halfCookedData =
parseTwoColumnCsv(import("some_data.csv"));
foreach (p; halfCookedData) {
dataLookup[p[0]] = p[1];
}
}
T
Hi,
That makes a lot of sense and it had crossed my mind.
In my case the data sets are super small, so I'm probably going
to be lazy/productive and leave it the way it is.
Anyone else from the internet copying these examples: try to
just do it H.S. Teoh's way from the start ;)
Thanks for the suggestion.
- Chad
OK I couldn't help it:
---
pure private string[][] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[][] result;
foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
result ~= [record[0],record[1]];
return result;
}
pure private string[string] twoColumnArrayToAA(const string[][]
arr)
{
string[string] result;
foreach ( pair; arr )
result[pair[0]] = pair[1];
return result;
}
pure private string[string] importTwoColumnCsv(string csvFile)()
{
// Force the parse to happen at compile time.
immutable string[][] tempArray =
import(csvFile).parseTwoColumnCsv();
// Convert the parsed array into a runtime Associative Array and
return it.
return tempArray.twoColumnArrayToAA();
}
immutable string[string] dataLookup;
immutable string[string] dataLookup1;
immutable string[string] dataLookup2;
static this()
{
import std.range;
dataLookup1 = importTwoColumnCsv!"some_data1.csv";
dataLookup2 = importTwoColumnCsv!"some_data2.csv";
foreach( pair; chain(dataLookup1.byKeyValue,
dataLookup2.byKeyValue) )
dataLookup[pair.key] = pair.value;
}
void main()
{
import std.stdio;
writefln("dataLookup = %s", dataLookup);
}
---
This example also shows joining associative arrays.
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tuesday, 21 February 2017 at 22:43:15 UTC, H. S. Teoh wrote:
Parsing strings at program startup is ugly and slow. What
about parsing at compile-time with CTFE into an array literal
and transforming that into an AA at startup, which should be a
lot faster?
// Warning: untested code
pure string[2][] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[2][] result;
foreach (record; csvReader!(Tuple!(string,string))(inputCsv))
{
result ~= [record[0], record[1]];
}
return result;
}
immutable string[string] dataLookup;
static this()
{
enum halfCookedData =
parseTwoColumnCsv(import("some_data.csv"));
foreach (p; halfCookedData) {
dataLookup[p[0]] = p[1];
}
}
T
Hi,
That makes a lot of sense and it had crossed my mind.
In my case the data sets are super small, so I'm probably going
to be lazy/productive and leave it the way it is.
Anyone else from the internet copying these examples: try to just
do it H.S. Teoh's way from the start ;)
Thanks for the suggestion.
- Chad
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tue, Feb 21, 2017 at 11:50:02PM +0100, Daniel Kozak via Digitalmars-d-learn wrote: > I have similar issue and I beleive I was able to workaround that > somehow, but it is so many years now :(. Have you tried enum > dataLookup instead of immutable string[string] dataLookup That may appear to work, but I would *strongly* recommend against it, because what happens when you use enum with an AA, is that the AA will be created *at runtime*, *every single time* it is referenced. (It is as if you copy-n-pasted the entire AA into the code each time you reference the enum.) Which will introduce ridiculous amounts of redundant work at runtime and cause a big performance penalty. T -- What did the alien say to Schubert? "Take me to your lieder."
Re: How do I use CTFE to generate an immutable associative array at compile time?
I have similar issue and I beleive I was able to workaround that
somehow, but it is so many years now :(. Have you tried enum dataLookup
instead of immutable string[string] dataLookup
Dne 21.2.2017 v 22:53 Chad Joan via Digitalmars-d-learn napsal(a):
Hello all,
I'm trying to make this work:
---
pure string[string] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[string] result;
foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
result[record[0]] = record[1];
return result;
}
immutable string[string] dataLookup =
parseTwoColumnCsv(import("some_data.csv"));
void main()
{
import std.stdio;
writefln("dataLookup = %s", dataLookup);
}
---
But (with DMD 2.073.1) I am getting this error:
main.d(14): Error: non-constant expression [['a', 'b', 'c']:['x', 'y',
'z'], ['1', '2', '3']:['4', '5', '6']]
The case with normal (non-associative) arrays seems to work fine, but
is not what I needed:
---
pure string[][] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[][] result;
foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
result ~= [record[0],record[1]];
return result;
}
immutable string[][] dataLookup =
parseTwoColumnCsv(import("some_data.csv"));
void main()
{
import std.stdio;
writefln("dataLookup = %s", dataLookup);
}
---
Any idea how I can get this working?
I have tried a couple other things, like having the parseTwoColumnCsv
function return an immutable string[string] and casting 'result' to
that in the return statement, and I also tried just casting the value
returned from parseTwoColumnCsv as it appears in the declaration of
'dataLookup'. I tried std.exception's assumeUnique, but the
function/template signature doesn't seem to support associative arrays.
Thanks.
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tue, Feb 21, 2017 at 10:34:57PM +, Chad Joan via Digitalmars-d-learn
wrote:
> On Tuesday, 21 February 2017 at 22:26:01 UTC, Chad Joan wrote:
> > On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:
> > > On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:
> > > > Hello all,
> > > >
> > > > I'm trying to make this work:
> > > >
> > > > [...]
> > >
> > > You cannot create AA's at ctfe and carry them over to runtime use.
> > > You'd have to use a costum dictionary-type.
> > > I think the vibe.d project has one you could use.
> >
> > I wondered if this might be the case.
> >
> > Well, I won't worry about it then. I'll have to find another way
> > around.
> >
> > Thanks a bunch!
>
> For the sake of the rest of the internet, here is what I'm probably going to
> stick with:
>
> ---
> pure string[string] parseTwoColumnCsv(string inputCsv)
> {
> import std.csv;
> import std.typecons;
>
> string[string] result;
>
> foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
> result[record[0]] = record[1];
>
> return result;
> }
>
> immutable string[string] dataLookup;
>
> static this()
> {
> dataLookup = parseTwoColumnCsv(import("some_data.csv"));
> }
>
> void main()
> {
> import std.stdio;
> writefln("dataLookup = %s", dataLookup);
> }
> ---
>
> In this case the AA isn't actually coded into the executable; but at least
> the configuration from some_data.csv will be in the executable as a string.
> The program will construct the AA at startup. It's not as "cool", but it
> should get the job done.
[...]
Parsing strings at program startup is ugly and slow. What about parsing
at compile-time with CTFE into an array literal and transforming that
into an AA at startup, which should be a lot faster?
// Warning: untested code
pure string[2][] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[2][] result;
foreach (record; csvReader!(Tuple!(string,string))(inputCsv)) {
result ~= [record[0], record[1]];
}
return result;
}
immutable string[string] dataLookup;
static this()
{
enum halfCookedData =
parseTwoColumnCsv(import("some_data.csv"));
foreach (p; halfCookedData) {
dataLookup[p[0]] = p[1];
}
}
T
--
Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at
it. -- Pete Bleackley
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tuesday, 21 February 2017 at 22:26:01 UTC, Chad Joan wrote:
On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote:
On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote:
Hello all,
I'm trying to make this work:
[...]
You cannot create AA's at ctfe and carry them over to runtime
use.
You'd have to use a costum dictionary-type.
I think the vibe.d project has one you could use.
I wondered if this might be the case.
Well, I won't worry about it then. I'll have to find another
way around.
Thanks a bunch!
For the sake of the rest of the internet, here is what I'm
probably going to stick with:
---
pure string[string] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[string] result;
foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
result[record[0]] = record[1];
return result;
}
immutable string[string] dataLookup;
static this()
{
dataLookup = parseTwoColumnCsv(import("some_data.csv"));
}
void main()
{
import std.stdio;
writefln("dataLookup = %s", dataLookup);
}
---
In this case the AA isn't actually coded into the executable; but
at least the configuration from some_data.csv will be in the
executable as a string. The program will construct the AA at
startup. It's not as "cool", but it should get the job done.
HTH.
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tuesday, 21 February 2017 at 21:58:07 UTC, Stefan Koch wrote: On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote: Hello all, I'm trying to make this work: [...] You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use. I wondered if this might be the case. Well, I won't worry about it then. I'll have to find another way around. Thanks a bunch!
Re: How do I use CTFE to generate an immutable associative array at compile time?
On Tuesday, 21 February 2017 at 21:53:23 UTC, Chad Joan wrote: Hello all, I'm trying to make this work: [...] You cannot create AA's at ctfe and carry them over to runtime use. You'd have to use a costum dictionary-type. I think the vibe.d project has one you could use.
How do I use CTFE to generate an immutable associative array at compile time?
Hello all,
I'm trying to make this work:
---
pure string[string] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[string] result;
foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
result[record[0]] = record[1];
return result;
}
immutable string[string] dataLookup =
parseTwoColumnCsv(import("some_data.csv"));
void main()
{
import std.stdio;
writefln("dataLookup = %s", dataLookup);
}
---
But (with DMD 2.073.1) I am getting this error:
main.d(14): Error: non-constant expression [['a', 'b', 'c']:['x',
'y', 'z'], ['1', '2', '3']:['4', '5', '6']]
The case with normal (non-associative) arrays seems to work fine,
but is not what I needed:
---
pure string[][] parseTwoColumnCsv(string inputCsv)
{
import std.csv;
import std.typecons;
string[][] result;
foreach ( record; csvReader!(Tuple!(string,string))(inputCsv) )
result ~= [record[0],record[1]];
return result;
}
immutable string[][] dataLookup =
parseTwoColumnCsv(import("some_data.csv"));
void main()
{
import std.stdio;
writefln("dataLookup = %s", dataLookup);
}
---
Any idea how I can get this working?
I have tried a couple other things, like having the
parseTwoColumnCsv function return an immutable string[string] and
casting 'result' to that in the return statement, and I also
tried just casting the value returned from parseTwoColumnCsv as
it appears in the declaration of 'dataLookup'. I tried
std.exception's assumeUnique, but the function/template signature
doesn't seem to support associative arrays.
Thanks.
