Re: How much to do
bearophile wrote:
> Recently lot of work has being done about "inout", and I think it is now
> usable in D2.
>
> So this has made me ask how much needs to be done (in D language and/or
> Phobos) to allow the correct compilation of exactly this useless demo
> program (I think it is correct):
>
>
> import std.algorithm, std.range, std.array;
> auto foo(in int[] data) pure {
> immutable int n = count!q{ a % 2 == 1 }(data);
> return map!q{ a * 2 }([n, n+1, n+2]);
> }
> void main() {
> auto a = array(iota(10));
> assert(equal(foo(a), [10, 12, 14]));
> }
>
>
> Currently count can't digest a const array and that map isn't pure. It
> runs if you remove "in" and "pure".
>
> Once this program compiles I think std.algorithm becomes significantly more
> usable.
>
> Bye,
> bearophile
Allow a template argument to treat a const(T[]) as a const(T)[] is a
different and a known problem IIRC.
Re: how to build up the library..
On Fri, 2011-10-07 at 08:49 +0200, Jacob Carlborg wrote:
[ . . . ]
> I think it's important to support the range interface (or if it's
> simpler, the array interface). I think ActiveRecord has a good high
> level API which allows to lazily evaluate SQL queries.
>
> persons = Person.all # no query is performed here
> persons = persons.where("name = 'Joe'") # or here
>
> persons.each do |p| # The actual query is performed here
> # do something with each Person
> end
>
> I don't think it would be strange to see something like this:
>
> Person.where("age > 10").map{ |p| p.name } # using "map" on an SQL query
Not just Ruby an Rails with ActiveRecord. Groovy/Grails/GORM does what
is basically the same. Python cannot do the same in the same way Ruby
and Groovy can, but it has it equivalent (augmented with SQLAlchemy).
Basically it comes down to constructing dynamic queries using the
dynamic and MOP nature of the languages. The crucial keyword here is
Builder. The trick is that what looks like function call is actually
interpreted via the builder/MOP as the construction of a data structure,
which then creates the query on demand -- with template placeholder
filling as required.
Does D have the reflection capabilities to do this in any way, shape or
form similar tp the way the dynamic languages do it?
--
Russel.
=
Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@russel.org.uk
London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
signature.asc
Description: This is a digitally signed message part
Re: how to build up the library..
On Fri, 2011-10-07 at 08:51 +0200, Jacob Carlborg wrote: [ . . . ] > I think it's important to have a solid low level database API which an > object oriented abstraction can be built upon. Why object oriented? Why not functional? The lesson form ORMs is that object oriented and relational can be linked with an adaptor but the impedance mismatch is high. But if you want to use an ORM use one, of course. The computational model having issued a query is just data transformation which is nicely functional, albeit possibly with side effects. But then the FP crowd have given everyone monads. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: How much to do
kennytm: > Allow a template argument to treat a const(T[]) as a const(T)[] is a > different and a known problem IIRC. I don't fully understand the meaning of your answer, but I didn't want to imply that those two problems are new or unknown. Bye, bearophile
Re: D on GDC announced on reddit
== Quote from Andrei Alexandrescu (seewebsiteforem...@erdani.org)'s article > http://www.reddit.com/r/programming/comments/l3tk5/merging_in_the_gnu_d_language_compiler_to_gcc/ > Andrei Cool beans. Iain
Re: Color your terminal's output
Jens Mueller wrote: >Hi, > >I started writing a simple module to color terminal output some time >ago. In a recent thread people seemed interested in having such >functionality. I cleaned up this code and kindly ask whether such a >module is considered a useful addition. > >On Posix systems it uses 4 Curses functions and on Windows systems the >Windows API is used. I tested it on Linux (using different terminal >emulators) and on Windows XP. >It allows setting foreground and background colors and setting bold, >underline, reverse and blink font faces. >Get the code from >https://raw.github.com/jkm/phobos/terminal/std/terminal.d > >To test (hopefully filling your terminal with colored output) run >on Posix >32 bit >$ dmd -unittest -m32 /usr/lib/libncurses.a -run terminal.d >64 bit >$ dmd -unittest -m64 /usr/lib/libncurses.a -run terminal.d > >(The library path may need to be adjusted.) > >and on Windows >$ dmd -unittest -run terminal.d > >At this point there are some issues that I need to figure out, namely: >* Is there a portable way to unset font face attributes on Posix? >* How to portably obtain the default foreground/background color on > Posix? >* How to properly test such a module? >* Possible license problems: I have no idea whether it's allowed to >link > against whatever license (the curses implementation uses). In doubt I > need to use the license that I link against, I suppose. You could use ANSI codes on posix to avoid a dependency on curses: http://en.wikipedia.org/wiki/ANSI_escape_code#Colors But I think using curses is ok. ncurses is MIT licensed and can be used as a dynamic library, so I don't think there are license problems. However, I'd recommend to load ncurses dynamically with dlopen/dlsym and fallback to simple text output if the ncurses library cannot be loaded. >Any help is very appreciated. > >Though this module is functionality-wise inferior to something like >ncurses it conveniently allows coloring output for most use cases. as you already use these functions: http://linux.die.net/man/3/setupterm it'd be nice to have wget-like progressbars and 'updateable' text labels. Shouldn't be as fancy as full ncurses, for most use cases it's good enough to modify the current line. +Points if it properly handles terminal width and resizing. >Jens -- Johannes Pfau
Re: D on GDC announced on reddit
On 10/6/2011 9:39 PM, Andrei Alexandrescu wrote: http://www.reddit.com/r/programming/comments/l3tk5/merging_in_the_gnu_d_language_compiler_to_gcc/ On Hacker News too: http://news.ycombinator.com/item?id=3083125
My ignorance, or a bug - help please
I was converting something that was a class to a struct, and ran into this
(minimized as much as possible):
import mysql;
// The import mysql.d for the test contains only:
/*
extern (C):
struct MYSQL{}
MYSQL* mysql_init(MYSQL* mysql);
*/
// If I put this stuff in-line the error does not happen
struct Connection
{
MYSQL _mysql;
~this()
{
}
this(int dummy = 0)
{
mysql_init(&_mysql);
}
}
// dmd -c mysqld.d (2.055 - Ubuntu)
//
// Error: can only initialize const member _mysql inside constructor
// Error: this is not mutable
Does anyone have any idea what's going on?
Thanks
Steve
Re: My ignorance, or a bug - help please
-- mysql.d
extern (C):
struct MYSQL{}
MYSQL* mysql_init(MYSQL* mysql);
-- main.d -
import mysql;
struct Connection
{
MYSQL _mysql;
~this()
{
}
this(int dummy = 0)
{
mysql_init(&_mysql);
}
}
void main()
{
Connection con1;
Connection *con2 = new Connection(123);
}
-
dmd main.d mysql.d -L-lmysqlclient
works with DMD64 D Compiler v2.056
Re: My ignorance, or a bug - help please
If you don't need the internal data of struct MYSQL and you don't want
to care about. Then just keep some reference in your program.
alias void MYSQL;
alias void MYSQL_RES;
and then only use MYSQL* and pass it to every function.
struct Connection
{
MYSQL* _mysql;
~this()
{
}
this(int dummy = 0)
{
mysql_init(_mysql);
}
}
Re: My ignorance, or a bug - help please
Steve Teale , dans le message (digitalmars.D:146210), a écrit :
> I was converting something that was a class to a struct, and ran into this
> (minimized as much as possible):
>
> import mysql;
> // The import mysql.d for the test contains only:
> /*
> extern (C):
> struct MYSQL{}
> MYSQL* mysql_init(MYSQL* mysql);
> */
> // If I put this stuff in-line the error does not happen
>
> struct Connection
> {
>MYSQL _mysql;
>
>~this()
>{
>}
>
>this(int dummy = 0)
>{
> mysql_init(&_mysql);
>}
> }
>
> // dmd -c mysqld.d (2.055 - Ubuntu)
> //
> // Error: can only initialize const member _mysql inside constructor
> // Error: this is not mutable
>
> Does anyone have any idea what's going on?
I don't know what is going one, but putting a MYSQL object in Connection
after having defined MYSQL as a dummy empty extern C struct does not
seem to be a good idea.
--
Christophe
Re: how to build up the library..
Am 07.10.2011, 09:26 Uhr, schrieb Russel Winder :
On Fri, 2011-10-07 at 08:49 +0200, Jacob Carlborg wrote:
[ . . . ]
I think it's important to support the range interface (or if it's
simpler, the array interface). I think ActiveRecord has a good high
level API which allows to lazily evaluate SQL queries.
persons = Person.all # no query is performed here
persons = persons.where("name = 'Joe'") # or here
persons.each do |p| # The actual query is performed here
# do something with each Person
end
I don't think it would be strange to see something like this:
Person.where("age > 10").map{ |p| p.name } # using "map" on an SQL query
Not just Ruby an Rails with ActiveRecord. Groovy/Grails/GORM does what
is basically the same. Python cannot do the same in the same way Ruby
and Groovy can, but it has it equivalent (augmented with SQLAlchemy).
Basically it comes down to constructing dynamic queries using the
dynamic and MOP nature of the languages. The crucial keyword here is
Builder. The trick is that what looks like function call is actually
interpreted via the builder/MOP as the construction of a data structure,
which then creates the query on demand -- with template placeholder
filling as required.
Does D have the reflection capabilities to do this in any way, shape or
form similar tp the way the dynamic languages do it?
This has already been done in D. There is a nice library that assembles
queries at compile time and fetches results lazily into structs. The name?
Sorry, I forgot. But the author will probably show up here.
Re: how to build up the library..
On 2011-10-07 09:30, Russel Winder wrote: On Fri, 2011-10-07 at 08:51 +0200, Jacob Carlborg wrote: [ . . . ] I think it's important to have a solid low level database API which an object oriented abstraction can be built upon. Why object oriented? Why not functional? The point was that the low level should not be object oriented like, an ORM, to allow different high level abstractions built on top of the low level API. I don't think that the low level API should have functions which takes a struct or an object and fills it fields with results from a query. That's up to the high level API. -- /Jacob Carlborg
Re: My ignorance, or a bug - help please
The empty struct is a red herring. I cut mysql.d down to the bare minimum to help isolate the bug. I get exactly the same error with the complete mysql.d
Re: My ignorance, or a bug - help please
On 2011-10-07 11:55, mta`chrono wrote: If you don't need the internal data of struct MYSQL and you don't want to care about. Then just keep some reference in your program. alias void MYSQL; alias void MYSQL_RES; The correct way to do this is to declare an opaque struct, just as in C: struct MYSQL; -- /Jacob Carlborg
Re: how to build up the library..
On 2011-10-07 09:26, Russel Winder wrote:
On Fri, 2011-10-07 at 08:49 +0200, Jacob Carlborg wrote:
[ . . . ]
I think it's important to support the range interface (or if it's
simpler, the array interface). I think ActiveRecord has a good high
level API which allows to lazily evaluate SQL queries.
persons = Person.all # no query is performed here
persons = persons.where("name = 'Joe'") # or here
persons.each do |p| # The actual query is performed here
# do something with each Person
end
I don't think it would be strange to see something like this:
Person.where("age> 10").map{ |p| p.name } # using "map" on an SQL query
Not just Ruby an Rails with ActiveRecord. Groovy/Grails/GORM does what
is basically the same. Python cannot do the same in the same way Ruby
and Groovy can, but it has it equivalent (augmented with SQLAlchemy).
Basically it comes down to constructing dynamic queries using the
dynamic and MOP nature of the languages. The crucial keyword here is
Builder. The trick is that what looks like function call is actually
interpreted via the builder/MOP as the construction of a data structure,
which then creates the query on demand -- with template placeholder
filling as required.
Does D have the reflection capabilities to do this in any way, shape or
form similar tp the way the dynamic languages do it?
It depends on what you want to achieve. If you want just the above
example (and similar) to work, including lazy loading that would be
possible and not very hard.
If you want a more complete ORM and do things like (in Ruby):
person = Person.find_all_by_name_and_age("Joe", 15)
p person.name
That's a little harder. You could use opDispatch to make
"find_all_by_name_and_age" or "name" work, but not both. The problem is
that you can not overload methods on static and you are not forced to
call a static method prefixed with the class name.
class Foo
{
void bar () {};
static void bar () {};
}
Currently the above doesn't work. You can also call a static method on
an instance, which will also cause problems. What we want here is to be
able to have two versions of opDispatch, one static and one non-static.
--
/Jacob Carlborg
Re: how to build up the library..
On Fri, 07 Oct 2011 07:40:01 +0100, Jacob Carlborg wrote: On 2011-10-06 21:44, Jesse Phillips wrote: Regan Heath Wrote: That's just it however, I am not generous enough with my time to be of use to you/D/phobos. I spend my free time doing other things, and not working on the code I have which could be of use. I suspect there are quite a number of people out there who are just like me. People who have some code which may be of use, in varying degrees of completeness from just an idea to a pretty much complete re-usable library which just needs the final touches etc. The problem is as you say, there are many out there like you. Even if some people take the rule of getting these polished, they couldn't keep up. Finding these people his hard. Even if we just get one contribution to Phobos through this method it would be better than nothing. Exactly. The people taking ideas from the pool don't have to keep up, they just need to be someone with the time/inclination to "complete" something/anything worthy of inclusion into phobos. All the pool does is give them a leg-up or provide the technical know-how/guts they might be missing. It would be something like dsource, but for code ranging from snippets to complete packages, and would have to be well organised, or at least searchable and rely on contributors including as much info about thier snippets etc as they can. But, that's a nice small investment for someone, especially if we make it easy to contribute the code, agree to a license and add info about it which is then searchable. I realise setting something like this up is no small amount of work, and that I'm in no position to do it, but it seems on the face of it to be a good idea. The pool would serve double duty, as it would supply code examples/snippets to people learning D also, or anyone looking for examples of how to do .. well, anything that gets contributed. Each of us probably has a collection of D code on our own HDDs somewhere which does interesting and useful things, but no-one else can see it, search it, or learn from it. I'm simply suggesting we put it somewhere to share. At it's most basic it could just be a wiki page, which we let google index for us. -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Re: Color your terminal's output
You could use ANSI codes on posix to avoid a dependency on curses: http://en.wikipedia.org/wiki/ANSI_escape_code#Colors But I think using curses is ok. ncurses is MIT licensed and can be used as a dynamic library, so I don't think there are license problems. However, I'd recommend to load ncurses dynamically with dlopen/dlsym and fallback to simple text output if the ncurses library cannot be loaded. +1 There shouldn't be a hard dependency on curses.
Re: D on GDC announced on reddit
Am 07.10.2011, 11:27 Uhr, schrieb Walter Bright : On 10/6/2011 9:39 PM, Andrei Alexandrescu wrote: http://www.reddit.com/r/programming/comments/l3tk5/merging_in_the_gnu_d_language_compiler_to_gcc/ On Hacker News too: http://news.ycombinator.com/item?id=3083125 And again they immediately start with the Go vs. D discussions.
Re: D on GDC announced on reddit
That will always happen. My self I think that Go is a better C and D a better C++. It remains to be seen how much they will be able to take from C and C++. Personaly I have been spending most of the time with Go, because of its similarity with Oberon. It brings back some good memories. Now D is also quite cool, I would just like for the language compilers to be a bit more stable. Currently I do have more sucess proposing C++11 based solutions as Go or D based ones, on the type of corporate environment I work in. Anyway, nice work! "Trass3r" wrote in message news:op.v2zd1eb43ncmek@enigma... > Am 07.10.2011, 11:27 Uhr, schrieb Walter Bright > : > >> On 10/6/2011 9:39 PM, Andrei Alexandrescu wrote: >>> http://www.reddit.com/r/programming/comments/l3tk5/merging_in_the_gnu_d_language_compiler_to_gcc/ >> >> On Hacker News too: >> >> http://news.ycombinator.com/item?id=3083125 > > And again they immediately start with the Go vs. D discussions.
Re: Next in the Review Queue?
On 07.10.2011 7:49, Andrei Alexandrescu wrote: On 10/6/11 9:17 PM, Jonathan M Davis wrote: On Sunday, October 02, 2011 23:27:35 Jonathan M Davis wrote: The review for the region allocator has completed, so we need to choos something else to review now. I believe that the current items in the review queue which are ready for review are - std.log - a CSV parsing module by Jesse Phillips - std.benchmark - GSoC changes to std.regex There are a couple of other items which are in a "pre-review" state (e.g. orange) as well as a few others that are supposed to be close to being ready for review (e.g. changes to std.variant), but I'm not aware of anything else which is actually ready for review at the moment. So, if I missed something, please bring it up. The item which has been in the queue the longest is std.log, but given that std.benchmark could affect further reviews, we may want to review that first. The GSoC changes to to std.regex are also probably of fairly high priority. Any thoughts on which item should be reviewed first? Jose hasn't responded at all, so std.log obviously isn't an option at this point, even though it's been in the queue the longest. So, should we move on with the review of either the CSV parser or the GSoC changes to std.regex, or should we wait until Andrei is ready with std.benchmark next week? - Jonathan M Davis I suggest we go ahead with std.regex. Dmitry? Well, if Jesse won't mind. I'll wrap the thing up in my phobos fork in a couple of moments. Anyone going for review manager? -- Dmitry Olshansky
Re: D on GDC announced on reddit
Now D is also quite cool, I would just like for the language compilers to be a bit more stable. They have been vastly improving, really. Currently I do have more sucess proposing C++11 based solutions as Go or D based ones, on the type of corporate environment I work in. That's not D's or Go's fault. Most guys especially in bigger corporations are plain ignorant and wear blinders. Strangely that even applies to universities. Hell, they didn't even know about clang even though they were progressive enough to use C++0x.
Re: Color your terminal's output
Trass3r wrote: > >You could use ANSI codes on posix to avoid a dependency on curses: > >http://en.wikipedia.org/wiki/ANSI_escape_code#Colors > >But I think using curses is ok. ncurses is MIT licensed and can be > >used as a dynamic library, so I don't think there are license problems. > > > >However, I'd recommend to load ncurses dynamically with dlopen/dlsym > >and fallback to simple text output if the ncurses library cannot be > >loaded. > > +1 > There shouldn't be a hard dependency on curses. I had the impression that even though there is this standard how do I know that I have a standard-compliant terminal. Can I just assume this? I started using curses because I had the impression there may be non-standard terminals. But this seems to be minor issue. I will change this if people are happy with Windows and ISO/IEC 6429 compliant terminals only. Thanks. Jens
Re: Next in the Review Queue?
On 10/6/2011 11:49 PM, Andrei Alexandrescu wrote: I suggest we go ahead with std.regex. Dmitry? Andrei My only concern is that Dmitry's code uses RegionAllocator. I was hoping to get RegionAllocator into Phobos first, but that obviously didn't happen.
Re: Color your terminal's output
Johannes Pfau wrote: > Jens Mueller wrote: > >Hi, > > > >I started writing a simple module to color terminal output some time > >ago. In a recent thread people seemed interested in having such > >functionality. I cleaned up this code and kindly ask whether such a > >module is considered a useful addition. > > > >On Posix systems it uses 4 Curses functions and on Windows systems the > >Windows API is used. I tested it on Linux (using different terminal > >emulators) and on Windows XP. > >It allows setting foreground and background colors and setting bold, > >underline, reverse and blink font faces. > >Get the code from > >https://raw.github.com/jkm/phobos/terminal/std/terminal.d > > > >To test (hopefully filling your terminal with colored output) run > >on Posix > >32 bit > >$ dmd -unittest -m32 /usr/lib/libncurses.a -run terminal.d > >64 bit > >$ dmd -unittest -m64 /usr/lib/libncurses.a -run terminal.d > > > >(The library path may need to be adjusted.) > > > >and on Windows > >$ dmd -unittest -run terminal.d > > > >At this point there are some issues that I need to figure out, namely: > >* Is there a portable way to unset font face attributes on Posix? > >* How to portably obtain the default foreground/background color on > > Posix? > >* How to properly test such a module? > >* Possible license problems: I have no idea whether it's allowed to > >link > > against whatever license (the curses implementation uses). In doubt I > > need to use the license that I link against, I suppose. > > You could use ANSI codes on posix to avoid a dependency on curses: > http://en.wikipedia.org/wiki/ANSI_escape_code#Colors > But I think using curses is ok. ncurses is MIT licensed and can be > used as a dynamic library, so I don't think there are license problems. > > However, I'd recommend to load ncurses dynamically with dlopen/dlsym > and fallback to simple text output if the ncurses library cannot be > loaded. Using the ANSI codes is fine with me. I assumed they aren't that portable but it seems fine. > >Any help is very appreciated. > > > >Though this module is functionality-wise inferior to something like > >ncurses it conveniently allows coloring output for most use cases. > > as you already use these functions: > http://linux.die.net/man/3/setupterm > it'd be nice to have wget-like progressbars and 'updateable' text > labels. Shouldn't be as fancy as full ncurses, for most use cases it's > good enough to modify the current line. +Points if it properly handles > terminal width and resizing. I believe progress bars are easy to add. Boost's progress_bar should be fairly easy to port. It'll be nice if you could provide a pull request for this. Is this feasible for you? Regarding update able text labels I'm not sure how they are typically used. So I would also prefer some pull request from somebody with a common use case. I'm just pushing this color support because I'm using it in some tool. And it may be useful to others. I believe it's a good thing to add the features you need for your project via a pull request. Jens
Re: Color your terminal's output
Am 07.10.2011, 14:51 Uhr, schrieb Jens Mueller : Trass3r wrote: >You could use ANSI codes on posix to avoid a dependency on curses: >http://en.wikipedia.org/wiki/ANSI_escape_code#Colors >But I think using curses is ok. ncurses is MIT licensed and can be >used as a dynamic library, so I don't think there are license problems. > >However, I'd recommend to load ncurses dynamically with dlopen/dlsym >and fallback to simple text output if the ncurses library cannot be >loaded. +1 There shouldn't be a hard dependency on curses. I had the impression that even though there is this standard how do I know that I have a standard-compliant terminal. Can I just assume this? I started using curses because I had the impression there may be non-standard terminals. But this seems to be minor issue. I will change this if people are happy with Windows and ISO/IEC 6429 compliant terminals only. Thanks. As Johannes already said, it's perfectly possible to implement both approaches and choose at runtime.
Re: Next in the Review Queue?
On 07.10.2011 17:04, dsimcha wrote: On 10/6/2011 11:49 PM, Andrei Alexandrescu wrote: I suggest we go ahead with std.regex. Dmitry? Andrei My only concern is that Dmitry's code uses RegionAllocator. I was hoping to get RegionAllocator into Phobos first, but that obviously didn't happen. I followed that review closely and did a couple of "plan B" moves ;) So now it's independent of RegionAllocator. At any rate it was used in one engine (backtracking), and it turned out that I could get away with raw malloc. Since I already kept an s-list of pages (arrays) with backtrack info, I sort of did bookmarking stuff of RegionAllocator twice. -- Dmitry Olshansky
Re: how to build up the library..
What's the big advantage of
Person.find_all_by_name_and_age("Joe", 15)
over
db.query("select * from people where name = ? and age = ?", "Joe", 15);
The latter is much easier to write in the library and you retain all
the flexibility of sql itself if needed.
Re: Color your terminal's output
Trass3r wrote: > Am 07.10.2011, 14:51 Uhr, schrieb Jens Mueller : > > >Trass3r wrote: > >>>You could use ANSI codes on posix to avoid a dependency on curses: > >>>http://en.wikipedia.org/wiki/ANSI_escape_code#Colors > >>>But I think using curses is ok. ncurses is MIT licensed and can be > >>>used as a dynamic library, so I don't think there are license problems. > >>> > >>>However, I'd recommend to load ncurses dynamically with dlopen/dlsym > >>>and fallback to simple text output if the ncurses library cannot be > >>>loaded. > >> > >>+1 > >>There shouldn't be a hard dependency on curses. > > > >I had the impression that even though there is this standard how do I > >know that I have a standard-compliant terminal. Can I just assume this? > >I started using curses because I had the impression there may be > >non-standard terminals. But this seems to be minor issue. I will change > >this if people are happy with Windows and ISO/IEC 6429 compliant > >terminals only. > >Thanks. > > As Johannes already said, it's perfectly possible to implement both > approaches and choose at runtime. I see. You mean using curses if available and falling back to ISO/IEC 6429. So you think that supporting ISO/IEC 6429 terminals is too limited, aren't you? Jens
Re: Color your terminal's output
I see. You mean using curses if available and falling back to ISO/IEC 6429. So you think that supporting ISO/IEC 6429 terminals is too limited, aren't you? Well I personally only care about bash and Windoze console. I guess these support ISO 6429?!
Database interface design - was how to build up the library.
I'm thinking that for each database that gets covered there will need to be two modules, like: etc.c.mysql etc.mysqld The etc.c.xxx modules would be completely different between databases - just translations of the necessary header files. The xxxd modules would be as similar as possible consistent with allowing the strong points of individual database systems to show through, and the weak points of others to be avoided. I don't think all should be reduced to some LCD. These modules should attempt to make a good range of capabilities available to the D programmer, but they not have to be all encompassing. Those users who want to do really fancy things can drop back to the low-level interface. They should probably have the following capabilities: 1) Establishing and closing database connections. 2) The capability to execute literal SQL statements - execSQL() if you like. Some of these will generate result sets, of which more below. 3) The capability to create prepared statements with in and out parameters and association of the parameters with a source, and then to execute these. This breaks down into several components/ capabilities, which could be labeled: 3a) createPreparedStatement() - marshal parameters, associate them with a sourceand have the server prepare the statement. 3b) execStatement() - for those SQL statements that don't have a result set. 3c) execStatementIncremental()/getNext() - execute the prepared statement, then fetch the result rows one at a time into some kind of result set. 3d) execStatementAll() - execute the prepared statement and get all the resulting rows into some kind of result set. 3e) (maybe) execScalar() - do the whole sequence prepare, execute, and get a single value result set placed into a D variable. 3f) (maybe) execStoredProcedure() - another 'do the whole thing' capability TBD. It is when we come to the nature of the result sets that there is likely to be dissent. I favor arrays of structs, but we should probably do arrays of arrays of variants too for those situations where structures can't be sorted out at compile time. There needs to be some symmetry between what is used here, and what can be used as input to operations such as a prepared insert. It is of course vital that this part of each middle layer produce exactly the same kind of results. Otherwise the top layer could become difficult. On top of this set of two modules for each database, I envisage a higher-level module - etc.dbops - that provides a bunch of convenience templates for various common database operations, spanning the databases. Once the middle layer is in place, this top layer should be relatively easy to implement. It should be noted though that all these database wrappers will be monstrously difficult to test. I am at the point with MySQL where I can get the result of a plain old query into an array of a checked structure type. I have the prepared statement stuff, and know how the result will be created from a prepared query (the execStatementAll() case) - I just have to plow through a bit more binding and fetching. This is probably rather general and vague, but I would like to get comments so we can iterate toward a practical design. Thanks Steve
Re: D on GDC announced on reddit
"Trass3r" wrote in message news:op.v2ze74ma3ncmek@enigma... >> Now D is also quite cool, I would just like for the language compilers >> to be a bit more stable. > > They have been vastly improving, really. > >> Currently I do have more sucess proposing C++11 based solutions as Go or >> D based ones, on the type of corporate environment I work in. > > That's not D's or Go's fault. Most guys especially in bigger corporations > are plain ignorant and wear blinders. > Strangely that even applies to universities. Not real surprising. Universities can be *enormously* ignorant and conceited. (Community colleges too...my god, some of the flaming egos and politics around there are mind-boggling, especially considering it's *just* a CC...) > Hell, they didn't even know about clang even though they were progressive > enough to use C++0x. I once had a university professor who openly admitted C was the only language he knew - and yet he didn't even understand how C's null-terminated strings work. So he didn't really even know that one language.
Re: Color your terminal's output
"Trass3r" wrote in message news:op.v2zhq60v3ncmek@enigma... >> I see. You mean using curses if available and falling back to ISO/IEC >> 6429. So you think that supporting ISO/IEC 6429 terminals is too >> limited, aren't you? > > Well I personally only care about bash and Windoze console. I guess these > support ISO 6429?! Windows console windows don't, oddly enough. Even though MS-DOS did if you had ANSI.SYS loaded. And, maybe I'm wrong, but I would think that shell interpreter wouldn't have anything to do with whether or not ANSI escape codes are avilable. Isn't that more a matter of terminal program (ie, xterm, vs...umm, all those others...)
Re: D on GDC announced on reddit
Hell, they didn't even know about clang even though they were progressive enough to use C++0x. I once had a university professor who openly admitted C was the only language he knew - and yet he didn't even understand how C's null-terminated strings work. So he didn't really even know that one language. Had a workmate who apparently was real good at using C for microprocessor programming. But he didn't have the slightest clue about how C++ works and what OOP overhead really means. And when I used Lua to create a small conversion script (which was only needed cause the legacy code was crap, hardcoded paths and the like) that even turned into some kind of running gag. I don't need to say that nobody had ever heard of it nor was anyone willing to try it out before judging.
Re: Database interface design - was how to build up the library.
On 10/7/11 9:11 AM, Steve Teale wrote:
I'm thinking that for each database that gets covered there will
need to be two modules, like:
etc.c.mysql
etc.mysqld
The etc.c.xxx modules would be completely different between databases -
just translations of the necessary header files. The xxxd modules
would be as similar as possible consistent with allowing the strong
points of individual database systems to show through, and the weak
points of others to be avoided. I don't think all should be reduced
to some LCD.
Well we may be hasty to go that way. The driver-based approach works
well for other languages and APIs - why wouldn't it work for us? The
differences across different DBMSs would be mostly in the SQL support,
not the basic APIs.
These modules should attempt to make a good range of capabilities
available to the D programmer, but they not have to be all encompassing.
Those users who want to do really fancy things can drop back to the
low-level interface. They should probably have the following capabilities:
1) Establishing and closing database connections.
Sounds good. Since DB connections are a precious resource, there must be
a RAII struct holding them. The functions below may be members of it.
2) The capability to execute literal SQL statements - execSQL()
if you like. Some of these will generate result sets, of which more below.
Great.
3) The capability to create prepared statements with in and out
parameters and association of the parameters with a source, and
then to execute these. This breaks down into several components/
capabilities, which could be labeled:
3a) createPreparedStatement() - marshal parameters, associate them
with a sourceand have the server prepare the statement.
3b) execStatement() - for those SQL statements that don't have a
result set.
3c) execStatementIncremental()/getNext() - execute the prepared statement,
then fetch the result rows one at a time into some kind of result set.
Here's where the range interface might be used. We might simply have
execStatement() that returns an input range. If the statement produced
no rows, the range will be empty. No need for distinguishing
execStatement() and execStatementIncremental().
3d) execStatementAll() - execute the prepared statement and get all
the resulting rows into some kind of result set.
This is not a primitive, but instead a convenience function that should
be used with caution and only for queries known to be small. I agree
that it's good to have.
3e) (maybe) execScalar() - do the whole sequence prepare, execute,
and get a single value result set placed into a D variable.
That's convenient for e.g. "SELECT COUNT(*) FROM table WHERE condition",
which people run all the time.
3f) (maybe) execStoredProcedure() - another 'do the whole thing'
capability TBD.
Well wouldn't that just execute a special SQL a la "CALL procedure"?
It is when we come to the nature of the result sets that there is
likely to be dissent. I favor arrays of structs, but we should
probably do arrays of arrays of variants too for those situations
where structures can't be sorted out at compile time. There needs
to be some symmetry between what is used here, and what can be used
as input to operations such as a prepared insert. It is of course
vital that this part of each middle layer produce exactly the same
kind of results. Otherwise the top layer could become difficult.
I'd like arrays of structs too but we must acknowledge that most
databases people use will be large, in which case loading the whole
thing eagerly takes a lot of RAM and potentially wastes time. So arrays
are out except for e.g. convenience functions - but then we already have
array() that converts an arbitrary input range into an array.
Now, regarding the "structs" part, I'd like that too when the schema is
statically known. Two issues:
1. MFC had at a point a wizard that generated one struct per resultset.
It was an absolute maintenance disaster and they recanted by offering
dynamically-bound result sets. The lesson there is that defining a
struct for each query won't likely play out, so we better use
Tuple!(ColType1, ColType2, ...). A possible API would be:
auto db = std.database.connect("cdb.mysql");
auto rows = db.sql!(double, ulong)
("SELECT SUM(x), COUNT(x) FROM table");
// We know that rows is a range of Tuple!(double, ulong).
writeln("Sum is ", rows.front[0], " count is ", rows.front[1]);
Cool beans. I'd love to use such an API!
2. Statically-bound tuples work only when we know the query beforehand.
We need to cater for resultsets of unknown column count and types. The
question here is whether we traffic in untyped memory a la ubyte[] or
some variant type. I favor Variant because it's efficient, accommodates
any SQL type easily, and is convenient to use.
On top of this set of two modules for each database, I envisage a
higher-level module - etc.dbops - that provides a bunch of convenience
templates for var
Re: Database interface design - was how to build up the library.
On 07-10-2011 16:11, Steve Teale wrote: I'm thinking that for each database that gets covered there will need to be two modules, like: etc.c.mysql etc.mysqld The etc.c.xxx modules would be completely different between databases - just translations of the necessary header files. The xxxd modules would be as similar as possible consistent with allowing the strong points of individual database systems to show through, and the weak points of others to be avoided. I don't think all should be reduced to some LCD. These modules should attempt to make a good range of capabilities available to the D programmer, but they not have to be all encompassing. Those users who want to do really fancy things can drop back to the low-level interface. They should probably have the following capabilities: 1) Establishing and closing database connections. 2) The capability to execute literal SQL statements - execSQL() if you like. Some of these will generate result sets, of which more below. 3) The capability to create prepared statements with in and out parameters and association of the parameters with a source, and then to execute these. This breaks down into several components/ capabilities, which could be labeled: 3a) createPreparedStatement() - marshal parameters, associate them with a sourceand have the server prepare the statement. 3b) execStatement() - for those SQL statements that don't have a result set. 3c) execStatementIncremental()/getNext() - execute the prepared statement, then fetch the result rows one at a time into some kind of result set. 3d) execStatementAll() - execute the prepared statement and get all the resulting rows into some kind of result set. 3e) (maybe) execScalar() - do the whole sequence prepare, execute, and get a single value result set placed into a D variable. 3f) (maybe) execStoredProcedure() - another 'do the whole thing' capability TBD. It is when we come to the nature of the result sets that there is likely to be dissent. I favor arrays of structs, but we should probably do arrays of arrays of variants too for those situations where structures can't be sorted out at compile time. There needs to be some symmetry between what is used here, and what can be used as input to operations such as a prepared insert. It is of course vital that this part of each middle layer produce exactly the same kind of results. Otherwise the top layer could become difficult. On top of this set of two modules for each database, I envisage a higher-level module - etc.dbops - that provides a bunch of convenience templates for various common database operations, spanning the databases. Once the middle layer is in place, this top layer should be relatively easy to implement. It should be noted though that all these database wrappers will be monstrously difficult to test. I am at the point with MySQL where I can get the result of a plain old query into an array of a checked structure type. I have the prepared statement stuff, and know how the result will be created from a prepared query (the execStatementAll() case) - I just have to plow through a bit more binding and fetching. This is probably rather general and vague, but I would like to get comments so we can iterate toward a practical design. Thanks Steve Just a note on module naming: mysqld is misleading - it could mean "mysql daemon". I would recommend just calling it mysql and calling the C interface etc.c.mysqlc or something like this. - Alex
Re: The problem with std.conv.emplace
Am 04.10.2011 20:44, schrieb kennytm: Timon Gehr wrote: Is there already an enhancement request/bug report for this? I did mention nullptr_t in 5899. There is already a similar enhancement request: http://d.puremagic.com/issues/show_bug.cgi?id=5416 I would really appreciate a comment by Walter or Andrei on this. Without this feature it is not possible to replace the deprecated new/delete operators in a nice way. -- Kind Regards Benjamin Thaut
Sorting algorithm
Hi, it's been years since I've posted here. I just wanted to share something I worked on, a new sorting algorithm. It's a variant of merge sort, but much more memory efficient with only a small loss in performance. The most similar algorithm I know of is Timsort. I had need for a stable sorting algorithm, but the performance of stable sort in Phobos is terrible. This pretty much left merge sort, which has good performance but requires O(n) space. That persuaded me to design my own sorting algorithm. Here, you can find the code, details of the algorithm, and benchmarks (introSort is the unstable sort in Phobos). http://www.neowin.net/forum/blog/422/entry-3737-sort-algorithm-complete/
Re: Database interface design - was how to build up the library.
Andrei, The suggestion in your penultimate paragraph is what I'm trying to get started. Maybe the answer to the problem with structs may lie somewhere in std.typecons in combination with the field discovery features supported by most databases. If we can manufacture the struct to suit the circumstances, then we've cracked it! I'm probably talking rubbish - Friday night, couple of beers. I will attempt top start a new thread - [std.database] - your 10 cents worth? Thanks Steve
Re: Database interface design - was how to build up the library.
On 10/7/11 11:51 AM, Steve Teale wrote: Andrei, The suggestion in your penultimate paragraph is what I'm trying to get started. Maybe the answer to the problem with structs may lie somewhere in std.typecons in combination with the field discovery features supported by most databases. If we can manufacture the struct to suit the circumstances, then we've cracked it! I'm probably talking rubbish - Friday night, couple of beers. No can do. This is a hard wall. Structs must be defined during compilation. Database structure can be discovered at run time. What you can do, however, is that you pass a struct type to store a row in the resultset and at runtime the fields of the struct are matched dynamically with the fields in the rowset. That is doable and useful. I will attempt top start a new thread - [std.database] - your 10 cents worth? http://www.youtube.com/watch?v=lmUZGdi7Ty4 Andrei
Re: Sorting algorithm
On 10/7/11 11:42 AM, Xinok wrote: Hi, it's been years since I've posted here. I just wanted to share something I worked on, a new sorting algorithm. It's a variant of merge sort, but much more memory efficient with only a small loss in performance. The most similar algorithm I know of is Timsort. I had need for a stable sorting algorithm, but the performance of stable sort in Phobos is terrible. This pretty much left merge sort, which has good performance but requires O(n) space. That persuaded me to design my own sorting algorithm. Here, you can find the code, details of the algorithm, and benchmarks (introSort is the unstable sort in Phobos). http://www.neowin.net/forum/blog/422/entry-3737-sort-algorithm-complete/ This is interesting. What do the numbers in the benchmark represent? Andrei
Re: Database interface design - was how to build up the library.
Andrei Alexandrescu wrote:
1. MFC had at a point a wizard that generated one struct per resultset.
It was an absolute maintenance disaster and they recanted by offering
dynamically-bound result sets. The lesson there is that defining a
struct for each query won't likely play out, so we better use
Tuple!(ColType1, ColType2, ...). A possible API would be:
auto db = std.database.connect("cdb.mysql");
auto rows = db.sql!(double, ulong)
("SELECT SUM(x), COUNT(x) FROM table");
// We know that rows is a range of Tuple!(double, ulong).
writeln("Sum is ", rows.front[0], " count is ", rows.front[1]);
Cool beans. I'd love to use such an API!
Andrei
Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples,
structs and arrays in similar manner.
Re: Database interface design - was how to build up the library.
On 10/7/11 12:02 PM, Piotr Szturmaj wrote:
Andrei Alexandrescu wrote:
1. MFC had at a point a wizard that generated one struct per resultset.
It was an absolute maintenance disaster and they recanted by offering
dynamically-bound result sets. The lesson there is that defining a
struct for each query won't likely play out, so we better use
Tuple!(ColType1, ColType2, ...). A possible API would be:
auto db = std.database.connect("cdb.mysql");
auto rows = db.sql!(double, ulong)
("SELECT SUM(x), COUNT(x) FROM table");
// We know that rows is a range of Tuple!(double, ulong).
writeln("Sum is ", rows.front[0], " count is ", rows.front[1]);
Cool beans. I'd love to use such an API!
Andrei
Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples,
structs and arrays in similar manner.
I did. Looks good. Why do you need DBRow? You could simply use
Variant[], the given struct, or Tuple directly.
Andrei
Re: Database interface design - was how to build up the library.
In my database.d, I used a database row struct to provide both integer and key based indexes.
Re: Sorting algorithm
On 10/7/2011 1:03 PM, Andrei Alexandrescu wrote:
On 10/7/11 11:42 AM, Xinok wrote:
Hi, it's been years since I've posted here. I just wanted to share
something I worked on, a new sorting algorithm. It's a variant of merge
sort, but much more memory efficient with only a small loss in
performance. The most similar algorithm I know of is Timsort.
I had need for a stable sorting algorithm, but the performance of stable
sort in Phobos is terrible. This pretty much left merge sort, which has
good performance but requires O(n) space. That persuaded me to design my
own sorting algorithm.
Here, you can find the code, details of the algorithm, and benchmarks
(introSort is the unstable sort in Phobos).
http://www.neowin.net/forum/blog/422/entry-3737-sort-algorithm-complete/
This is interesting. What do the numbers in the benchmark represent?
Andrei
I'll just post the code I used for benchmarking. Simply put, smaller
numbers are faster.
ubyte[16] digest;
uint[] base;
base.length = 1024 * 1024 * 16;
foreach(i, ref v; base) v = i;
randomShuffle(base);
writeln("Is sorted: ", isSorted(base));
writeln("Is length: ", base.length);
SetThreadPriority(GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL);
long start, finish;
auto copy = base.dup;
QueryPerformanceCounter(&start);
xinokSort(copy);
QueryPerformanceCounter(&finish);
sum(digest, copy);
writeln("xinokSort: ", finish - start, "\t", digestToString(digest));
assert(isSorted(copy), "Array not sorted");
Re: Database interface design - was how to build up the library.
Andrei, I've actually already got that in a way. Before I attempt to populate fields into a struct, the struct is checked to see if it conforms to the field properties. Steve
Re: Next in the Review Queue?
Dmitry Olshansky Wrote: > Well, if Jesse won't mind. > I'll wrap the thing up in my phobos fork in a couple of moments. > Anyone going for review manager? > > > -- > Dmitry Olshansky I'm good with that. Hmm, maybe I can take on managing the review for you.
Re: Database interface design - was how to build up the library.
On Oct 7, 2011, at 8:46 AM, Andrei Alexandrescu wrote: > On 10/7/11 9:11 AM, Steve Teale wrote: >> I'm thinking that for each database that gets covered there will >> need to be two modules, like: >> >> etc.c.mysql >> etc.mysqld >> >> The etc.c.xxx modules would be completely different between databases - >> just translations of the necessary header files. The xxxd modules >> would be as similar as possible consistent with allowing the strong >> points of individual database systems to show through, and the weak >> points of others to be avoided. I don't think all should be reduced >> to some LCD. > > Well we may be hasty to go that way. The driver-based approach works well for > other languages and APIs - why wouldn't it work for us? The differences > across different DBMSs would be mostly in the SQL support, not the basic APIs. It may be worth targeting ODBC early on as well, since that would get the API working for pretty much everything. Then targeting a DBMS-specific API would be an optimization step rather than a necessary one. >> These modules should attempt to make a good range of capabilities >> available to the D programmer, but they not have to be all encompassing. >> Those users who want to do really fancy things can drop back to the >> low-level interface. They should probably have the following capabilities: >> >> 1) Establishing and closing database connections. > > Sounds good. Since DB connections are a precious resource, there must be a > RAII struct holding them. The functions below may be members of it. The way I've done this before is via reference counting, since the connection handle often has to be held by all related statements, resultsets, etc. Would scope(exit) be sufficient to provide RAII functionality here? >> 3) The capability to create prepared statements with in and out >> parameters and association of the parameters with a source, and >> then to execute these. This breaks down into several components/ >> capabilities, which could be labeled: >> >> 3a) createPreparedStatement() - marshal parameters, associate them >> with a sourceand have the server prepare the statement. >> >> 3b) execStatement() - for those SQL statements that don't have a >> result set. >> >> 3c) execStatementIncremental()/getNext() - execute the prepared statement, >> then fetch the result rows one at a time into some kind of result set. > > Here's where the range interface might be used. We might simply have > execStatement() that returns an input range. If the statement produced no > rows, the range will be empty. No need for distinguishing execStatement() and > execStatementIncremental(). Absolutely, though things may be a bit tricky here. It's possible for a SQL query to return multiple resultsets, each of which is ideally represented by a range. >> 3d) execStatementAll() - execute the prepared statement and get all >> the resulting rows into some kind of result set. > > This is not a primitive, but instead a convenience function that should be > used with caution and only for queries known to be small. I agree that it's > good to have. > >> 3e) (maybe) execScalar() - do the whole sequence prepare, execute, >> and get a single value result set placed into a D variable. > > That's convenient for e.g. "SELECT COUNT(*) FROM table WHERE condition", > which people run all the time. > >> 3f) (maybe) execStoredProcedure() - another 'do the whole thing' >> capability TBD. > > Well wouldn't that just execute a special SQL a la "CALL procedure"? A stored procedure can have a return value and multiple resultsets, I believe. Certainly one or the other anyway. I think SPs are covered by the other query methods, though perhaps someone can think of a QOI feature that would benefit from a special call. >> It is when we come to the nature of the result sets that there is >> likely to be dissent. I favor arrays of structs, but we should >> probably do arrays of arrays of variants too for those situations >> where structures can't be sorted out at compile time. There needs >> to be some symmetry between what is used here, and what can be used >> as input to operations such as a prepared insert. It is of course >> vital that this part of each middle layer produce exactly the same >> kind of results. Otherwise the top layer could become difficult. > > I'd like arrays of structs too but we must acknowledge that most databases > people use will be large, in which case loading the whole thing eagerly takes > a lot of RAM and potentially wastes time. So arrays are out except for e.g. > convenience functions - but then we already have array() that converts an > arbitrary input range into an array. > > Now, regarding the "structs" part, I'd like that too when the schema is > statically known. Two issues: > > 1. MFC had at a point a wizard that generated one struct per resultset. It > was an absolute maintenance disaster and they recanted by offering > dynamically-bou
Re: Sorting algorithm
On 10/7/11 12:23 PM, Xinok wrote: http://www.neowin.net/forum/blog/422/entry-3737-sort-algorithm-complete/ This is interesting. What do the numbers in the benchmark represent? Andrei I'll just post the code I used for benchmarking. Simply put, smaller numbers are faster. [snip] Thanks. It would be great if you wanted to contribute your stable sort to Phobos via a pull request. Also, which version of D are you using? I'm seeing that std.algorithm.sort (introSort) performs quite badly; for example, it's twice as slow on shuffled data against quickSort, and it also deals badly with already sorted data. Generally it does much worse than the quickSort baseline. Wonder why. Andrei
Re: Sorting algorithm
tinkered with timsort a bit a few months ago. comparing that to your sort, I get numbers like xinokSort random: 77 ascending: 0 descending: 21 timsort random: 354 ascending: 1 descending: 4 where each are sorting a 500k element array of int, times are msecs, compilation flags were -O -inline -release, sources are http://personal.utulsa.edu/~ellery-newcomer/timsort.d http://personal.utulsa.edu/~ellery-newcomer/xinokSort.d Nice job, Xinok. anyone want to try to optimize my timsort? :)
Re: Database interface design - was how to build up the library.
Andrei, So if I'm understanding you correctly, if structs are used, and there's a database change, then the app has to be rebuilt. If Variants, then only the app's config file has to be updated (or the app can look in the database to get such information) - have I got this right? Steve
Re: Sorting algorithm
On 10/07/2011 01:20 PM, Andrei Alexandrescu wrote: > Also, which version of D are you using? I'm seeing that > std.algorithm.sort (introSort) performs quite badly; for example, it's > twice as slow on shuffled data against quickSort, and it also deals > badly with already sorted data. Generally it does much worse than the > quickSort baseline. Wonder why. > > > Andrei stable sort is flat out broken http://d.puremagic.com/issues/show_bug.cgi?id=4584
Re: D on GDC announced on reddit
Andrei Alexandrescu Wrote: > http://www.reddit.com/r/programming/comments/l3tk5/merging_in_the_gnu_d_language_compiler_to_gcc/ > > Andrei I like your comment about being in 1981. If I'm not mistaken, what he describes in this http://youtu.be/HxaD_trXwRE?t=7m55s is that toString is a Unique addition to Go.
Re: Database interface design - was how to build up the library.
On Oct 7, 2011, at 10:14 AM, Adam Ruppe wrote:
> In my database.d, I used a database row struct to provide both
> integer and key based indexes.
Funny… I have a ResultSet object which contains an array of Column objects.
Here's the basic idea:
class ResultSet {
static class Column {
int index();
string name();
int size();
int length();
void[] data();
bool opEquals(NullValue);
T as(T)(); // return raw if T matches underlying or call to!(T) on
field as appropriate
}
size_t numCols();
bool nextResultSet();
bool next();
bool prev();
bool first();
bool last();
void update(); // SQLSetPos(rowNumber=0)
Column opIndex(size_t);
// easy enough to add opIndex(string)
}
Does your Row equate to the ResultSet as above?
Re: Database interface design - was how to build up the library.
On Oct 7, 2011, at 11:27 AM, Steve Teale wrote: > Andrei, > > So if I'm understanding you correctly, if structs are used, and there's a > database > change, then the app has to be rebuilt. If Variants, then only the app's > config > file has to be updated (or the app can look in the database to get such > information) - have I got this right? I'd say yes, though qualify this by saying that a properly designed app will target only views and stored procedures, which insulate the app from schema changes.
Re: Sorting algorithm
On 10/7/2011 2:20 PM, Andrei Alexandrescu wrote: On 10/7/11 12:23 PM, Xinok wrote: http://www.neowin.net/forum/blog/422/entry-3737-sort-algorithm-complete/ This is interesting. What do the numbers in the benchmark represent? Andrei I'll just post the code I used for benchmarking. Simply put, smaller numbers are faster. [snip] Thanks. It would be great if you wanted to contribute your stable sort to Phobos via a pull request. Also, which version of D are you using? I'm seeing that std.algorithm.sort (introSort) performs quite badly; for example, it's twice as slow on shuffled data against quickSort, and it also deals badly with already sorted data. Generally it does much worse than the quickSort baseline. Wonder why. Andrei I'm not familiar with the process. What all would I need to do to contribute my sort to Phobos? On another note, my sort is most efficient on random access ranges. A simple merge sort would be more practical for other data structures like linked lists. I used DMD 2.051 for those benchmarks, so I did new benchmarks using DMD 2.055. Unstable sort saw a big improvement, but stable sort still did poorly. I find it unusual since I thought stable sort was supposed to use merge sort. xinokSort: 7564654 shellSort: 8843484 quickSort: 6005074 mergeSort: 6625306 radixSort: 2837697 phobos Unstable: 5559726 phobos Stable: 113924852
Re: Database interface design - was how to build up the library.
Sean Kelly wote:
> Does your Row equate to the ResultSet above?
Fairly similar.
Mine looks something like this:
interface ResultSet {
// name for associative array to result index
int getFieldIndex(string field);
string[] fieldNames();
bool empty();
Row front();
void popFront();
int length();
}
struct Row {
// the actual column is returned as a string - probably should
// change that, but mysql, postgres, and sqlite all offered that
// and it works for me, so I went with it
string opIndex(size_t idx) {}
string opIndex(string columnName) {}
int opApply(...) {}
private string[] data;
}
Then, each of the database implementations use that little
ResultSet interface to feed Row structs back to the user code.
Re: Database interface design - was how to build up the library.
On 2011-10-07 16:11, Steve Teale wrote: I'm thinking that for each database that gets covered there will need to be two modules, like: etc.c.mysql etc.mysqld The etc.c.xxx modules would be completely different between databases - just translations of the necessary header files. The xxxd modules would be as similar as possible consistent with allowing the strong points of individual database systems to show through, and the weak points of others to be avoided. I don't think all should be reduced to some LCD. These modules should attempt to make a good range of capabilities available to the D programmer, but they not have to be all encompassing. Those users who want to do really fancy things can drop back to the low-level interface. They should probably have the following capabilities: 1) Establishing and closing database connections. 2) The capability to execute literal SQL statements - execSQL() if you like. Some of these will generate result sets, of which more below. 3) The capability to create prepared statements with in and out parameters and association of the parameters with a source, and then to execute these. This breaks down into several components/ capabilities, which could be labeled: 3a) createPreparedStatement() - marshal parameters, associate them with a sourceand have the server prepare the statement. 3b) execStatement() - for those SQL statements that don't have a result set. 3c) execStatementIncremental()/getNext() - execute the prepared statement, then fetch the result rows one at a time into some kind of result set. 3d) execStatementAll() - execute the prepared statement and get all the resulting rows into some kind of result set. 3e) (maybe) execScalar() - do the whole sequence prepare, execute, and get a single value result set placed into a D variable. 3f) (maybe) execStoredProcedure() - another 'do the whole thing' capability TBD. It is when we come to the nature of the result sets that there is likely to be dissent. I favor arrays of structs, but we should probably do arrays of arrays of variants too for those situations where structures can't be sorted out at compile time. There needs to be some symmetry between what is used here, and what can be used as input to operations such as a prepared insert. It is of course vital that this part of each middle layer produce exactly the same kind of results. Otherwise the top layer could become difficult. On top of this set of two modules for each database, I envisage a higher-level module - etc.dbops - that provides a bunch of convenience templates for various common database operations, spanning the databases. Once the middle layer is in place, this top layer should be relatively easy to implement. It should be noted though that all these database wrappers will be monstrously difficult to test. I am at the point with MySQL where I can get the result of a plain old query into an array of a checked structure type. I have the prepared statement stuff, and know how the result will be created from a prepared query (the execStatementAll() case) - I just have to plow through a bit more binding and fetching. This is probably rather general and vague, but I would like to get comments so we can iterate toward a practical design. Thanks Steve I don't know if this will be necessary to have a special function for but it would be nice to be able to get batches of results from a query. Or this might be solved by the range interface. -- /Jacob Carlborg
Re: how to build up the library..
On 2011-10-07 15:34, Adam Ruppe wrote:
What's the big advantage of
Person.find_all_by_name_and_age("Joe", 15)
over
db.query("select * from people where name = ? and age = ?", "Joe", 15);
The latter is much easier to write in the library and you retain all
the flexibility of sql itself if needed.
I guess it just depends on what you like. I prefer to use the first
example. One advantage the first example could have would be to validate
the column names. If it already knows the column names it won't have to
query the database if the column names don't match.
Of course it should always be possible to execute raw sql queries, like
that last example.
I suggest you take a look at the documentation for ActiveRecord, just to
get a feel for the API:
http://guides.rubyonrails.org/active_record_querying.html
--
/Jacob Carlborg
Re: Database interface design - was how to build up the library.
On 10/7/11 1:27 PM, Steve Teale wrote: Andrei, So if I'm understanding you correctly, if structs are used, and there's a database change, then the app has to be rebuilt. Yes. Touched to update the structs and rebuilt. If Variants, then only the app's config file has to be updated (or the app can look in the database to get such information) - have I got this right? Yah. The exact behavior depends on the assumption the app makes about the resultset. Consider: auto result = db.sql(queryString); enforce(result.front[0].type == typeid(double)); ... In this case you'd pretty much need to touch the code if the queryString returns something else than a double in the first column. Andrei
Re: Database interface design - was how to build up the library.
Andrei Alexandrescu wrote: On 10/7/11 12:02 PM, Piotr Szturmaj wrote: Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples, structs and arrays in similar manner. I did. Looks good. Why do you need DBRow? You could simply use Variant[], the given struct, or Tuple directly. Sure, that's a good idea. I used DBRow because I wanted generic set/setNull for any of its base types, but it could be done with function templates anyway... The only thing that makes DBRow useful is opIndex access to Variant[] fields, like row["field"].
Re: Next in the Review Queue?
On 07.10.2011 21:48, Jesse Phillips wrote: Dmitry Olshansky Wrote: Well, if Jesse won't mind. I'll wrap the thing up in my phobos fork in a couple of moments. Anyone going for review manager? -- Dmitry Olshansky I'm good with that. Hmm, maybe I can take on managing the review for you. Thanks, that would be great. On the bright side of things I'm hitting a funky error after merging FReD into phobos, compiler stops recognizing std.c.xxx in std.file and issues errors like: std/file.d(537) Error: undefined identifier package c.stdio For those interested in hacking the thing: git://github.com/blackwhale/phobos.git on win32 it now runs ok. I fixed one place where compiler won't eat std.c.stdio.xxx to core.stdc.xxx and it worked. On linux it's far more complicated. So ... let's start with FReD as a standalone package and maybe with some help I'll figure out WTF is going on. It's even easier to try it out this way, no need to know git/phobos machinery. Source + docs and goodies, packaged here: https://github.com/downloads/blackwhale/FReD/FReD.zip Docs should be viewable here as well: blackwhale.github.com -- Dmitry Olshansky
Re: Sorting algorithm
On 10/07/11 13:36, Ellery Newcomer wrote: On 10/07/2011 01:20 PM, Andrei Alexandrescu wrote: Also, which version of D are you using? I'm seeing that std.algorithm.sort (introSort) performs quite badly; for example, it's twice as slow on shuffled data against quickSort, and it also deals badly with already sorted data. Generally it does much worse than the quickSort baseline. Wonder why. Andrei stable sort is flat out broken http://d.puremagic.com/issues/show_bug.cgi?id=4584 Then the contribution would be even more welcome! Andrei
Re: Database interface design - was how to build up the library.
On 10/07/11 15:00, Piotr Szturmaj wrote: Andrei Alexandrescu wrote: On 10/7/11 12:02 PM, Piotr Szturmaj wrote: Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples, structs and arrays in similar manner. I did. Looks good. Why do you need DBRow? You could simply use Variant[], the given struct, or Tuple directly. Sure, that's a good idea. I used DBRow because I wanted generic set/setNull for any of its base types, but it could be done with function templates anyway... The only thing that makes DBRow useful is opIndex access to Variant[] fields, like row["field"]. Oh that makes sense, thanks. Andrei
Re: Sorting algorithm
On 10/07/11 13:50, Xinok wrote: On 10/7/2011 2:20 PM, Andrei Alexandrescu wrote: On 10/7/11 12:23 PM, Xinok wrote: http://www.neowin.net/forum/blog/422/entry-3737-sort-algorithm-complete/ This is interesting. What do the numbers in the benchmark represent? Andrei I'll just post the code I used for benchmarking. Simply put, smaller numbers are faster. [snip] Thanks. It would be great if you wanted to contribute your stable sort to Phobos via a pull request. Also, which version of D are you using? I'm seeing that std.algorithm.sort (introSort) performs quite badly; for example, it's twice as slow on shuffled data against quickSort, and it also deals badly with already sorted data. Generally it does much worse than the quickSort baseline. Wonder why. Andrei I'm not familiar with the process. What all would I need to do to contribute my sort to Phobos? D's standard library is on github: https://github.com/D-Programming-Language/phobos. Anyone can fork it, modify it as they please, and then submit the changes for review via a so-called "pull request". Here's an example of a pull request with comments and all: https://github.com/D-Programming-Language/phobos/pull/272. There is documentation available on github.com about how to use the site. It's some work but it's time well invested - the kin of git and github are here to stay. Would anyone in the community be willing to shepherd Xinok through the first pull request? Andrei
Re: Garbage collection book
I'm just wondering, does Glasgow Haskell Compile (GHC) have the most advanced GC? I remember reading where it said that GHC is like 10 years ahead of all the other compilers, or something to that effect.
Re: Next in the Review Queue?
On Sat, 08 Oct 2011 00:34:26 +0400, Dmitry Olshansky wrote: > Thanks, that would be great. > > On the bright side of things I'm hitting a funky error after merging > FReD into phobos, compiler stops recognizing std.c.xxx in std.file and > issues errors like: > std/file.d(537) Error: undefined identifier package c.stdio > > For those interested in hacking the thing: > git://github.com/blackwhale/phobos.git on win32 it now runs ok. I fixed > one place where compiler won't eat std.c.stdio.xxx to core.stdc.xxx and > it worked. On linux it's far more complicated. > > > So ... let's start with FReD as a standalone package and maybe with some > help I'll figure out WTF is going on. It's even easier to try it out > this way, no need to know git/phobos machinery. > > Source + docs and goodies, packaged here: > https://github.com/downloads/blackwhale/FReD/FReD.zip > > Docs should be viewable here as well: blackwhale.github.com Thank you I'm going to try and get your phobos branch working, and do some initial build testing. Little tip, you should keep your master branch mirroring that of upstream. Don't worry about it this time though.
Re: Garbage collection book
Caligo: > I'm just wondering, does Glasgow Haskell Compile (GHC) have the most advanced > GC? > I remember reading where it said that GHC is like 10 years ahead of all the > other compilers, > or something to that effect. If you want to find an advanced GC that is years ahead of all other ones, take a look at the garbage collectors inside the Oracle JavaVM. Haskell is almost purely functional, and its GC has to do a work different from a Java or D GC. A D GC has to do a work different from a Java GC, and more similar to a C# GC (but not exactly the same of C# because I think in D there is a larger percentage of pinned down data). Even if it is not perfectly fit, I think the recently created good GC for the C# Mono is good enough for D, maybe with some tuning. Unfortunately, despite being both Mono and D open source projects, there is a furiously intense "not invented here" syndrome in the whole planetary effort of Open Source. Every open source language seems to implement its own GC If you look at this situation from 15000 feet above it looks like an incredibly dumb situation. In practice once you get closer, you see incompatible open source licenses, and differences in the language semantics that make GC transplants hard or not so useful. Devil is in the details. Bye, bearophile
Re: Next in the Review Queue?
On Sat, 08 Oct 2011 00:34:26 +0400, Dmitry Olshansky wrote: > On the bright side of things I'm hitting a funky error after merging > FReD into phobos, compiler stops recognizing std.c.xxx in std.file and > issues errors like: > std/file.d(537) Error: undefined identifier package c.stdio > > For those interested in hacking the thing: > git://github.com/blackwhale/phobos.git on win32 it now runs ok. I fixed > one place where compiler won't eat std.c.stdio.xxx to core.stdc.xxx and > it worked. On linux it's far more complicated. After setting up my environment again, Latest druntime/phobos/dmd it builds without issue for me.
Re: Sorting algorithm
On 10/7/2011 2:27 PM, Ellery Newcomer wrote: tinkered with timsort a bit a few months ago. comparing that to your sort, I get numbers like xinokSort random: 77 ascending: 0 descending: 21 timsort random: 354 ascending: 1 descending: 4 where each are sorting a 500k element array of int, times are msecs, compilation flags were -O -inline -release, sources are http://personal.utulsa.edu/~ellery-newcomer/timsort.d http://personal.utulsa.edu/~ellery-newcomer/xinokSort.d Nice job, Xinok. anyone want to try to optimize my timsort? :) The benchmark for descending is no surprise since timsort explicitly searches for runs in descending order. However, the random benchmark isn't what I expected. My sort is a bit slower than a standard merge sort, so I would expect Timsort to be faster. While part of the reason may be your code is unoptimized, it may also be the fault of a bug. I ran some more benchmarks and I found a few cases where your code failed, specifically when the data is mostly random. Just a guess, but your code may have a problem when the "runs" are too small. I also found a case when std.algorithm.sort performs poorly, under Small Shuffle + Descending. Numbers represent time taken; Smaller numbers are faster An MD5 hash is generated for each sort, to verify it was sorted correctly phoboSort is unstable std.algorithm.sort Ascending Is length: 16777216 xinokSort: 50722C8B8D3A2B4D9882ABCFC31721EF27145 shellSort: 726046 C8B8D3A2B4D9882ABCFC31721EF27145 quickSort: 943475 C8B8D3A2B4D9882ABCFC31721EF27145 mergeSort: 1778944 C8B8D3A2B4D9882ABCFC31721EF27145 radixSort: 3082901 C8B8D3A2B4D9882ABCFC31721EF27145 phoboSort: 955285 C8B8D3A2B4D9882ABCFC31721EF27145 tim Sort: 89201C8B8D3A2B4D9882ABCFC31721EF27145 Descending Is length: 16777216 xinokSort: 1916452 C8B8D3A2B4D9882ABCFC31721EF27145 shellSort: 2238446 C8B8D3A2B4D9882ABCFC31721EF27145 quickSort: 1581095 C8B8D3A2B4D9882ABCFC31721EF27145 mergeSort: 3390033 C8B8D3A2B4D9882ABCFC31721EF27145 radixSort: 3067271 C8B8D3A2B4D9882ABCFC31721EF27145 phoboSort: 1035827 C8B8D3A2B4D9882ABCFC31721EF27145 tim Sort: 240896 C8B8D3A2B4D9882ABCFC31721EF27145 Complete Shuffle Is length: 16777216 xinokSort: 7607511 C8B8D3A2B4D9882ABCFC31721EF27145 shellSort: 8814887 C8B8D3A2B4D9882ABCFC31721EF27145 quickSort: 5623837 C8B8D3A2B4D9882ABCFC31721EF27145 mergeSort: 6704733 C8B8D3A2B4D9882ABCFC31721EF27145 radixSort: 2825567 C8B8D3A2B4D9882ABCFC31721EF27145 phoboSort: 5532275 C8B8D3A2B4D9882ABCFC31721EF27145 tim Sort: 29142913 E03C4778321F69D8AD10F624E6093599 Small Shuffle (1024 pairs were picked at random and swapped) Is length: 16777216 xinokSort: 589882 C8B8D3A2B4D9882ABCFC31721EF27145 shellSort: 3651222 C8B8D3A2B4D9882ABCFC31721EF27145 quickSort: 2655391 C8B8D3A2B4D9882ABCFC31721EF27145 mergeSort: 2162840 C8B8D3A2B4D9882ABCFC31721EF27145 radixSort: 2988630 C8B8D3A2B4D9882ABCFC31721EF27145 phoboSort: 963103 C8B8D3A2B4D9882ABCFC31721EF27145 tim Sort: 6523251 C8B8D3A2B4D9882ABCFC31721EF27145 Large Shuffle (1,048,576 pairs were picked at random and swapped) Is length: 16777216 xinokSort: 1956126 C8B8D3A2B4D9882ABCFC31721EF27145 shellSort: 7617207 C8B8D3A2B4D9882ABCFC31721EF27145 quickSort: 3089674 C8B8D3A2B4D9882ABCFC31721EF27145 mergeSort: 2606200 C8B8D3A2B4D9882ABCFC31721EF27145 radixSort: 2932306 C8B8D3A2B4D9882ABCFC31721EF27145 phoboSort: 1619484 C8B8D3A2B4D9882ABCFC31721EF27145 tim Sort: 14883251 DE54E94D1A058459FEA3A80096FCBB18 Small Shuffle + Descending Is length: 16777216 xinokSort: 2288869 C8B8D3A2B4D9882ABCFC31721EF27145 shellSort: 4599417 C8B8D3A2B4D9882ABCFC31721EF27145 quickSort: 2604222 C8B8D3A2B4D9882ABCFC31721EF27145 mergeSort: 3457241 C8B8D3A2B4D9882ABCFC31721EF27145 radixSort: 3055080 C8B8D3A2B4D9882ABCFC31721EF27145 phoboSort: 261768564C8B8D3A2B4D9882ABCFC31721EF27145 tim Sort: 10149160 C8B8D3A2B4D9882ABCFC31721EF27145 Large Shuffle + Descending xinokSort: 2923813 C8B8D3A2B4D9882ABCFC31721EF27145 shellSort: 7678966 C8B8D3A2B4D9882ABCFC31721EF27145 quickSort: 3833138 C8B8D3A2B4D9882ABCFC31721EF27145 mergeSort: 3971124 C8B8D3A2B4D9882ABCFC31721EF27145 radixSort: 2941206 C8B8D3A2B4D9882ABCFC31721EF27145 phoboSort: 3749512 C8B8D3A2B4D9882ABCFC31721EF27145 tim Sort: 22709918 191BD30B3D0E52C922A7E6A16E5E63A5
Re: Sorting algorithm
On 10/7/2011 5:08 PM, Andrei Alexandrescu wrote: On 10/07/11 13:50, Xinok wrote: On 10/7/2011 2:20 PM, Andrei Alexandrescu wrote: On 10/7/11 12:23 PM, Xinok wrote: http://www.neowin.net/forum/blog/422/entry-3737-sort-algorithm-complete/ This is interesting. What do the numbers in the benchmark represent? Andrei I'll just post the code I used for benchmarking. Simply put, smaller numbers are faster. [snip] Thanks. It would be great if you wanted to contribute your stable sort to Phobos via a pull request. Also, which version of D are you using? I'm seeing that std.algorithm.sort (introSort) performs quite badly; for example, it's twice as slow on shuffled data against quickSort, and it also deals badly with already sorted data. Generally it does much worse than the quickSort baseline. Wonder why. Andrei I'm not familiar with the process. What all would I need to do to contribute my sort to Phobos? D's standard library is on github: https://github.com/D-Programming-Language/phobos. Anyone can fork it, modify it as they please, and then submit the changes for review via a so-called "pull request". Here's an example of a pull request with comments and all: https://github.com/D-Programming-Language/phobos/pull/272. There is documentation available on github.com about how to use the site. It's some work but it's time well invested - the kin of git and github are here to stay. Would anyone in the community be willing to shepherd Xinok through the first pull request? Andrei Thanks, I'll look into it when I have a little more time. If my algorithm were to be implemented in Phobos, the template arguments for std.algorithm.sort would need to be modified. For my algorithm to be stable, it requires defining two conditions, both a 'less' and 'greater'.
[OT] TortoiseGit and Hg
I've been getting more familiar with both git and hg, and I'm coming to the conclusion that I like TortoiseGit far more than TortoiseHg, but when I have to use them at the cmd line (which I hate to have to do), I can't stand Git's cmd line interface, but Hg seems to be pretty nice (and I like Hg's revision/changeset/whatever identier system far better). I know it's (theoretically) possible to use TortoiseHg with Git repos via hg-git (which only barely works at all, in my experience), but that's opposite of what I've realized I want. Anyone know if there's any way to go the other way around? Use TortoiseGit to work with Hg repos? Meh, frankly, I'm real unhappy with DVCSes. I'm completely sold on them, of course, a definite improvement over SVN. But before I learned about DVCSes I was very happy with SVN; not so happy with the DVCS landscape now that I'm here :/
[std.database]
I use this title at Andrei's suggestion, and repeat his idea that it be used as a prefix for discussions as we navigate toward a design. Unless there is resistance to the idea, I will on the job of implementing whatever we decide is appropriate. I am retired, and have the time to do it. It seems that every man, and possibly his dog, has a private implementation for at least a favorite DB, so we should have plenty of material to build on. At this point I would like to get responses from those who feel they are likely to contribute to the design through to completion. I'd also like to get a feel for the magnitude of the task, so I'd like to ask what database systems you think should be supported. I have started a github account, and will put my mysqld stuff there shortly, then you can kick me out if you don't like what you see. Steve
