Re: Database interface design - was how to build up the library.
On 10/07/11 15:00, Piotr Szturmaj wrote: Andrei Alexandrescu wrote: On 10/7/11 12:02 PM, Piotr Szturmaj wrote: Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples, structs and arrays in similar manner. I did. Looks good. Why do you need DBRow? You could simply use Variant[], the given struct, or Tuple directly. Sure, that's a good idea. I used DBRow because I wanted generic set/setNull for any of its base types, but it could be done with function templates anyway... The only thing that makes DBRow useful is opIndex access to Variant[] fields, like row["field"]. Oh that makes sense, thanks. Andrei
Re: Database interface design - was how to build up the library.
Andrei Alexandrescu wrote: On 10/7/11 12:02 PM, Piotr Szturmaj wrote: Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples, structs and arrays in similar manner. I did. Looks good. Why do you need DBRow? You could simply use Variant[], the given struct, or Tuple directly. Sure, that's a good idea. I used DBRow because I wanted generic set/setNull for any of its base types, but it could be done with function templates anyway... The only thing that makes DBRow useful is opIndex access to Variant[] fields, like row["field"].
Re: Database interface design - was how to build up the library.
On 10/7/11 1:27 PM, Steve Teale wrote: Andrei, So if I'm understanding you correctly, if structs are used, and there's a database change, then the app has to be rebuilt. Yes. Touched to update the structs and rebuilt. If Variants, then only the app's config file has to be updated (or the app can look in the database to get such information) - have I got this right? Yah. The exact behavior depends on the assumption the app makes about the resultset. Consider: auto result = db.sql(queryString); enforce(result.front[0].type == typeid(double)); ... In this case you'd pretty much need to touch the code if the queryString returns something else than a double in the first column. Andrei
Re: how to build up the library..
On 2011-10-07 15:34, Adam Ruppe wrote:
What's the big advantage of
Person.find_all_by_name_and_age("Joe", 15)
over
db.query("select * from people where name = ? and age = ?", "Joe", 15);
The latter is much easier to write in the library and you retain all
the flexibility of sql itself if needed.
I guess it just depends on what you like. I prefer to use the first
example. One advantage the first example could have would be to validate
the column names. If it already knows the column names it won't have to
query the database if the column names don't match.
Of course it should always be possible to execute raw sql queries, like
that last example.
I suggest you take a look at the documentation for ActiveRecord, just to
get a feel for the API:
http://guides.rubyonrails.org/active_record_querying.html
--
/Jacob Carlborg
Re: Database interface design - was how to build up the library.
On 2011-10-07 16:11, Steve Teale wrote: I'm thinking that for each database that gets covered there will need to be two modules, like: etc.c.mysql etc.mysqld The etc.c.xxx modules would be completely different between databases - just translations of the necessary header files. The xxxd modules would be as similar as possible consistent with allowing the strong points of individual database systems to show through, and the weak points of others to be avoided. I don't think all should be reduced to some LCD. These modules should attempt to make a good range of capabilities available to the D programmer, but they not have to be all encompassing. Those users who want to do really fancy things can drop back to the low-level interface. They should probably have the following capabilities: 1) Establishing and closing database connections. 2) The capability to execute literal SQL statements - execSQL() if you like. Some of these will generate result sets, of which more below. 3) The capability to create prepared statements with in and out parameters and association of the parameters with a source, and then to execute these. This breaks down into several components/ capabilities, which could be labeled: 3a) createPreparedStatement() - marshal parameters, associate them with a sourceand have the server prepare the statement. 3b) execStatement() - for those SQL statements that don't have a result set. 3c) execStatementIncremental()/getNext() - execute the prepared statement, then fetch the result rows one at a time into some kind of result set. 3d) execStatementAll() - execute the prepared statement and get all the resulting rows into some kind of result set. 3e) (maybe) execScalar() - do the whole sequence prepare, execute, and get a single value result set placed into a D variable. 3f) (maybe) execStoredProcedure() - another 'do the whole thing' capability TBD. It is when we come to the nature of the result sets that there is likely to be dissent. I favor arrays of structs, but we should probably do arrays of arrays of variants too for those situations where structures can't be sorted out at compile time. There needs to be some symmetry between what is used here, and what can be used as input to operations such as a prepared insert. It is of course vital that this part of each middle layer produce exactly the same kind of results. Otherwise the top layer could become difficult. On top of this set of two modules for each database, I envisage a higher-level module - etc.dbops - that provides a bunch of convenience templates for various common database operations, spanning the databases. Once the middle layer is in place, this top layer should be relatively easy to implement. It should be noted though that all these database wrappers will be monstrously difficult to test. I am at the point with MySQL where I can get the result of a plain old query into an array of a checked structure type. I have the prepared statement stuff, and know how the result will be created from a prepared query (the execStatementAll() case) - I just have to plow through a bit more binding and fetching. This is probably rather general and vague, but I would like to get comments so we can iterate toward a practical design. Thanks Steve I don't know if this will be necessary to have a special function for but it would be nice to be able to get batches of results from a query. Or this might be solved by the range interface. -- /Jacob Carlborg
Re: Database interface design - was how to build up the library.
Sean Kelly wote:
> Does your Row equate to the ResultSet above?
Fairly similar.
Mine looks something like this:
interface ResultSet {
// name for associative array to result index
int getFieldIndex(string field);
string[] fieldNames();
bool empty();
Row front();
void popFront();
int length();
}
struct Row {
// the actual column is returned as a string - probably should
// change that, but mysql, postgres, and sqlite all offered that
// and it works for me, so I went with it
string opIndex(size_t idx) {}
string opIndex(string columnName) {}
int opApply(...) {}
private string[] data;
}
Then, each of the database implementations use that little
ResultSet interface to feed Row structs back to the user code.
Re: Database interface design - was how to build up the library.
On Oct 7, 2011, at 11:27 AM, Steve Teale wrote: > Andrei, > > So if I'm understanding you correctly, if structs are used, and there's a > database > change, then the app has to be rebuilt. If Variants, then only the app's > config > file has to be updated (or the app can look in the database to get such > information) - have I got this right? I'd say yes, though qualify this by saying that a properly designed app will target only views and stored procedures, which insulate the app from schema changes.
Re: Database interface design - was how to build up the library.
On Oct 7, 2011, at 10:14 AM, Adam Ruppe wrote:
> In my database.d, I used a database row struct to provide both
> integer and key based indexes.
Funny… I have a ResultSet object which contains an array of Column objects.
Here's the basic idea:
class ResultSet {
static class Column {
int index();
string name();
int size();
int length();
void[] data();
bool opEquals(NullValue);
T as(T)(); // return raw if T matches underlying or call to!(T) on
field as appropriate
}
size_t numCols();
bool nextResultSet();
bool next();
bool prev();
bool first();
bool last();
void update(); // SQLSetPos(rowNumber=0)
Column opIndex(size_t);
// easy enough to add opIndex(string)
}
Does your Row equate to the ResultSet as above?
Re: Database interface design - was how to build up the library.
Andrei, So if I'm understanding you correctly, if structs are used, and there's a database change, then the app has to be rebuilt. If Variants, then only the app's config file has to be updated (or the app can look in the database to get such information) - have I got this right? Steve
Re: Database interface design - was how to build up the library.
On Oct 7, 2011, at 8:46 AM, Andrei Alexandrescu wrote: > On 10/7/11 9:11 AM, Steve Teale wrote: >> I'm thinking that for each database that gets covered there will >> need to be two modules, like: >> >> etc.c.mysql >> etc.mysqld >> >> The etc.c.xxx modules would be completely different between databases - >> just translations of the necessary header files. The xxxd modules >> would be as similar as possible consistent with allowing the strong >> points of individual database systems to show through, and the weak >> points of others to be avoided. I don't think all should be reduced >> to some LCD. > > Well we may be hasty to go that way. The driver-based approach works well for > other languages and APIs - why wouldn't it work for us? The differences > across different DBMSs would be mostly in the SQL support, not the basic APIs. It may be worth targeting ODBC early on as well, since that would get the API working for pretty much everything. Then targeting a DBMS-specific API would be an optimization step rather than a necessary one. >> These modules should attempt to make a good range of capabilities >> available to the D programmer, but they not have to be all encompassing. >> Those users who want to do really fancy things can drop back to the >> low-level interface. They should probably have the following capabilities: >> >> 1) Establishing and closing database connections. > > Sounds good. Since DB connections are a precious resource, there must be a > RAII struct holding them. The functions below may be members of it. The way I've done this before is via reference counting, since the connection handle often has to be held by all related statements, resultsets, etc. Would scope(exit) be sufficient to provide RAII functionality here? >> 3) The capability to create prepared statements with in and out >> parameters and association of the parameters with a source, and >> then to execute these. This breaks down into several components/ >> capabilities, which could be labeled: >> >> 3a) createPreparedStatement() - marshal parameters, associate them >> with a sourceand have the server prepare the statement. >> >> 3b) execStatement() - for those SQL statements that don't have a >> result set. >> >> 3c) execStatementIncremental()/getNext() - execute the prepared statement, >> then fetch the result rows one at a time into some kind of result set. > > Here's where the range interface might be used. We might simply have > execStatement() that returns an input range. If the statement produced no > rows, the range will be empty. No need for distinguishing execStatement() and > execStatementIncremental(). Absolutely, though things may be a bit tricky here. It's possible for a SQL query to return multiple resultsets, each of which is ideally represented by a range. >> 3d) execStatementAll() - execute the prepared statement and get all >> the resulting rows into some kind of result set. > > This is not a primitive, but instead a convenience function that should be > used with caution and only for queries known to be small. I agree that it's > good to have. > >> 3e) (maybe) execScalar() - do the whole sequence prepare, execute, >> and get a single value result set placed into a D variable. > > That's convenient for e.g. "SELECT COUNT(*) FROM table WHERE condition", > which people run all the time. > >> 3f) (maybe) execStoredProcedure() - another 'do the whole thing' >> capability TBD. > > Well wouldn't that just execute a special SQL a la "CALL procedure"? A stored procedure can have a return value and multiple resultsets, I believe. Certainly one or the other anyway. I think SPs are covered by the other query methods, though perhaps someone can think of a QOI feature that would benefit from a special call. >> It is when we come to the nature of the result sets that there is >> likely to be dissent. I favor arrays of structs, but we should >> probably do arrays of arrays of variants too for those situations >> where structures can't be sorted out at compile time. There needs >> to be some symmetry between what is used here, and what can be used >> as input to operations such as a prepared insert. It is of course >> vital that this part of each middle layer produce exactly the same >> kind of results. Otherwise the top layer could become difficult. > > I'd like arrays of structs too but we must acknowledge that most databases > people use will be large, in which case loading the whole thing eagerly takes > a lot of RAM and potentially wastes time. So arrays are out except for e.g. > convenience functions - but then we already have array() that converts an > arbitrary input range into an array. > > Now, regarding the "structs" part, I'd like that too when the schema is > statically known. Two issues: > > 1. MFC had at a point a wizard that generated one struct per resultset. It > was an absolute maintenance disaster and they recanted by offering > dynamically-bou
Re: Database interface design - was how to build up the library.
Andrei, I've actually already got that in a way. Before I attempt to populate fields into a struct, the struct is checked to see if it conforms to the field properties. Steve
Re: Database interface design - was how to build up the library.
In my database.d, I used a database row struct to provide both integer and key based indexes.
Re: Database interface design - was how to build up the library.
On 10/7/11 12:02 PM, Piotr Szturmaj wrote:
Andrei Alexandrescu wrote:
1. MFC had at a point a wizard that generated one struct per resultset.
It was an absolute maintenance disaster and they recanted by offering
dynamically-bound result sets. The lesson there is that defining a
struct for each query won't likely play out, so we better use
Tuple!(ColType1, ColType2, ...). A possible API would be:
auto db = std.database.connect("cdb.mysql");
auto rows = db.sql!(double, ulong)
("SELECT SUM(x), COUNT(x) FROM table");
// We know that rows is a range of Tuple!(double, ulong).
writeln("Sum is ", rows.front[0], " count is ", rows.front[1]);
Cool beans. I'd love to use such an API!
Andrei
Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples,
structs and arrays in similar manner.
I did. Looks good. Why do you need DBRow? You could simply use
Variant[], the given struct, or Tuple directly.
Andrei
Re: Database interface design - was how to build up the library.
Andrei Alexandrescu wrote:
1. MFC had at a point a wizard that generated one struct per resultset.
It was an absolute maintenance disaster and they recanted by offering
dynamically-bound result sets. The lesson there is that defining a
struct for each query won't likely play out, so we better use
Tuple!(ColType1, ColType2, ...). A possible API would be:
auto db = std.database.connect("cdb.mysql");
auto rows = db.sql!(double, ulong)
("SELECT SUM(x), COUNT(x) FROM table");
// We know that rows is a range of Tuple!(double, ulong).
writeln("Sum is ", rows.front[0], " count is ", rows.front[1]);
Cool beans. I'd love to use such an API!
Andrei
Did you see http://pszturmaj.github.com/ddb/db.html ? It maps tuples,
structs and arrays in similar manner.
Re: Database interface design - was how to build up the library.
On 10/7/11 11:51 AM, Steve Teale wrote: Andrei, The suggestion in your penultimate paragraph is what I'm trying to get started. Maybe the answer to the problem with structs may lie somewhere in std.typecons in combination with the field discovery features supported by most databases. If we can manufacture the struct to suit the circumstances, then we've cracked it! I'm probably talking rubbish - Friday night, couple of beers. No can do. This is a hard wall. Structs must be defined during compilation. Database structure can be discovered at run time. What you can do, however, is that you pass a struct type to store a row in the resultset and at runtime the fields of the struct are matched dynamically with the fields in the rowset. That is doable and useful. I will attempt top start a new thread - [std.database] - your 10 cents worth? http://www.youtube.com/watch?v=lmUZGdi7Ty4 Andrei
Re: Database interface design - was how to build up the library.
Andrei, The suggestion in your penultimate paragraph is what I'm trying to get started. Maybe the answer to the problem with structs may lie somewhere in std.typecons in combination with the field discovery features supported by most databases. If we can manufacture the struct to suit the circumstances, then we've cracked it! I'm probably talking rubbish - Friday night, couple of beers. I will attempt top start a new thread - [std.database] - your 10 cents worth? Thanks Steve
Re: Database interface design - was how to build up the library.
On 07-10-2011 16:11, Steve Teale wrote: I'm thinking that for each database that gets covered there will need to be two modules, like: etc.c.mysql etc.mysqld The etc.c.xxx modules would be completely different between databases - just translations of the necessary header files. The xxxd modules would be as similar as possible consistent with allowing the strong points of individual database systems to show through, and the weak points of others to be avoided. I don't think all should be reduced to some LCD. These modules should attempt to make a good range of capabilities available to the D programmer, but they not have to be all encompassing. Those users who want to do really fancy things can drop back to the low-level interface. They should probably have the following capabilities: 1) Establishing and closing database connections. 2) The capability to execute literal SQL statements - execSQL() if you like. Some of these will generate result sets, of which more below. 3) The capability to create prepared statements with in and out parameters and association of the parameters with a source, and then to execute these. This breaks down into several components/ capabilities, which could be labeled: 3a) createPreparedStatement() - marshal parameters, associate them with a sourceand have the server prepare the statement. 3b) execStatement() - for those SQL statements that don't have a result set. 3c) execStatementIncremental()/getNext() - execute the prepared statement, then fetch the result rows one at a time into some kind of result set. 3d) execStatementAll() - execute the prepared statement and get all the resulting rows into some kind of result set. 3e) (maybe) execScalar() - do the whole sequence prepare, execute, and get a single value result set placed into a D variable. 3f) (maybe) execStoredProcedure() - another 'do the whole thing' capability TBD. It is when we come to the nature of the result sets that there is likely to be dissent. I favor arrays of structs, but we should probably do arrays of arrays of variants too for those situations where structures can't be sorted out at compile time. There needs to be some symmetry between what is used here, and what can be used as input to operations such as a prepared insert. It is of course vital that this part of each middle layer produce exactly the same kind of results. Otherwise the top layer could become difficult. On top of this set of two modules for each database, I envisage a higher-level module - etc.dbops - that provides a bunch of convenience templates for various common database operations, spanning the databases. Once the middle layer is in place, this top layer should be relatively easy to implement. It should be noted though that all these database wrappers will be monstrously difficult to test. I am at the point with MySQL where I can get the result of a plain old query into an array of a checked structure type. I have the prepared statement stuff, and know how the result will be created from a prepared query (the execStatementAll() case) - I just have to plow through a bit more binding and fetching. This is probably rather general and vague, but I would like to get comments so we can iterate toward a practical design. Thanks Steve Just a note on module naming: mysqld is misleading - it could mean "mysql daemon". I would recommend just calling it mysql and calling the C interface etc.c.mysqlc or something like this. - Alex
Re: Database interface design - was how to build up the library.
On 10/7/11 9:11 AM, Steve Teale wrote:
I'm thinking that for each database that gets covered there will
need to be two modules, like:
etc.c.mysql
etc.mysqld
The etc.c.xxx modules would be completely different between databases -
just translations of the necessary header files. The xxxd modules
would be as similar as possible consistent with allowing the strong
points of individual database systems to show through, and the weak
points of others to be avoided. I don't think all should be reduced
to some LCD.
Well we may be hasty to go that way. The driver-based approach works
well for other languages and APIs - why wouldn't it work for us? The
differences across different DBMSs would be mostly in the SQL support,
not the basic APIs.
These modules should attempt to make a good range of capabilities
available to the D programmer, but they not have to be all encompassing.
Those users who want to do really fancy things can drop back to the
low-level interface. They should probably have the following capabilities:
1) Establishing and closing database connections.
Sounds good. Since DB connections are a precious resource, there must be
a RAII struct holding them. The functions below may be members of it.
2) The capability to execute literal SQL statements - execSQL()
if you like. Some of these will generate result sets, of which more below.
Great.
3) The capability to create prepared statements with in and out
parameters and association of the parameters with a source, and
then to execute these. This breaks down into several components/
capabilities, which could be labeled:
3a) createPreparedStatement() - marshal parameters, associate them
with a sourceand have the server prepare the statement.
3b) execStatement() - for those SQL statements that don't have a
result set.
3c) execStatementIncremental()/getNext() - execute the prepared statement,
then fetch the result rows one at a time into some kind of result set.
Here's where the range interface might be used. We might simply have
execStatement() that returns an input range. If the statement produced
no rows, the range will be empty. No need for distinguishing
execStatement() and execStatementIncremental().
3d) execStatementAll() - execute the prepared statement and get all
the resulting rows into some kind of result set.
This is not a primitive, but instead a convenience function that should
be used with caution and only for queries known to be small. I agree
that it's good to have.
3e) (maybe) execScalar() - do the whole sequence prepare, execute,
and get a single value result set placed into a D variable.
That's convenient for e.g. "SELECT COUNT(*) FROM table WHERE condition",
which people run all the time.
3f) (maybe) execStoredProcedure() - another 'do the whole thing'
capability TBD.
Well wouldn't that just execute a special SQL a la "CALL procedure"?
It is when we come to the nature of the result sets that there is
likely to be dissent. I favor arrays of structs, but we should
probably do arrays of arrays of variants too for those situations
where structures can't be sorted out at compile time. There needs
to be some symmetry between what is used here, and what can be used
as input to operations such as a prepared insert. It is of course
vital that this part of each middle layer produce exactly the same
kind of results. Otherwise the top layer could become difficult.
I'd like arrays of structs too but we must acknowledge that most
databases people use will be large, in which case loading the whole
thing eagerly takes a lot of RAM and potentially wastes time. So arrays
are out except for e.g. convenience functions - but then we already have
array() that converts an arbitrary input range into an array.
Now, regarding the "structs" part, I'd like that too when the schema is
statically known. Two issues:
1. MFC had at a point a wizard that generated one struct per resultset.
It was an absolute maintenance disaster and they recanted by offering
dynamically-bound result sets. The lesson there is that defining a
struct for each query won't likely play out, so we better use
Tuple!(ColType1, ColType2, ...). A possible API would be:
auto db = std.database.connect("cdb.mysql");
auto rows = db.sql!(double, ulong)
("SELECT SUM(x), COUNT(x) FROM table");
// We know that rows is a range of Tuple!(double, ulong).
writeln("Sum is ", rows.front[0], " count is ", rows.front[1]);
Cool beans. I'd love to use such an API!
2. Statically-bound tuples work only when we know the query beforehand.
We need to cater for resultsets of unknown column count and types. The
question here is whether we traffic in untyped memory a la ubyte[] or
some variant type. I favor Variant because it's efficient, accommodates
any SQL type easily, and is convenient to use.
On top of this set of two modules for each database, I envisage a
higher-level module - etc.dbops - that provides a bunch of convenience
templates for var
Database interface design - was how to build up the library.
I'm thinking that for each database that gets covered there will need to be two modules, like: etc.c.mysql etc.mysqld The etc.c.xxx modules would be completely different between databases - just translations of the necessary header files. The xxxd modules would be as similar as possible consistent with allowing the strong points of individual database systems to show through, and the weak points of others to be avoided. I don't think all should be reduced to some LCD. These modules should attempt to make a good range of capabilities available to the D programmer, but they not have to be all encompassing. Those users who want to do really fancy things can drop back to the low-level interface. They should probably have the following capabilities: 1) Establishing and closing database connections. 2) The capability to execute literal SQL statements - execSQL() if you like. Some of these will generate result sets, of which more below. 3) The capability to create prepared statements with in and out parameters and association of the parameters with a source, and then to execute these. This breaks down into several components/ capabilities, which could be labeled: 3a) createPreparedStatement() - marshal parameters, associate them with a sourceand have the server prepare the statement. 3b) execStatement() - for those SQL statements that don't have a result set. 3c) execStatementIncremental()/getNext() - execute the prepared statement, then fetch the result rows one at a time into some kind of result set. 3d) execStatementAll() - execute the prepared statement and get all the resulting rows into some kind of result set. 3e) (maybe) execScalar() - do the whole sequence prepare, execute, and get a single value result set placed into a D variable. 3f) (maybe) execStoredProcedure() - another 'do the whole thing' capability TBD. It is when we come to the nature of the result sets that there is likely to be dissent. I favor arrays of structs, but we should probably do arrays of arrays of variants too for those situations where structures can't be sorted out at compile time. There needs to be some symmetry between what is used here, and what can be used as input to operations such as a prepared insert. It is of course vital that this part of each middle layer produce exactly the same kind of results. Otherwise the top layer could become difficult. On top of this set of two modules for each database, I envisage a higher-level module - etc.dbops - that provides a bunch of convenience templates for various common database operations, spanning the databases. Once the middle layer is in place, this top layer should be relatively easy to implement. It should be noted though that all these database wrappers will be monstrously difficult to test. I am at the point with MySQL where I can get the result of a plain old query into an array of a checked structure type. I have the prepared statement stuff, and know how the result will be created from a prepared query (the execStatementAll() case) - I just have to plow through a bit more binding and fetching. This is probably rather general and vague, but I would like to get comments so we can iterate toward a practical design. Thanks Steve
Re: how to build up the library..
What's the big advantage of
Person.find_all_by_name_and_age("Joe", 15)
over
db.query("select * from people where name = ? and age = ?", "Joe", 15);
The latter is much easier to write in the library and you retain all
the flexibility of sql itself if needed.
Re: how to build up the library..
On Fri, 07 Oct 2011 07:40:01 +0100, Jacob Carlborg wrote: On 2011-10-06 21:44, Jesse Phillips wrote: Regan Heath Wrote: That's just it however, I am not generous enough with my time to be of use to you/D/phobos. I spend my free time doing other things, and not working on the code I have which could be of use. I suspect there are quite a number of people out there who are just like me. People who have some code which may be of use, in varying degrees of completeness from just an idea to a pretty much complete re-usable library which just needs the final touches etc. The problem is as you say, there are many out there like you. Even if some people take the rule of getting these polished, they couldn't keep up. Finding these people his hard. Even if we just get one contribution to Phobos through this method it would be better than nothing. Exactly. The people taking ideas from the pool don't have to keep up, they just need to be someone with the time/inclination to "complete" something/anything worthy of inclusion into phobos. All the pool does is give them a leg-up or provide the technical know-how/guts they might be missing. It would be something like dsource, but for code ranging from snippets to complete packages, and would have to be well organised, or at least searchable and rely on contributors including as much info about thier snippets etc as they can. But, that's a nice small investment for someone, especially if we make it easy to contribute the code, agree to a license and add info about it which is then searchable. I realise setting something like this up is no small amount of work, and that I'm in no position to do it, but it seems on the face of it to be a good idea. The pool would serve double duty, as it would supply code examples/snippets to people learning D also, or anyone looking for examples of how to do .. well, anything that gets contributed. Each of us probably has a collection of D code on our own HDDs somewhere which does interesting and useful things, but no-one else can see it, search it, or learn from it. I'm simply suggesting we put it somewhere to share. At it's most basic it could just be a wiki page, which we let google index for us. -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Re: how to build up the library..
On 2011-10-07 09:26, Russel Winder wrote:
On Fri, 2011-10-07 at 08:49 +0200, Jacob Carlborg wrote:
[ . . . ]
I think it's important to support the range interface (or if it's
simpler, the array interface). I think ActiveRecord has a good high
level API which allows to lazily evaluate SQL queries.
persons = Person.all # no query is performed here
persons = persons.where("name = 'Joe'") # or here
persons.each do |p| # The actual query is performed here
# do something with each Person
end
I don't think it would be strange to see something like this:
Person.where("age> 10").map{ |p| p.name } # using "map" on an SQL query
Not just Ruby an Rails with ActiveRecord. Groovy/Grails/GORM does what
is basically the same. Python cannot do the same in the same way Ruby
and Groovy can, but it has it equivalent (augmented with SQLAlchemy).
Basically it comes down to constructing dynamic queries using the
dynamic and MOP nature of the languages. The crucial keyword here is
Builder. The trick is that what looks like function call is actually
interpreted via the builder/MOP as the construction of a data structure,
which then creates the query on demand -- with template placeholder
filling as required.
Does D have the reflection capabilities to do this in any way, shape or
form similar tp the way the dynamic languages do it?
It depends on what you want to achieve. If you want just the above
example (and similar) to work, including lazy loading that would be
possible and not very hard.
If you want a more complete ORM and do things like (in Ruby):
person = Person.find_all_by_name_and_age("Joe", 15)
p person.name
That's a little harder. You could use opDispatch to make
"find_all_by_name_and_age" or "name" work, but not both. The problem is
that you can not overload methods on static and you are not forced to
call a static method prefixed with the class name.
class Foo
{
void bar () {};
static void bar () {};
}
Currently the above doesn't work. You can also call a static method on
an instance, which will also cause problems. What we want here is to be
able to have two versions of opDispatch, one static and one non-static.
--
/Jacob Carlborg
Re: how to build up the library..
On 2011-10-07 09:30, Russel Winder wrote: On Fri, 2011-10-07 at 08:51 +0200, Jacob Carlborg wrote: [ . . . ] I think it's important to have a solid low level database API which an object oriented abstraction can be built upon. Why object oriented? Why not functional? The point was that the low level should not be object oriented like, an ORM, to allow different high level abstractions built on top of the low level API. I don't think that the low level API should have functions which takes a struct or an object and fills it fields with results from a query. That's up to the high level API. -- /Jacob Carlborg
Re: how to build up the library..
Am 07.10.2011, 09:26 Uhr, schrieb Russel Winder :
On Fri, 2011-10-07 at 08:49 +0200, Jacob Carlborg wrote:
[ . . . ]
I think it's important to support the range interface (or if it's
simpler, the array interface). I think ActiveRecord has a good high
level API which allows to lazily evaluate SQL queries.
persons = Person.all # no query is performed here
persons = persons.where("name = 'Joe'") # or here
persons.each do |p| # The actual query is performed here
# do something with each Person
end
I don't think it would be strange to see something like this:
Person.where("age > 10").map{ |p| p.name } # using "map" on an SQL query
Not just Ruby an Rails with ActiveRecord. Groovy/Grails/GORM does what
is basically the same. Python cannot do the same in the same way Ruby
and Groovy can, but it has it equivalent (augmented with SQLAlchemy).
Basically it comes down to constructing dynamic queries using the
dynamic and MOP nature of the languages. The crucial keyword here is
Builder. The trick is that what looks like function call is actually
interpreted via the builder/MOP as the construction of a data structure,
which then creates the query on demand -- with template placeholder
filling as required.
Does D have the reflection capabilities to do this in any way, shape or
form similar tp the way the dynamic languages do it?
This has already been done in D. There is a nice library that assembles
queries at compile time and fetches results lazily into structs. The name?
Sorry, I forgot. But the author will probably show up here.
Re: how to build up the library..
On Fri, 2011-10-07 at 08:51 +0200, Jacob Carlborg wrote: [ . . . ] > I think it's important to have a solid low level database API which an > object oriented abstraction can be built upon. Why object oriented? Why not functional? The lesson form ORMs is that object oriented and relational can be linked with an adaptor but the impedance mismatch is high. But if you want to use an ORM use one, of course. The computational model having issued a query is just data transformation which is nicely functional, albeit possibly with side effects. But then the FP crowd have given everyone monads. -- Russel. = Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net 41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@russel.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder signature.asc Description: This is a digitally signed message part
Re: how to build up the library..
On Fri, 2011-10-07 at 08:49 +0200, Jacob Carlborg wrote:
[ . . . ]
> I think it's important to support the range interface (or if it's
> simpler, the array interface). I think ActiveRecord has a good high
> level API which allows to lazily evaluate SQL queries.
>
> persons = Person.all # no query is performed here
> persons = persons.where("name = 'Joe'") # or here
>
> persons.each do |p| # The actual query is performed here
> # do something with each Person
> end
>
> I don't think it would be strange to see something like this:
>
> Person.where("age > 10").map{ |p| p.name } # using "map" on an SQL query
Not just Ruby an Rails with ActiveRecord. Groovy/Grails/GORM does what
is basically the same. Python cannot do the same in the same way Ruby
and Groovy can, but it has it equivalent (augmented with SQLAlchemy).
Basically it comes down to constructing dynamic queries using the
dynamic and MOP nature of the languages. The crucial keyword here is
Builder. The trick is that what looks like function call is actually
interpreted via the builder/MOP as the construction of a data structure,
which then creates the query on demand -- with template placeholder
filling as required.
Does D have the reflection capabilities to do this in any way, shape or
form similar tp the way the dynamic languages do it?
--
Russel.
=
Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net
41 Buckmaster Roadm: +44 7770 465 077 xmpp: rus...@russel.org.uk
London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
signature.asc
Description: This is a digitally signed message part
Re: how to build up the library..
On 2011-10-07 07:24, Steve Teale wrote: Andrei, I was thinking the same thing about the C level interface sometime in the middle of last night. A straight translation of mysqld could be put there almost immediately. I have to make myself translate the character set stuff, and add that, but it's probably only a days work. As for the interface, I was influenced at the intermediate level by my memory of using C#. I think that a DB interface for D should be something that provides an understood set of facilities rather than an interface in the language sense that provides certain specific methods. This would allow a decent job to be done on individual databases strengths, while making it easy for D programmers to move from one to another. I will write up a summary of my thoughts later today, and post it here. Briefly, I had thought of variants, but am also attracted to the idea of 'compatible structs'. These would be struct definition that when fed to a suitable template function, could be determined to be a match for a stated query - a bit like isForwardRange ;=). Other templates would be provided to automate the process of 'applying' arrays of such structs to/from the database. Similar facilities could be provided for arrays of arrays of variants. At an even higher level one could have templates that implemented entire operations, like updating a batch of records. D's template facilities make such stuff easy to implement once you have certain primitives to base them on. I am at that point now. I have bridgehead and can consider breakout. Steve As I replied to another post: I think it's important to have a solid low level database API which later an object oriented abstraction can be built upon. In this low level API I don't think that results from SQL quires should be converted to structs. I think that's up to the higher, object oriented, level to do. -- /Jacob Carlborg
Re: how to build up the library..
On 2011-10-07 04:56, Piotr Szturmaj wrote: Nick Sabalausky wrote: "Andrei Alexandrescu" wrote in message news:j6ljvs$1if5$1...@digitalmars.com... For D in particular, I think the range aspect is secondary. Probably it will be more important to interoperate with Variant. Most APIs store columns in a Variant structure that captures all possible types of the database. Resultsets come as input ranges of array of Variant. It's not important to make them compatible with the input range interface because it's unlikely you'll pass that array of Variant directly to some algorithm. That being said, people will naturally expect to use the canonical range interface. It would probably be gratuitous to choose other primitive(s) than empty/front/popFront. Personally, I would *expect* a good DB API to provide statically-typed columns. I use D because I hate dynamic typing, not because it provides a Variant. Whether or not the DB API *also* has messy^H^H^H^H^HVariant capability for those who want it, I wouldn't care at all either way. What do you think about ddb(http://pszturmaj.github.com/ddb/db.html) ?. It provides statically typed columns or dynamic Variant columns. What you want is what you get :) I think it's important to have a solid low level database API which an object oriented abstraction can be built upon. -- /Jacob Carlborg
Re: how to build up the library..
On 2011-10-07 03:17, Andrei Alexandrescu wrote:
On 10/6/11 10:27 AM, Steve Teale wrote:
Andrei,
Are there guidelines for the Phobos process?
We follow the Boost review process:
http://www.boost.org/community/reviews.html
I have put a fair amount of work, two years ago and recently, into a
piece for
accessing MySQL via D. I'm looking at this as a bottom-up
investigation into what
might be a more generalized interface for database access.
I've got to the point where I can support a good range of database
operations, and
present result sets as arrays of D structs, which presumably implies
Ranges.
There's also lower level stuff to do more complex operations. So I'm
wondering if
I should take the first process steps. I'm retired, so I do have time
to spend on it.
The MySQL C api is not what you could call orthogonal - in
crystallography terms
it is probably Triclinic, so it may not have been the best place to
start. There's
another guy I've had some correspondence with who is working on
Postgres. He's
interested in the same idea.
What do you think?
I think this is a great initiative. Perhaps it would be nice to have a
C-level API translation in etc.mysql. Then we should have a driver-style
interface a la Java's JDBC, Perl's DBI, Python's PEP249 etc.
For D in particular, I think the range aspect is secondary. Probably it
will be more important to interoperate with Variant. Most APIs store
columns in a Variant structure that captures all possible types of the
database.
Resultsets come as input ranges of array of Variant. It's not important
to make them compatible with the input range interface because it's
unlikely you'll pass that array of Variant directly to some algorithm.
That being said, people will naturally expect to use the canonical range
interface. It would probably be gratuitous to choose other primitive(s)
than empty/front/popFront.
I think it's important to support the range interface (or if it's
simpler, the array interface). I think ActiveRecord has a good high
level API which allows to lazily evaluate SQL queries.
persons = Person.all # no query is performed here
persons = persons.where("name = 'Joe'") # or here
persons.each do |p| # The actual query is performed here
# do something with each Person
end
I don't think it would be strange to see something like this:
Person.where("age > 10").map{ |p| p.name } # using "map" on an SQL query
--
/Jacob Carlborg
Re: how to build up the library..
On 2011-10-06 21:44, Jesse Phillips wrote: Regan Heath Wrote: That's just it however, I am not generous enough with my time to be of use to you/D/phobos. I spend my free time doing other things, and not working on the code I have which could be of use. I suspect there are quite a number of people out there who are just like me. People who have some code which may be of use, in varying degrees of completeness from just an idea to a pretty much complete re-usable library which just needs the final touches etc. The problem is as you say, there are many out there like you. Even if some people take the rule of getting these polished, they couldn't keep up. Finding these people his hard. Even if we just get one contribution to Phobos through this method it would be better than nothing. -- /Jacob Carlborg
Re: how to build up the library..
Andrei, I was thinking the same thing about the C level interface sometime in the middle of last night. A straight translation of mysqld could be put there almost immediately. I have to make myself translate the character set stuff, and add that, but it's probably only a days work. As for the interface, I was influenced at the intermediate level by my memory of using C#. I think that a DB interface for D should be something that provides an understood set of facilities rather than an interface in the language sense that provides certain specific methods. This would allow a decent job to be done on individual databases strengths, while making it easy for D programmers to move from one to another. I will write up a summary of my thoughts later today, and post it here. Briefly, I had thought of variants, but am also attracted to the idea of 'compatible structs'. These would be struct definition that when fed to a suitable template function, could be determined to be a match for a stated query - a bit like isForwardRange ;=). Other templates would be provided to automate the process of 'applying' arrays of such structs to/from the database. Similar facilities could be provided for arrays of arrays of variants. At an even higher level one could have templates that implemented entire operations, like updating a batch of records. D's template facilities make such stuff easy to implement once you have certain primitives to base them on. I am at that point now. I have bridgehead and can consider breakout. Steve
Re: how to build up the library..
On 10/6/11 8:55 PM, Nick Sabalausky wrote: "Andrei Alexandrescu" wrote in message news:j6ljvs$1if5$1...@digitalmars.com... For D in particular, I think the range aspect is secondary. Probably it will be more important to interoperate with Variant. Most APIs store columns in a Variant structure that captures all possible types of the database. Resultsets come as input ranges of array of Variant. It's not important to make them compatible with the input range interface because it's unlikely you'll pass that array of Variant directly to some algorithm. That being said, people will naturally expect to use the canonical range interface. It would probably be gratuitous to choose other primitive(s) than empty/front/popFront. Personally, I would *expect* a good DB API to provide statically-typed columns. I use D because I hate dynamic typing, not because it provides a Variant. Whether or not the DB API *also* has messy^H^H^H^H^HVariant capability for those who want it, I wouldn't care at all either way. A statically-typed interface is nice when you know the schema a priori, but definitely there's a need for dynamically-created queries too. Andrei
Re: how to build up the library..
Nick Sabalausky wrote: "Andrei Alexandrescu" wrote in message news:j6ljvs$1if5$1...@digitalmars.com... For D in particular, I think the range aspect is secondary. Probably it will be more important to interoperate with Variant. Most APIs store columns in a Variant structure that captures all possible types of the database. Resultsets come as input ranges of array of Variant. It's not important to make them compatible with the input range interface because it's unlikely you'll pass that array of Variant directly to some algorithm. That being said, people will naturally expect to use the canonical range interface. It would probably be gratuitous to choose other primitive(s) than empty/front/popFront. Personally, I would *expect* a good DB API to provide statically-typed columns. I use D because I hate dynamic typing, not because it provides a Variant. Whether or not the DB API *also* has messy^H^H^H^H^HVariant capability for those who want it, I wouldn't care at all either way. What do you think about ddb(http://pszturmaj.github.com/ddb/db.html) ?. It provides statically typed columns or dynamic Variant columns. What you want is what you get :)
Re: how to build up the library..
"Andrei Alexandrescu" wrote in message news:j6ljvs$1if5$1...@digitalmars.com... > > For D in particular, I think the range aspect is secondary. Probably it > will be more important to interoperate with Variant. Most APIs store > columns in a Variant structure that captures all possible types of the > database. > > Resultsets come as input ranges of array of Variant. It's not important to > make them compatible with the input range interface because it's unlikely > you'll pass that array of Variant directly to some algorithm. That being > said, people will naturally expect to use the canonical range interface. > It would probably be gratuitous to choose other primitive(s) than > empty/front/popFront. > Personally, I would *expect* a good DB API to provide statically-typed columns. I use D because I hate dynamic typing, not because it provides a Variant. Whether or not the DB API *also* has messy^H^H^H^H^HVariant capability for those who want it, I wouldn't care at all either way.
Re: how to build up the library..
On 10/6/11 10:27 AM, Steve Teale wrote: Andrei, Are there guidelines for the Phobos process? We follow the Boost review process: http://www.boost.org/community/reviews.html I have put a fair amount of work, two years ago and recently, into a piece for accessing MySQL via D. I'm looking at this as a bottom-up investigation into what might be a more generalized interface for database access. I've got to the point where I can support a good range of database operations, and present result sets as arrays of D structs, which presumably implies Ranges. There's also lower level stuff to do more complex operations. So I'm wondering if I should take the first process steps. I'm retired, so I do have time to spend on it. The MySQL C api is not what you could call orthogonal - in crystallography terms it is probably Triclinic, so it may not have been the best place to start. There's another guy I've had some correspondence with who is working on Postgres. He's interested in the same idea. What do you think? I think this is a great initiative. Perhaps it would be nice to have a C-level API translation in etc.mysql. Then we should have a driver-style interface a la Java's JDBC, Perl's DBI, Python's PEP249 etc. For D in particular, I think the range aspect is secondary. Probably it will be more important to interoperate with Variant. Most APIs store columns in a Variant structure that captures all possible types of the database. Resultsets come as input ranges of array of Variant. It's not important to make them compatible with the input range interface because it's unlikely you'll pass that array of Variant directly to some algorithm. That being said, people will naturally expect to use the canonical range interface. It would probably be gratuitous to choose other primitive(s) than empty/front/popFront. Making the database interface generic over other SQL databases will probably be considerable more work, and will need study of existing APIs and input from the community. Thanks, Andrei
Re: how to build up the library..
Steve Teale Wrote: > Andrei, > > Are there guidelines for the Phobos process? > > I have put a fair amount of work, two years ago and recently, into a piece for > accessing MySQL via D. I'm looking at this as a bottom-up investigation into > what > might be a more generalized interface for database access. > > I've got to the point where I can support a good range of database > operations, and > present result sets as arrays of D structs, which presumably implies Ranges. > There's also lower level stuff to do more complex operations. So I'm > wondering if > I should take the first process steps. I'm retired, so I do have time to > spend on it. > > The MySQL C api is not what you could call orthogonal - in crystallography > terms > it is probably Triclinic, so it may not have been the best place to start. > There's > another guy I've had some correspondence with who is working on Postgres. He's > interested in the same idea. > > What do you think? > > Steve My suggestion, work with those you can find to help you. And keep these points in mind. * There is interest in having a standard Database interface. * It does not have to support ever obscure option of every database * The Database specific "glue layer" does not need to be part of Phobos, though the popular ones would be good. * The glue layer should expose the more intricate options and hopefully work with the generic interface. * You will have to keep pushing for it. If you have a good solid API that you think is ready for inclusion in Phobos you will get more attention and a more critical review. * The generic interface will be the main interest, and I think many envision it looking like C#'s LINQ over Java's prepare and execute. (C# has this too, I couldn't figure out how to use LINQ as it seemed to require more setup, I think D can do better)
Re: how to build up the library..
Regan Heath Wrote: > That's just it however, I am not generous enough with my time to be of use > to you/D/phobos. I spend my free time doing other things, and not working > on the code I have which could be of use. I suspect there are quite a > number of people out there who are just like me. People who have some > code which may be of use, in varying degrees of completeness from just an > idea to a pretty much complete re-usable library which just needs the > final touches etc. The problem is as you say, there are many out there like you. Even if some people take the rule of getting these polished, they couldn't keep up. Finding these people his hard. > The size of a contribution for phobos is (one complete module) whereas in > many open source projects you can contribute as little as a line of > code/fix for a problem. The goal of this idea is to lower the bar for > "contributions" and "contributors" to include more people and more code > until even very small ideas/examples may eventually contribute to the > whole if only indirectly. Contributions can be of any size. You don't even need to use Git/pull requests. Just attach a diff to a bug in BugZilla. And these changes don't go throw the formal review, just those with Phobos repo access. What you are looking for is Dsource, or Github, or any other source repository. You place your code up on the web and license it under Boost, someone can modify it for Phobos. But the problem is, organizing all of them. People don't have interest in placing projects on Dsource anymore because it can't be organized well and it is relies on one person to update the software. So people go else where, but they don't want to take the time to manage what their project does over on Wiki4D, so someone else does it but isn't informed when updates need made. So please keep writing software for yourself. Choose an appropriate license for how you feel it should be used and do what you do.
Re: how to build up the library..
On Thu, 06 Oct 2011 15:47:03 +0100, Andrei Alexandrescu wrote: On 10/6/11 5:25 AM, Regan Heath wrote: I've been lurking and contributing to this news group on and off for years (since 2005/2006, maybe earlier). But, in all that time I have contributed nothing to Phobos. I have contributed a set of digest/hashing routines to Tango, because at the time Phobos wasn't taking submissions. I have been meaning to clean my originals of these up, get them properly documented, etc and submit them for review for Phobos, but I just haven't had the time, and/or inclination to do so (my free time is precious and I've just not been feeling the urge/itch to code.. blame minecraft :p) However, I am more than happy to hand them off in their current state to anyone who does have both time and inclination .. and I wonder how many other blocks of code are out there, just like mine, just waiting for the right person to take charge of them. Would this be useful do you think? Or would the time it takes someone to pick up new code, learn it, fine tune it and document it.. etc be more than if they just started again from scratch. It seems to me that sometimes, all that is needed to get a new module off the ground is a working prototype for the guts of it, which is what people like me who have some experience/knowledge but little time/energy could do, before handing it to someone who has a better idea of the D/Phobos 'way' and can organise the guts into a well formed module which complies with D style and Phobos guidelines etc. Part of what puts people off (I suspect) is the 'relative' complexity of submitting code (obtaining/learning GIT etc), the standard the code needs to be at (well organised, documented etc), and the implied promise/commitment that submitting code brings with it (that you'll hang around and maintain it). But, what if we were to create a system where people could submit code, no strings attached, in any state (they would define the state they believe it to be in) for the more dedicated contributors to pick up, clean up, and include as and when they could? It could be as simple as a web page, where code is pasted, files attached, and a license waiver agreed to. I know I have several pieces of code floating about the place which do useful things and would be motivated to create more if it would help the effort, and I could get a mention in the comments at the top of the finished module :p Thanks for your thoughts and generosity. It's great you are sharing your code in the hope it adds value to the community. That's just it however, I am not generous enough with my time to be of use to you/D/phobos. I spend my free time doing other things, and not working on the code I have which could be of use. I suspect there are quite a number of people out there who are just like me. People who have some code which may be of use, in varying degrees of completeness from just an idea to a pretty much complete re-usable library which just needs the final touches etc. If we collect these pieces of code somewhere and make it searchable then anyone could pick up a piece and re-use it, or work on it, or .. well anything they want. That's all I'm suggesting, that we create a "well of lost code" and let people dump things there. Also, if past is indicative of future, it's possible but unlikely that someone will take your code, bring it to perfection, and go through the hurdles of submitting it to Phobos. Making a body of code into a reusable library is often comparable work with writing the code itself. It's also seen by many as unpleasant work, which means most people would be more motivated to do it for their own work rather than others'. You're probably right, people prefer their own code. And, doing the work required to get something to the level required for Phobos is not as pleasant as hacking away at more 'exciting' things, this is exactly why my code remains incomplete (from a Phobos submission point of view). But, they don't have to work on 'my' code, they might take it and tear it to pieces and only really take the ideas away to create something new. The point is, it's a starting point. Things have changed a lot in the way D is evolving. This includes migration to git, opening the compiler, runtime, and standard library for contributions, and a review process for adding to the standard library. Focusing on the last aspect, this has been a big change from the previous approach in which the standard library was closed, and from an awkward stage in which we were inclined to accept somewhat random contributions (encoding, xml, json) without much due process. By any measure I can think of, establishing a review process has been a tremendous success, and everything else failed. We have std.datetime, std.parallelism to show, and there are many more to come. The difficulty in carrying through a Phobos submission also means establishing a stro
Re: how to build up the library..
Andrei, Are there guidelines for the Phobos process? I have put a fair amount of work, two years ago and recently, into a piece for accessing MySQL via D. I'm looking at this as a bottom-up investigation into what might be a more generalized interface for database access. I've got to the point where I can support a good range of database operations, and present result sets as arrays of D structs, which presumably implies Ranges. There's also lower level stuff to do more complex operations. So I'm wondering if I should take the first process steps. I'm retired, so I do have time to spend on it. The MySQL C api is not what you could call orthogonal - in crystallography terms it is probably Triclinic, so it may not have been the best place to start. There's another guy I've had some correspondence with who is working on Postgres. He's interested in the same idea. What do you think? Steve
Re: how to build up the library..
On 10/6/11 5:25 AM, Regan Heath wrote: I've been lurking and contributing to this news group on and off for years (since 2005/2006, maybe earlier). But, in all that time I have contributed nothing to Phobos. I have contributed a set of digest/hashing routines to Tango, because at the time Phobos wasn't taking submissions. I have been meaning to clean my originals of these up, get them properly documented, etc and submit them for review for Phobos, but I just haven't had the time, and/or inclination to do so (my free time is precious and I've just not been feeling the urge/itch to code.. blame minecraft :p) However, I am more than happy to hand them off in their current state to anyone who does have both time and inclination .. and I wonder how many other blocks of code are out there, just like mine, just waiting for the right person to take charge of them. Would this be useful do you think? Or would the time it takes someone to pick up new code, learn it, fine tune it and document it.. etc be more than if they just started again from scratch. It seems to me that sometimes, all that is needed to get a new module off the ground is a working prototype for the guts of it, which is what people like me who have some experience/knowledge but little time/energy could do, before handing it to someone who has a better idea of the D/Phobos 'way' and can organise the guts into a well formed module which complies with D style and Phobos guidelines etc. Part of what puts people off (I suspect) is the 'relative' complexity of submitting code (obtaining/learning GIT etc), the standard the code needs to be at (well organised, documented etc), and the implied promise/commitment that submitting code brings with it (that you'll hang around and maintain it). But, what if we were to create a system where people could submit code, no strings attached, in any state (they would define the state they believe it to be in) for the more dedicated contributors to pick up, clean up, and include as and when they could? It could be as simple as a web page, where code is pasted, files attached, and a license waiver agreed to. I know I have several pieces of code floating about the place which do useful things and would be motivated to create more if it would help the effort, and I could get a mention in the comments at the top of the finished module :p Thanks for your thoughts and generosity. It's great you are sharing your code in the hope it adds value to the community. I'm sure it does, but I am skeptical the impact is nearly as strong as it could. Also, if past is indicative of future, it's possible but unlikely that someone will take your code, bring it to perfection, and go through the hurdles of submitting it to Phobos. Making a body of code into a reusable library is often comparable work with writing the code itself. It's also seen by many as unpleasant work, which means most people would be more motivated to do it for their own work rather than others'. Things have changed a lot in the way D is evolving. This includes migration to git, opening the compiler, runtime, and standard library for contributions, and a review process for adding to the standard library. Focusing on the last aspect, this has been a big change from the previous approach in which the standard library was closed, and from an awkward stage in which we were inclined to accept somewhat random contributions (encoding, xml, json) without much due process. By any measure I can think of, establishing a review process has been a tremendous success, and everything else failed. We have std.datetime, std.parallelism to show, and there are many more to come. The difficulty in carrying through a Phobos submission also means establishing a strong esprit de corps, a commitment to quality, and a sense of meaningfully moving things forward together. As far as I know many other open source projects follow a similar model. There is essentially no evidence in this space to suggest that your hypothetical model of development would work. Andrei
Re: how to build up the library..
On 2011-10-06 12:25, Regan Heath wrote: It's obvious from my last post that I think a change should be allowed. I think you'd agree, were it not for the urgent need of other library components. I've been lurking and contributing to this news group on and off for years (since 2005/2006, maybe earlier). But, in all that time I have contributed nothing to Phobos. I have contributed a set of digest/hashing routines to Tango, because at the time Phobos wasn't taking submissions. I have been meaning to clean my originals of these up, get them properly documented, etc and submit them for review for Phobos, but I just haven't had the time, and/or inclination to do so (my free time is precious and I've just not been feeling the urge/itch to code.. blame minecraft :p) However, I am more than happy to hand them off in their current state to anyone who does have both time and inclination .. and I wonder how many other blocks of code are out there, just like mine, just waiting for the right person to take charge of them. Would this be useful do you think? Or would the time it takes someone to pick up new code, learn it, fine tune it and document it.. etc be more than if they just started again from scratch. It seems to me that sometimes, all that is needed to get a new module off the ground is a working prototype for the guts of it, which is what people like me who have some experience/knowledge but little time/energy could do, before handing it to someone who has a better idea of the D/Phobos 'way' and can organise the guts into a well formed module which complies with D style and Phobos guidelines etc. Part of what puts people off (I suspect) is the 'relative' complexity of submitting code (obtaining/learning GIT etc), the standard the code needs to be at (well organised, documented etc), and the implied promise/commitment that submitting code brings with it (that you'll hang around and maintain it). But, what if we were to create a system where people could submit code, no strings attached, in any state (they would define the state they believe it to be in) for the more dedicated contributors to pick up, clean up, and include as and when they could? It could be as simple as a web page, where code is pasted, files attached, and a license waiver agreed to. I know I have several pieces of code floating about the place which do useful things and would be motivated to create more if it would help the effort, and I could get a mention in the comments at the top of the finished module :p I think it sounds like a good idea, assuming there actually is people willing to do these clean ups. -- /Jacob Carlborg
Re: how to build up the library..
On Thu, 06 Oct 2011 10:44:42 +0100, Regan Heath wrote: On Wed, 05 Oct 2011 16:44:31 +0100, Andrei Alexandrescu wrote: On 10/5/11 10:27 AM, Regan Heath wrote: I understand the issue, and the point you're making below, and I agree completely. At the same time, this particular change being as simple as it is, and as obviously beneficial as I hope I have managed to show, would have taken less time to simply change than it has taken to argue about. There is the issue that every small change creates a precedent for similar or smaller changes. I spent this much time on this particular issue hoping that it would improve an entire trend going forward. I think it may have backfired somewhat. Now people are going to think no change is possible and Andrei is an 'Ogre' when it comes to his own modules. In situations like this some sort of precedent or impression will always be created. The best you can do is take control of it, and clearly define it. In this case clearly define the conditions under which the change is allowed .. or not allowed. :p It's obvious from my last post that I think a change should be allowed. I think you'd agree, were it not for the urgent need of other library components. I've been lurking and contributing to this news group on and off for years (since 2005/2006, maybe earlier). But, in all that time I have contributed nothing to Phobos. I have contributed a set of digest/hashing routines to Tango, because at the time Phobos wasn't taking submissions. I have been meaning to clean my originals of these up, get them properly documented, etc and submit them for review for Phobos, but I just haven't had the time, and/or inclination to do so (my free time is precious and I've just not been feeling the urge/itch to code.. blame minecraft :p) However, I am more than happy to hand them off in their current state to anyone who does have both time and inclination .. and I wonder how many other blocks of code are out there, just like mine, just waiting for the right person to take charge of them. Would this be useful do you think? Or would the time it takes someone to pick up new code, learn it, fine tune it and document it.. etc be more than if they just started again from scratch. It seems to me that sometimes, all that is needed to get a new module off the ground is a working prototype for the guts of it, which is what people like me who have some experience/knowledge but little time/energy could do, before handing it to someone who has a better idea of the D/Phobos 'way' and can organise the guts into a well formed module which complies with D style and Phobos guidelines etc. Part of what puts people off (I suspect) is the 'relative' complexity of submitting code (obtaining/learning GIT etc), the standard the code needs to be at (well organised, documented etc), and the implied promise/commitment that submitting code brings with it (that you'll hang around and maintain it). But, what if we were to create a system where people could submit code, no strings attached, in any state (they would define the state they believe it to be in) for the more dedicated contributors to pick up, clean up, and include as and when they could? It could be as simple as a web page, where code is pasted, files attached, and a license waiver agreed to. I know I have several pieces of code floating about the place which do useful things and would be motivated to create more if it would help the effort, and I could get a mention in the comments at the top of the finished module :p -- Using Opera's revolutionary email client: http://www.opera.com/mail/
