Re: [std.database]
On Sun, 16 Oct 2011 02:13:05 -0400, Steve Teale wrote: There's a discussion going on about Windows header files that has discussed whether header files can be copyright. Header files may be an issue with the database implementations. For example my mysql.d is a straight translation of mysql.h (and a couple of others). Does that mean it is tainted by GPL and I can't make it Boost? A direct translation is a derivative work. So yes, it must be GPL. However, there must be ways around this. I believe headers have certain rules in most licenses. You will definitely need some sort of non-translated header though. I'm not a license expert, so I don't know to what lengths you need to go to re-license the header. However, what about mysql itself? If the header is GPL, so is the library, no? I'm assuming you are not reimplementing the mysql client lib? Linking against a GPL library is definitely not going to be acceptable in a phobos module. -Steve
Re: [std.database]
On Mon, 17 Oct 2011 09:42:13 -0400, Steven Schveighoffer wrote: > On Sun, 16 Oct 2011 02:13:05 -0400, Steve Teale > wrote: > >> There's a discussion going on about Windows header files that has >> discussed whether header files can be copyright. >> >> Header files may be an issue with the database implementations. For >> example my mysql.d is a straight translation of mysql.h (and a couple >> of others). Does that mean it is tainted by GPL and I can't make it >> Boost? > > A direct translation is a derivative work. So yes, it must be GPL. > > However, there must be ways around this. I believe headers have certain > rules in most licenses. > > You will definitely need some sort of non-translated header though. I'm > not a license expert, so I don't know to what lengths you need to go to > re-license the header. > > However, what about mysql itself? If the header is GPL, so is the > library, no? I'm assuming you are not reimplementing the mysql client > lib? Linking against a GPL library is definitely not going to be > acceptable in a phobos module. > > -Steve Hmm, I just did a quick check, and the MySQL client/server protocol is GPL also, so there's nowhere to go. How do Python and PHP communicate with MySQL. Is it just that they have the clout to get a dispensation from MySQL AB? Does this stuff have to go in some repository like the proposed Deimos (a figure representing dread in Greek Mythology) where you will presumably often encounter dread licensing gotchas? Steve
Re: [std.database]
Steve Teale Wrote: > Hmm, I just did a quick check, and the MySQL client/server protocol is > GPL also, so there's nowhere to go. That was fixed. > How do Python and PHP communicate with MySQL. Is it just that they have > the clout to get a dispensation from MySQL AB? MySQL license has FLOSS exception: opensource software can use MySQL, but not *relicense* its code. http://mysql.com/about/legal/licensing/foss-exception/ Though boost is not in the list.
Re: [std.database]
On Mon, 17 Oct 2011 10:25:13 -0400, Steve Teale wrote: On Mon, 17 Oct 2011 09:42:13 -0400, Steven Schveighoffer wrote: On Sun, 16 Oct 2011 02:13:05 -0400, Steve Teale wrote: There's a discussion going on about Windows header files that has discussed whether header files can be copyright. Header files may be an issue with the database implementations. For example my mysql.d is a straight translation of mysql.h (and a couple of others). Does that mean it is tainted by GPL and I can't make it Boost? A direct translation is a derivative work. So yes, it must be GPL. However, there must be ways around this. I believe headers have certain rules in most licenses. You will definitely need some sort of non-translated header though. I'm not a license expert, so I don't know to what lengths you need to go to re-license the header. However, what about mysql itself? If the header is GPL, so is the library, no? I'm assuming you are not reimplementing the mysql client lib? Linking against a GPL library is definitely not going to be acceptable in a phobos module. -Steve Hmm, I just did a quick check, and the MySQL client/server protocol is GPL also, so there's nowhere to go. Protocol cannot be copyrighted. A protocol is carefully formatted data, but not *MySQL's* data, it's *your* data. If they want to attempt to say my passing "select name, bar, gobbledegook from myPrivateDatabase" to a server makes my code GPL, be my guest, I don't even think I'd need a lawyer to defend that :) Here is a good post discussing it: http://krow.livejournal.com/684068.html?thread=2670116 But the library can be copyrighted (and the protocol description). Given mysql's sales model (and the company behind it), you would need a very meticulously documented process to clean-room implement it in a way that could be defended, and even then Walter may not go for inclusion in phobos, he is allergic to even the *notion* that something might be challenged in court being in D/phobos. How do Python and PHP communicate with MySQL. Is it just that they have the clout to get a dispensation from MySQL AB? little searching reveals: http://www.mysql.com/about/legal/licensing/foss-exception/ Note PHP and Python are both in the license list, whereas Boost is not :( Does this stuff have to go in some repository like the proposed Deimos (a figure representing dread in Greek Mythology) where you will presumably often encounter dread licensing gotchas? I'd imagine so. Another option is to reimplement libmysql. Sorry :( -Steve
Re: [std.database]
> A direct translation is a derivative work. So yes, it must be GPL. > > However, there must be ways around this. I believe headers have certain > rules in most licenses. > Steve, do you think this provides any relief> http://www.mysql.com/about/legal/licensing/foss-exception/ Steve
Re: [std.database]
On Mon, 17 Oct 2011 10:56:45 -0400, Kagamin wrote: Steve Teale Wrote: Hmm, I just did a quick check, and the MySQL client/server protocol is GPL also, so there's nowhere to go. That was fixed. That is good news! do you have a supporting link? Or is it something that quietly went away? -Steve
Re: [std.database]
Steven Schveighoffer Wrote: > That is good news! do you have a supporting link? Or is it something > that quietly went away? http://forge.mysql.com/w/index.php?title=MySQL_Internals_ClientServer_Protocol&diff=5078&oldid=4374 I have shown this to a guy who was going to reimplement a mysql client based on those docs.
Re: [std.database]
Steve Teale Wrote: > There's a discussion going on about Windows header files that has > discussed whether header files can be copyright. > > Header files may be an issue with the database implementations. For > example my mysql.d is a straight translation of mysql.h (and a couple of > others). Does that mean it is tainted by GPL and I can't make it Boost? You can't license mysql.d under terms of boost, you can ask Oracle to include boost into the foss exception list and license mysql.d under terms of mysql foss exception, but first you should ask Walter whether he wants code under terms of foss exception in phobos.
Re: [std.database]
On Mon, 17 Oct 2011 11:00:13 -0400, Steve Teale wrote: A direct translation is a derivative work. So yes, it must be GPL. However, there must be ways around this. I believe headers have certain rules in most licenses. Steve, do you think this provides any relief> http://www.mysql.com/about/legal/licensing/foss-exception/ I know our messages crossed paths probably, but for completeness, no, I don't think this allows relief. Boost is not listed as an allowed exception. But this doesn't matter anyways. The *end product*, not phobos, must be licensed free and open source under the boost license. This is an important distinction. This means, effectively, even if boost was included on the list, and we put mysql client bindings in phobos, in order to ship with mysql client library the user of phobos would have to license *their* product under boost! Even if they didn't ship the source for their application, anyone who obtained the application source through any means would be free to copy it at will. This limitation is too severe for inclusion in the core library of a language such as D, which might be used in closed-source settings. Note that php and python are somewhat open-source anyways since they are interpreted, and commonly you do not need to distribute the source of your server-side code for it to be used. -Steve
Re: [std.database]
On Mon, 17 Oct 2011 11:11:37 -0400, Kagamin wrote: Steven Schveighoffer Wrote: That is good news! do you have a supporting link? Or is it something that quietly went away? http://forge.mysql.com/w/index.php?title=MySQL_Internals_ClientServer_Protocol&diff=5078&oldid=4374 I have shown this to a guy who was going to reimplement a mysql client based on those docs. That is good! Since 2007, huh I'm surprised I could find no discussion on this, it seems like it would be a big deal for those who wanted to reimplement mysql clients. -Steve
Re: [std.database]
Kagamin wrote: Steven Schveighoffer Wrote: That is good news! do you have a supporting link? Or is it something that quietly went away? http://forge.mysql.com/w/index.php?title=MySQL_Internals_ClientServer_Protocol&diff=5078&oldid=4374 I have shown this to a guy who was going to reimplement a mysql client based on those docs. You probably meant me. If we create MySQL client without using C bindings then we would have one of the fastest bindings at all. It may attract some people who write web apps to D. The same applies to PostgreSQL for which I wrote client without using a C binding. I know it adds some maintaining effort but we may choose opportunistic approach. Use direct access where possible and fallback to C lib otherwise. Another argument for using direct access is easy deployment of applications, especially when client resides in the std library.
Re: [std.database]
Steven Schveighoffer Wrote: > That is good! Since 2007, huh I'm surprised I could find no > discussion on this http://krow.livejournal.com/684068.html?thread=2674468#t2674468
Re: [std.database]
Steve Teale Wrote: > Header files may be an issue with the database implementations. For > example my mysql.d is a straight translation of mysql.h (and a couple of > others). Does that mean it is tainted by GPL and I can't make it Boost? As an alternative why not make ODBC bindings? This way you'll be able to use virtually any database. ODBC is the most important database binding.
Re: [std.database]
On Mon, 17 Oct 2011 11:37:43 -0400, Kagamin wrote: Steven Schveighoffer Wrote: That is good! Since 2007, huh I'm surprised I could find no discussion on this http://krow.livejournal.com/684068.html?thread=2674468#t2674468 Yes, I saw that. But that is hardly "discussion in the community." :) In any case, it's good that it's gone. I'm all for a d-based mysql protocol implementation, and it looks like the only legal option anyways (if you want it in phobos, that is). -Steve
Re: [std.database]
On 17.10.2011 17:26, Piotr Szturmaj wrote: You probably meant me. If we create MySQL client without using C bindings then we would have one of the fastest bindings at all. It may attract some people who write web apps to D. The same applies to PostgreSQL for which I wrote client without using a C binding. Why would a reimplementation be much faster? Is the C library "slow"? Or is there any overhead I'm not thinking of?
Re: [std.database]
On Mon, 17 Oct 2011 11:40:44 -0400, Kagamin wrote: > Steve Teale Wrote: > >> Header files may be an issue with the database implementations. For >> example my mysql.d is a straight translation of mysql.h (and a couple >> of others). Does that mean it is tainted by GPL and I can't make it >> Boost? > > As an alternative why not make ODBC bindings? This way you'll be able to > use virtually any database. ODBC is the most important database binding. We are/were heading down the road of having ODBC as one of the options, but what's the status of its header files I wonder? But it sounds to me as if Walter's CAPI concept should be expanded to have three directories - C headers, equivalent D interfaces, and derived works. Steve
Re: [std.database]
simendsjo wrote: On 17.10.2011 17:26, Piotr Szturmaj wrote: You probably meant me. If we create MySQL client without using C bindings then we would have one of the fastest bindings at all. It may attract some people who write web apps to D. The same applies to PostgreSQL for which I wrote client without using a C binding. Why would a reimplementation be much faster? Is the C library "slow"? Or is there any overhead I'm not thinking of? There is always a function call overhead and also most of the fields are returned as strings, whereas in underlying protocol they are encoded binary, i.e. int is not sent as its decimal string representation but as 4 bytes. This saves some time taken by string to int conversion (parsing).
Re: [std.database]
On 17.10.2011 17:55, Piotr Szturmaj wrote: simendsjo wrote: On 17.10.2011 17:26, Piotr Szturmaj wrote: You probably meant me. If we create MySQL client without using C bindings then we would have one of the fastest bindings at all. It may attract some people who write web apps to D. The same applies to PostgreSQL for which I wrote client without using a C binding. Why would a reimplementation be much faster? Is the C library "slow"? Or is there any overhead I'm not thinking of? There is always a function call overhead and also most of the fields are returned as strings, whereas in underlying protocol they are encoded binary, i.e. int is not sent as its decimal string representation but as 4 bytes. This saves some time taken by string to int conversion (parsing). I see. I've looked a bit at the C library, and it seems the protocol isn't very stable. Trying to support older MySQL versions and keep up with protocol changes might be (too) tedious.
Re: [std.database]
simendsjo wrote: On 17.10.2011 17:55, Piotr Szturmaj wrote: simendsjo wrote: On 17.10.2011 17:26, Piotr Szturmaj wrote: You probably meant me. If we create MySQL client without using C bindings then we would have one of the fastest bindings at all. It may attract some people who write web apps to D. The same applies to PostgreSQL for which I wrote client without using a C binding. Why would a reimplementation be much faster? Is the C library "slow"? Or is there any overhead I'm not thinking of? There is always a function call overhead and also most of the fields are returned as strings, whereas in underlying protocol they are encoded binary, i.e. int is not sent as its decimal string representation but as 4 bytes. This saves some time taken by string to int conversion (parsing). I see. I've looked a bit at the C library, and it seems the protocol isn't very stable. Trying to support older MySQL versions and keep up with protocol changes might be (too) tedious. PostgreSQL's protocol is stable since 2003, but MySQL's is not very friendly indeed. Phobos might follow opportunistic path and support direct access with recent MySQL versions and C wrapper for older ones.
Re: [std.database]
> PostgreSQL's protocol is stable since 2003, but MySQL's is not very > friendly indeed. Phobos might follow opportunistic path and support > direct access with recent MySQL versions and C wrapper for older ones. But it looks like the C wrapper approach for MySQL won't fly for Phobos because of the GPL taint. MySQL support might have to be consigned to the Deimos 'derived works' directory. Either that or Phobos only supports versions > 5.xx. Steve
Re: [std.database]
On 17.10.2011 18:16, Piotr Szturmaj wrote: simendsjo wrote: On 17.10.2011 17:55, Piotr Szturmaj wrote: simendsjo wrote: On 17.10.2011 17:26, Piotr Szturmaj wrote: You probably meant me. If we create MySQL client without using C bindings then we would have one of the fastest bindings at all. It may attract some people who write web apps to D. The same applies to PostgreSQL for which I wrote client without using a C binding. Why would a reimplementation be much faster? Is the C library "slow"? Or is there any overhead I'm not thinking of? There is always a function call overhead and also most of the fields are returned as strings, whereas in underlying protocol they are encoded binary, i.e. int is not sent as its decimal string representation but as 4 bytes. This saves some time taken by string to int conversion (parsing). I see. I've looked a bit at the C library, and it seems the protocol isn't very stable. Trying to support older MySQL versions and keep up with protocol changes might be (too) tedious. PostgreSQL's protocol is stable since 2003, but MySQL's is not very friendly indeed. Phobos might follow opportunistic path and support direct access with recent MySQL versions and C wrapper for older ones. Since 2003? That's pretty impressive! About MySql: The reason D is missing a lot of wrappers and libraries is the lack of manpower. This "duplicated" effort sounds like too much job at the current time. I think more people will flock to D once modules such as database and web-programming exists, and then more focus can go into optimizing.
Re: [std.database]
> > As an alternative why not make ODBC bindings? This way you'll be able to > use virtually any database. ODBC is the most important database binding. I can't find any definitive source for the ODBC header files. I've seen various versions that seem to make conflicting copyright claims or to have conflicting license statements. Does anyone have what might be considered to be a definitive version of these? Steve
Re: [std.database]
Am 17.10.2011, 18:38 Uhr, schrieb Steve Teale : PostgreSQL's protocol is stable since 2003, but MySQL's is not very friendly indeed. Phobos might follow opportunistic path and support direct access with recent MySQL versions and C wrapper for older ones. But it looks like the C wrapper approach for MySQL won't fly for Phobos because of the GPL taint. MySQL support might have to be consigned to the Deimos 'derived works' directory. Either that or Phobos only supports versions > 5.xx. Steve Do people not upgrade their database to MySQL 5? I never had to deal with that and especially large complicated databases. If it is easy to upgrade to MySQL 5 and it is faster and more secure there are probably few *actively developed* projects accessing MySQL 4 DBs. (MySQL 5 is pretty much exactly 6 years old now.)
Re: [std.database]
On Monday, October 17, 2011 09:38 Steve Teale wrote: > > PostgreSQL's protocol is stable since 2003, but MySQL's is not very > > friendly indeed. Phobos might follow opportunistic path and support > > direct access with recent MySQL versions and C wrapper for older ones. > > But it looks like the C wrapper approach for MySQL won't fly for Phobos > because of the GPL taint. MySQL support might have to be consigned to the > Deimos 'derived works' directory. Either that or Phobos only supports > versions > 5.xx. However, if we go with an appropriately pluggable approach with the DB engines, then it should be perfectly possible to have a 3rd party library which provides a DB engine for mysql which you can then use with Phobos if your program is GPL or GPL-compatible. So, no it won't be in Phobos if we have to use the C headers, but that doesn't necssarily mean that it couldn't be used with Phobos' DB solution. - Jonathan M Davis
Re: [std.database]
Am 17.10.2011, 19:46 Uhr, schrieb Marco Leise : Am 17.10.2011, 18:38 Uhr, schrieb Steve Teale : PostgreSQL's protocol is stable since 2003, but MySQL's is not very friendly indeed. Phobos might follow opportunistic path and support direct access with recent MySQL versions and C wrapper for older ones. But it looks like the C wrapper approach for MySQL won't fly for Phobos because of the GPL taint. MySQL support might have to be consigned to the Deimos 'derived works' directory. Either that or Phobos only supports versions > 5.xx. Steve Do people not upgrade their database to MySQL 5? I never had to deal with that and especially large complicated databases. If it is easy to upgrade to MySQL 5 and it is faster and more secure there are probably few *actively developed* projects accessing MySQL 4 DBs. (MySQL 5 is pretty much exactly 6 years old now.) I found this: http://www.gentoo.org/doc/en/mysql-upgrading.xml#doc_chap1 Leaving away the safety net it looks like a call to mysql_upgrade_shell is all that's needed in the general case.
Re: [std.database]
On 17.10.2011 19:46, Marco Leise wrote: Am 17.10.2011, 18:38 Uhr, schrieb Steve Teale : PostgreSQL's protocol is stable since 2003, but MySQL's is not very friendly indeed. Phobos might follow opportunistic path and support direct access with recent MySQL versions and C wrapper for older ones. But it looks like the C wrapper approach for MySQL won't fly for Phobos because of the GPL taint. MySQL support might have to be consigned to the Deimos 'derived works' directory. Either that or Phobos only supports versions > 5.xx. Steve Do people not upgrade their database to MySQL 5? I never had to deal with that and especially large complicated databases. If it is easy to upgrade to MySQL 5 and it is faster and more secure there are probably few *actively developed* projects accessing MySQL 4 DBs. (MySQL 5 is pretty much exactly 6 years old now.) You'll always find people not upgrading technology for various reasons. We continue to use MySQL 4 at work for an internal legacy application because it just works and we _know_ upgrading will be a lot of work. And see how IE6 is still in use. I'm not advocating that D should strive to support legacy technologies though. I don't think Win98 or earlier are supported either.
Re: [std.database]
On 2011-10-17 19:55, Jonathan M Davis wrote: On Monday, October 17, 2011 09:38 Steve Teale wrote: PostgreSQL's protocol is stable since 2003, but MySQL's is not very friendly indeed. Phobos might follow opportunistic path and support direct access with recent MySQL versions and C wrapper for older ones. But it looks like the C wrapper approach for MySQL won't fly for Phobos because of the GPL taint. MySQL support might have to be consigned to the Deimos 'derived works' directory. Either that or Phobos only supports versions> 5.xx. However, if we go with an appropriately pluggable approach with the DB engines, then it should be perfectly possible to have a 3rd party library which provides a DB engine for mysql which you can then use with Phobos if your program is GPL or GPL-compatible. So, no it won't be in Phobos if we have to use the C headers, but that doesn't necssarily mean that it couldn't be used with Phobos' DB solution. - Jonathan M Davis I think that the Phobos database API needs to support database drivers written by third part users, that can be easily plugged in, for exactly the above mentioned reason. -- /Jacob Carlborg
Re: [std.database]
On Monday, October 17, 2011 11:07 simendsjo wrote: > On 17.10.2011 19:46, Marco Leise wrote: > > Am 17.10.2011, 18:38 Uhr, schrieb Steve Teale > > > > : > >>> PostgreSQL's protocol is stable since 2003, but MySQL's is not very > >>> friendly indeed. Phobos might follow opportunistic path and support > >>> direct access with recent MySQL versions and C wrapper for older ones. > >> > >> But it looks like the C wrapper approach for MySQL won't fly for Phobos > >> because of the GPL taint. MySQL support might have to be consigned to > >> the Deimos 'derived works' directory. Either that or Phobos only > >> supports versions > 5.xx. > >> > >> Steve > > > > Do people not upgrade their database to MySQL 5? I never had to deal > > with that and especially large complicated databases. If it is easy to > > upgrade to MySQL 5 and it is faster and more secure there are probably > > few *actively developed* projects accessing MySQL 4 DBs. (MySQL 5 is > > pretty much exactly 6 years old now.) > > You'll always find people not upgrading technology for various reasons. > We continue to use MySQL 4 at work for an internal legacy application > because it just works and we _know_ upgrading will be a lot of work. > And see how IE6 is still in use. > > I'm not advocating that D should strive to support legacy technologies > though. I don't think Win98 or earlier are supported either. There is code in druntime and Phobos which special-cases for Windows 98 and earlier (e.g. std.file using the A functions instead of the W functions if the version of Windows that you're running on is too old to have the W functions). Now, personally, I would love it if we just said that you have to have at least Win2k if not WinXP. It would be _great_ to be able to assume at least Vista, since it added a number of useful functions, but XP is still too prevalent for that to be reasonable. Regardless, supporting older versions of Windows is just plain irritating, since it restricts what you can do, and D is new enough and Win2k old enough that I find it perfectly reasonable to insist that you have WinXP or newer, but that's not what we're doing at this point. - Jonathan M Davis
Re: [std.database]
On 2011-10-17 20:07, simendsjo wrote: On 17.10.2011 19:46, Marco Leise wrote: Am 17.10.2011, 18:38 Uhr, schrieb Steve Teale : PostgreSQL's protocol is stable since 2003, but MySQL's is not very friendly indeed. Phobos might follow opportunistic path and support direct access with recent MySQL versions and C wrapper for older ones. But it looks like the C wrapper approach for MySQL won't fly for Phobos because of the GPL taint. MySQL support might have to be consigned to the Deimos 'derived works' directory. Either that or Phobos only supports versions > 5.xx. Steve Do people not upgrade their database to MySQL 5? I never had to deal with that and especially large complicated databases. If it is easy to upgrade to MySQL 5 and it is faster and more secure there are probably few *actively developed* projects accessing MySQL 4 DBs. (MySQL 5 is pretty much exactly 6 years old now.) You'll always find people not upgrading technology for various reasons. We continue to use MySQL 4 at work for an internal legacy application because it just works and we _know_ upgrading will be a lot of work. And see how IE6 is still in use. I'm not advocating that D should strive to support legacy technologies though. I don't think Win98 or earlier are supported either. We can start with supporting the latest version and if there is a demand for earlier versions those can be provided by third party or similar if they can't be included in Phobos. -- /Jacob Carlborg
Re: [std.database]
On Fri, 14 Oct 2011 14:23:32 +, Steve Teale wrote: > OK, for what it's worth, the compiler generated documentation (well, > more or less) for my mysqlD effort is now on my web site. > > http://britseyeview.com/software/mysqld/ Updated this so it now also has database and table listings, column metadata, and details of stored functions and procedures. Added Command methods specialized on stored functions and procedures. Only one comment so far, so it's either absolute crap or perfect. Steve
Re: [std.database]
On 17.10.2011 20:24, Jonathan M Davis wrote: There is code in druntime and Phobos which special-cases for Windows 98 and earlier (e.g. std.file using the A functions instead of the W functions if the version of Windows that you're running on is too old to have the W functions). Now, personally, I would love it if we just said that you have to have at least Win2k if not WinXP. It would be_great_ to be able to assume at least Vista, since it added a number of useful functions, but XP is still too prevalent for that to be reasonable. Regardless, supporting older versions of Windows is just plain irritating, since it restricts what you can do, and D is new enough and Win2k old enough that I find it perfectly reasonable to insist that you have WinXP or newer, but that's not what we're doing at this point. I'm not even sure W2K support is in great demand. Even XP is on a strong downwards slope, and Win7 has a greater share of the market now. XP is still needed for several years, but Win98...? When was the last time anyone encountered Win98? Does anyone even make software with Win98 support anymore? I doubt D looses any market share by ditching support for operating systems that doesn't even have vendor support. If it restricts usage on newer operating systems I'm in favor of dropping it, but I guess there's a good reason why Win98 is supported (or just legacy from 10 years back..?)
Re: [std.database]
On Monday, October 17, 2011 11:37 simendsjo wrote: > On 17.10.2011 20:24, Jonathan M Davis wrote: > > There is code in druntime and Phobos which special-cases for Windows 98 > > and earlier (e.g. std.file using the A functions instead of the W > > functions if the version of Windows that you're running on is too old to > > have the W functions). Now, personally, I would love it if we just said > > that you have to have at least Win2k if not WinXP. It would be_great_ > > to be able to assume at least Vista, since it added a number of useful > > functions, but XP is still too prevalent for that to be reasonable. > > Regardless, supporting older versions of Windows is just plain > > irritating, since it restricts what you can do, and D is new enough and > > Win2k old enough that I find it perfectly reasonable to insist that you > > have WinXP or newer, but that's not what we're doing at this point. > > I'm not even sure W2K support is in great demand. Even XP is on a strong > downwards slope, and Win7 has a greater share of the market now. XP is > still needed for several years, but Win98...? When was the last time > anyone encountered Win98? Does anyone even make software with Win98 > support anymore? > I doubt D looses any market share by ditching support for operating > systems that doesn't even have vendor support. If it restricts usage on > newer operating systems I'm in favor of dropping it, but I guess there's > a good reason why Win98 is supported (or just legacy from 10 years back..?) There have been bug reports on it (so someone is using D with Windows 98), and Walter seems to think that D should run on as much as it possibly can. So, it looks like until we have a solid reason why we _can't_ reasonably support Windows 98, we're going to. - Jonathan M Davis
Re: [std.database]
It looks as if it is not a big deal to use the MySQL protocol rather than the client library. I can now log in that way, so the rest should follow, and I am working on the changeover. The current MySQL protocol has been around since version 4.1. Given that Piotr has, I think, already done the same work at protocol level for Postgres, that SQLite is public domain, and that a similar API can be done with ODBC, we should be able to cover a fair range of systems without falling foul of GPL. Steve
Re: [std.database]
Steve Teale wrote: It looks as if it is not a big deal to use the MySQL protocol rather than the client library. I can now log in that way, so the rest should follow, and I am working on the changeover. The current MySQL protocol has been around since version 4.1. Good to hear that! One note though. MySQL protocol has two row encoding modes, text or binary. Please consider using the latter for better peformance. Given that Piotr has, I think, already done the same work at protocol level for Postgres, that SQLite is public domain, and that a similar API can be done with ODBC, we should be able to cover a fair range of systems without falling foul of GPL. Yes, that would be great.
Re: [std.database]
> > Good to hear that! One note though. MySQL protocol has two row encoding > modes, text or binary. Please consider using the latter for better > performance. > > Certainly - it would be pretty inefficient given the implementation intended to use text. Steve
Re: [std.database]
Steve Teale Wrote: > Given that Piotr has, I think, already done the same work at protocol > level for Postgres, that SQLite is public domain, and that a similar API > can be done with ODBC, we should be able to cover a fair range of systems > without falling foul of GPL. It is said ODBC is compatible with SQL/CLI https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?catalogno=c451 and applications written for SQL/CLI will work when compiled with ODBC headers. The description says the spec gives C binding, so should be viable to write D translation. I took a look at iODBC and MS ODBC headers they look similar, probably copied from SQL/CLI specification.
Re: [std.database]
> Steve Teale wrote: >> It looks as if it is not a big deal to use the MySQL protocol rather >> than the client library. I can now log in that way, so the rest should >> follow, and I am working on the changeover. The current MySQL protocol >> has been around since version 4.1. > Unfortunately I am now reasonably sure that such documentation as there is about the protocol tells lies. So getting it right is going to be like trench warfare. I was hoping to do it without resorting to dissection of the MySQL source code, but the latter may be required.
Re: [std.database]
Steve Teale wrote: Steve Teale wrote: It looks as if it is not a big deal to use the MySQL protocol rather than the client library. I can now log in that way, so the rest should follow, and I am working on the changeover. The current MySQL protocol has been around since version 4.1. Unfortunately I am now reasonably sure that such documentation as there is about the protocol tells lies. So getting it right is going to be like trench warfare. I was hoping to do it without resorting to dissection of the MySQL source code, but the latter may be required. I guess you're right about that requirement. I also did read some Postgres source code to see how data types are encoded. Don't give up! :-)
Re: [std.database]
On Thu, 20 Oct 2011 13:41:05 -0400, Piotr Szturmaj wrote: Steve Teale wrote: Steve Teale wrote: It looks as if it is not a big deal to use the MySQL protocol rather than the client library. I can now log in that way, so the rest should follow, and I am working on the changeover. The current MySQL protocol has been around since version 4.1. Unfortunately I am now reasonably sure that such documentation as there is about the protocol tells lies. So getting it right is going to be like trench warfare. I was hoping to do it without resorting to dissection of the MySQL source code, but the latter may be required. I guess you're right about that requirement. I also did read some Postgres source code to see how data types are encoded. Don't give up! :-) Please be cautious about reading GPL'd source code to understand the protocol. It's possible to be in violation of the license based on this. It generally takes two people to do this correctly, one to read and understand the original code, and one to implement the new version based on information conveyed by the first person. I'd recommend someone taking the existing protocol document and updating it with the corrections, then you using that document to fix the protocol implementation in your code. -Steve
Re: [std.database]
> > Please be cautious about reading GPL'd source code to understand the > protocol. It's possible to be in violation of the license based on > this. > > It generally takes two people to do this correctly, one to read and > understand the original code, and one to implement the new version based > on information conveyed by the first person. > > I'd recommend someone taking the existing protocol document and updating > it with the corrections, then you using that document to fix the > protocol implementation in your code. Will do. I shall attempt to do it on the basis of the network packets I get from the server in the first instance. Steve > -Steve
Re: [std.database]
Kagamin Wrote: > Steve Teale Wrote: > > > Given that Piotr has, I think, already done the same work at protocol > > level for Postgres, that SQLite is public domain, and that a similar API > > can be done with ODBC, we should be able to cover a fair range of systems > > without falling foul of GPL. > > It is said ODBC is compatible with SQL/CLI > https://www2.opengroup.org/ogsys/jsp/publications/PublicationDetails.jsp?catalogno=c451 > and applications written for SQL/CLI will work when compiled with ODBC > headers. > The description says the spec gives C binding, so should be viable to write D > translation. I took a look at iODBC and MS ODBC headers they look similar, > probably copied from SQL/CLI specification. yes, ISO 9075-3 provides whole sqlcli.h C header which looks like a copy to/from ms odbc sql.h header.
Re: [std.database]
Steven Schveighoffer Wrote: > Please be cautious about reading GPL'd source code to understand the > protocol. It's possible to be in violation of the license based on this. As long as he doesn't copy the code, there's no violation. He can even organize code better (or worse), e.g. use OOP, templates and/or ranges.
Re: [std.database]
On Thu, 20 Oct 2011 15:56:24 -0400, Kagamin wrote: Steven Schveighoffer Wrote: Please be cautious about reading GPL'd source code to understand the protocol. It's possible to be in violation of the license based on this. As long as he doesn't copy the code, there's no violation. He can even organize code better (or worse), e.g. use OOP, templates and/or ranges. Isn't it copying if I retype exactly what I'm reading? If so, wouldn't it be copying if I read code, then the code I write looks similar? It may feel like not copying, and it may not be copying, but it's sometimes (and I stress sometimes) difficult to prove that it's not copying. The easiest way to prove is not to read the code. Then any coincidental similarities are provable coincidences. It shouldn't be too difficult. The protocol spec should be nearly accurate (it technically should be 100% accurate, seems like mysql doesn't keep it up well enough), so you just have to fix any discrepancies. It sounds like Steve is already on the right track. If it comes down to it, someone can volunteer to help debug the code by comparing it to the GPL'd library in areas where the spec seems to be incorrect and completing the spec. I can help with this if you really need it, I'd love to see native D support for MySQL, as it's my DB of choice ;) -Steve
Re: [std.database]
> If it comes down to it, someone can volunteer to help debug the code by > comparing it to the GPL'd library in areas where the spec seems to be > incorrect and completing the spec. I can help with this if you really > need it, I'd love to see native D support for MySQL, as it's my DB of > choice ;) Steve, It's coming along quite well. I can log in and do simple queries, and clean up after them (how MySQL does that is horrifying), and now I'm attacking prepared statements - so far so good. I'd say in another 3-4 days I'll be back to where I was with the library dependent version. I agree with you 100% that getting the library out of the way is a top priority. What's more, now I'm getting my head round the protocol I'm thinking the binding for prepared statements can be simplified from what I had before. Kagamin, I'll get that header file. I already have a good chunk of the ODBC stuff done from a Windows header file a couple of years ago, so I should be able to get a quick start. Steve
Re: [std.database]
Just a quick progress report. Since it was clear that my original ideas for a MySQL D interface were not going to make it into Phobos at least because of license issues, I have been investigating the use of the published MySQL client/server protocol (this was expressly removed from GPL, if it could ever have belonged there in the first place, some time ago.) The results are encouraging, and I am pretty well back to the point I had reached using the C API, but now without files derived from GPL material, linking to a library, and mimicry of the C API methods. In the course of going over the whole thing again, and with a knowledge now of how the MySQL client/server relationship works, I think I may have improved the interface, even in its present experimental form. Steve
Re: [std.database]
As in the initial discussions on database interfaces, I am still of the view that such support should be provided at three levels. I also suggest that we adopt a proposal that was hinted at in the initial discussions, and think in the longer term of having components that are devoted to SQL, and those that will apply to a broader view of what comprises a database. In other words we should have std.sql and std.database. (I use std purely for illustration.) It's possibly worth mentioning that Microsoft had similar intentions years ago when they introduced OLE DB - a generalized interface to data sources. OLE DB is now being dumped. I should make it clear that what I'm working on at present are modules for xxx.sql. It has been pointed out in several postings that I don't seem to have any clear idea of the design requirements for std.database, and I admit that this is the case, though I don't think I'm alone in this. The initial discussions did not reach any well defined conclusions. If there is anyone who wants to leap into the std.database role, please volunteer. It could take me some time to get the SQL stuff in some sort of order. ODBC is a set of dialects, and the wire protocol definitions are not easy to get up-to-speed on. It seems that given licensing considerations, D support for SQL will probably need to be distributed. Some C header translations will have to go in Deimos - e.g. MySQL, some would probably be OK in Phobos, e.g. SQLite, and _maybe_ ODBC, (Would you believe that some driver and driver manager implementations have attached GPL wording to the ODBC header files!) and PostgreSQL. Having done quite a bit of work on implementations for MySQL (C API and wire protocol), and SQL Server (ODBC from Linux and Windows), I'm beginning to have some fairly clear ideas about what D components for dealing with SQL databases, using SQL directly, could look like. More detail on that separately. An immediate point that arises is that because some of the D 'header files' are derivative, and the associated D components need to link with external libraries, we're missing a place to put such components. They can't go in Phobos because of the licensing and linking considerations, and they can't go in Deimos because that is for translations of C header files only. At the same time, It seems to me that they should follow the same quite tight implementation pattern as those that can potentially go in Phobos. Someone going from say the C API version of the MySQL interface should be able to use the same, or closely similar code as when using the wire-protocol version. So one question is where should such implementations go? Another questions relates to the definition of interfaces at module level. We have interfaces that go hand-in-hand with classes built in to the language. But if I wanted to say that two sql interface modules had the same 'interface', how could this be formalized. Could it be done using contracts? I'd welcome suggestions on this. Steve
Re: [std.database]
Steve Teale Wrote: > So one question is where should such implementations go? github? > Another questions relates to the definition of interfaces at module > level. We have interfaces that go hand-in-hand with classes built in > to the language. But if I wanted to say that two sql interface modules > had the same 'interface', how could this be formalized. Could it be > done using contracts? I'd welcome suggestions on this. It can be done using concepts: a template which instantiates to a set of static asserts about what you want.
Re: [std.database]
On Fri, 25 Nov 2011 12:53:36 -0500, Kagamin wrote: > Steve Teale Wrote: > >> So one question is where should such implementations go? > > github? Well, probably yes, but that sounds a bit like "if you build it they will come", which doesn't always work. > >> Another questions relates to the definition of interfaces at module >> level. We have interfaces that go hand-in-hand with classes built in to >> the language. But if I wanted to say that two sql interface modules had >> the same 'interface', how could this be formalized. Could it be done >> using contracts? I'd welcome suggestions on this. > > It can be done using concepts: a template which instantiates to a set of > static asserts about what you want. Is that what we do with Ranges?
Re: [std.database]
Steve Teale Wrote: > > It can be done using concepts: a template which instantiates to a set of > > static asserts about what you want. > > Is that what we do with Ranges? Range concepts are boolean. There was a discussion on how to get detailed diagnostic if concept is not met, so that one would know what exactly is wrong. This is done with a sequence of static asserts.
Re: [std.database]
On 10/9/2011 2:50 AM, Jacob Carlborg wrote: On 2011-10-08 19:00, Andrei Alexandrescu wrote: 1. If we build a D wrapper for ODBC, then we allow people to write code for any database that has an ODBC driver. This, assuming we commit to ODBC as D's standard database interface, would complete the project. 2. If we want to go the route of "one std.database API with drivers for each DBMS" and consider ODBC one of several DBMSs, then we need to define our own driver architecture, write a few drivers ourselves (including probably ODBC), and hope that people will add more drivers. That's a larger project but it unties us from ODBC. 3. If we want to go the route of "similar but not identical specialized APIs per database system" and consider ODBC again only one of the database systems, then we need to define one specialized API per DBMS and force users to essentially choose upfront what DBMS they'll use, and code for it. It's an even larger project and I don't see obvious advantages for it than less of a need for upfront design. Andrei I would say that we declare a high level interface for database drivers. std.database can use this interface to connect to all databases there are drivers for. We then provide driver implementations for this interface for the databases we choose to support. It should also be possible for a user to create his/her own driver implementation for a database we haven't yet implemented or choose not to implement. Driver implementations could be: * MySQL/MariaDB * PostgreSQL * SQLite * ODBC * Oracle * SQL Server The smart thing would probably be to implement ODBC as the first database driver since it would allow most of the common databases to be used. But I don't want ODBC to be the only driver. I have some bad experience with ODBC from work. It can't handle multiple result sets and it's not very good at handle invalid Unicode characters. This might not be true for ODBC in general but it's a problem we have with the implementation we currently use. Also, ODBC adds an extra layer between the application and the database. One thing I notice is everyone seems to only be Targeting Relational Databases. Any plans to support Flat, Object, Key Value, Hierarchical, or Network Model systems? It would be nice to have at least specification support for systems like membase(memcache), hbase, vertica, csv files, and similar systems.
Re: [std.database]
On Friday, December 02, 2011 16:02:59 Hans Uhlig wrote: > On 10/9/2011 2:50 AM, Jacob Carlborg wrote: > > On 2011-10-08 19:00, Andrei Alexandrescu wrote: > >> 1. If we build a D wrapper for ODBC, then we allow people to write > >> code > >> for any database that has an ODBC driver. This, assuming we commit to > >> ODBC as D's standard database interface, would complete the project. > >> > >> 2. If we want to go the route of "one std.database API with drivers > >> for > >> each DBMS" and consider ODBC one of several DBMSs, then we need to > >> define our own driver architecture, write a few drivers ourselves > >> (including probably ODBC), and hope that people will add more drivers. > >> That's a larger project but it unties us from ODBC. > >> > >> 3. If we want to go the route of "similar but not identical > >> specialized > >> APIs per database system" and consider ODBC again only one of the > >> database systems, then we need to define one specialized API per DBMS > >> and force users to essentially choose upfront what DBMS they'll use, > >> and > >> code for it. It's an even larger project and I don't see obvious > >> advantages for it than less of a need for upfront design. > >> > >> > >> Andrei > > > > I would say that we declare a high level interface for database drivers. > > std.database can use this interface to connect to all databases there > > are drivers for. > > > > We then provide driver implementations for this interface for the > > databases we choose to support. It should also be possible for a user to > > create his/her own driver implementation for a database we haven't yet > > implemented or choose not to implement. > > > > Driver implementations could be: > > > > * MySQL/MariaDB > > * PostgreSQL > > * SQLite > > * ODBC > > * Oracle > > * SQL Server > > > > The smart thing would probably be to implement ODBC as the first > > database driver since it would allow most of the common databases to be > > used. > > > > But I don't want ODBC to be the only driver. I have some bad experience > > with ODBC from work. It can't handle multiple result sets and it's not > > very good at handle invalid Unicode characters. This might not be true > > for ODBC in general but it's a problem we have with the implementation > > we currently use. Also, ODBC adds an extra layer between the application > > and the database. > > One thing I notice is everyone seems to only be Targeting Relational > Databases. Any plans to support Flat, Object, Key Value, Hierarchical, > or Network Model systems? It would be nice to have at least > specification support for systems like membase(memcache), hbase, > vertica, csv files, and similar systems. Well, I wouldn't expect them to use the same API, but it's stuff like that why it's been suggested suggested that we make it std.sql instead of std.database. - Jonathan M Davis
Re: [std.database]
Le 03/12/2011 01:02, Hans Uhlig a écrit : > > One thing I notice is everyone seems to only be Targeting Relational > Databases. Any plans to support Flat, Object, Key Value, Hierarchical, > or Network Model systems? It would be nice to have at least > specification support for systems like membase(memcache), hbase, > vertica, csv files, and similar systems. > I don't think it's a good idea to put everything in the same bag. It doesn't help at all and in the end, and it usually makes things more complicated because there really is nothing in common between a flat file and a relational database, or even a key-value store. I think it's better to leave specific products like memcache or HBase to their own, specific APIs, at least until there is an established standard. It's not the role of a standard library to implement every product API out there, especially in the NoSQL domain, where there is no established standard at all.
Re: [std.database]
> One thing I notice is everyone seems to only be Targeting Relational > Databases. Any plans to support Flat, Object, Key Value, Hierarchical, > or Network Model systems? It would be nice to have at least > specification support for systems like membase(memcache), hbase, > vertica, csv files, and similar systems. That is the reason why I originally suggested std.database is perhaps a wrong place to cover all use-cases. Second, there will be native (pure D) interfaces to various data sources, and also there will be bindings to C libraries with the same purpose. Third, handling of relational databases will not be the same as with non- relational. Fourth, SQL - there are many people who like mister C.J.Date think SQL is not good for working with relational databases. So I would strongly recommend separation of SQL from the rest because there are other query languages around, including C.J.Date's "Tutorial D". IMHO, the center building block should be the DataSource. DataSource (or DateSet perhaps is a better name) should be ANYTHING that can be represented as collection of tuples. Examples: 1) directory list - it is a data source representing the content of a file- system directory. 2) result of an SQL query 3) *ANY tuple* is a DataSet with a single tuple as an element. 4) *ANY collection* can also be seen as DataSet (if the key is part of the tuple) With this kind of abstraction std.data (that is my preferred name for this package) would probably be the most powerful data abstraction layer I have seen so far. With std.loader we can even have pluggable drivers for various types of DataSources ... Regards
Re: [std.database]
Am 03.12.2011, 13:07 Uhr, schrieb Dejan Lekic : One thing I notice is everyone seems to only be Targeting Relational Databases. Any plans to support Flat, Object, Key Value, Hierarchical, or Network Model systems? It would be nice to have at least specification support for systems like membase(memcache), hbase, vertica, csv files, and similar systems. That is the reason why I originally suggested std.database is perhaps a wrong place to cover all use-cases. Second, there will be native (pure D) interfaces to various data sources, and also there will be bindings to C libraries with the same purpose. Third, handling of relational databases will not be the same as with non- relational. Fourth, SQL - there are many people who like mister C.J.Date think SQL is not good for working with relational databases. So I would strongly recommend separation of SQL from the rest because there are other query languages around, including C.J.Date's "Tutorial D". IMHO, the center building block should be the DataSource. DataSource (or DateSet perhaps is a better name) should be ANYTHING that can be represented as collection of tuples. Examples: 1) directory list - it is a data source representing the content of a file- system directory. 2) result of an SQL query 3) *ANY tuple* is a DataSet with a single tuple as an element. 4) *ANY collection* can also be seen as DataSet (if the key is part of the tuple) With this kind of abstraction std.data (that is my preferred name for this package) would probably be the most powerful data abstraction layer I have seen so far. With std.loader we can even have pluggable drivers for various types of DataSources ... Regards Sounds interesting until you talk about collections. This puts more interfaces on every collection (Range + DataSet), is that good? On the other hand I remember how nice the latest GUI toolkits work with data sources for table views, edit boxes, labels and so on. So I also see a long term benefit of this in terms of software architecture. Maybe you should do a sample implementation we can evaluate. Some CSV, two different container types, the directory listing and how it works with the data set. Also what effect would this have on how foreach iterates different collection types?
Re: [std.database]
I'm willing to try and contribute as best I can. Steve Teale wrote: > I use this title at Andrei's suggestion, and repeat his idea that it be > used as a prefix for discussions as we navigate toward a design. Unless > there is resistance to the idea, I will on the job of implementing > whatever we decide is appropriate. I am retired, and have the time to do > it. I am fine with that as long as people can still dip in if they see bugs or potential improvements. > > It seems that every man, and possibly his dog, has a private > implementation for at least a favorite DB, so we should have plenty of > material to build on. > > At this point I would like to get responses from those who feel they are > likely to contribute to the design through to completion. > > I'd also like to get a feel for the magnitude of the task, so I'd like to > ask what database systems you think should be supported. mysql, postgresql, sqllite are the 3 I am aiming at in my personal implementation. > > I have started a github account, and will put my mysqld stuff there > shortly, then you can kick me out if you don't like what you see. > > Steve
Re: [std.database]
Andrei, I had a go at odbcd at about the time I first started on mysqld. I must dig it out and get it up to the same state so that I understand ODBC again But basically from what you're saying, all we need is the ODBC header files translated into D. There appear to be driver managers and plenty of ODBC drivers for Linux. But that does not get us to where I was thinking of. ODBC is not much easier to use than the native C apis for the databases. I had thought that ODBC was just one of the C database apis that we would have to cover. Steve
Re: [std.database]
On 10/8/11 11:49 AM, Steve Teale wrote: Andrei, I had a go at odbcd at about the time I first started on mysqld. I must dig it out and get it up to the same state so that I understand ODBC again But basically from what you're saying, all we need is the ODBC header files translated into D. There's also the matter of dynamically linking with drivers I think. There appear to be driver managers and plenty of ODBC drivers for Linux. I'm not seeing all that bonanza. First hit for ==ODBC linux== yields http://www.unixodbc.org/ and last update is from April 2010. Not sure how good it is or anything. But that does not get us to where I was thinking of. ODBC is not much easier to use than the native C apis for the databases. I had thought that ODBC was just one of the C database apis that we would have to cover. 1. If we build a D wrapper for ODBC, then we allow people to write code for any database that has an ODBC driver. This, assuming we commit to ODBC as D's standard database interface, would complete the project. 2. If we want to go the route of "one std.database API with drivers for each DBMS" and consider ODBC one of several DBMSs, then we need to define our own driver architecture, write a few drivers ourselves (including probably ODBC), and hope that people will add more drivers. That's a larger project but it unties us from ODBC. 3. If we want to go the route of "similar but not identical specialized APIs per database system" and consider ODBC again only one of the database systems, then we need to define one specialized API per DBMS and force users to essentially choose upfront what DBMS they'll use, and code for it. It's an even larger project and I don't see obvious advantages for it than less of a need for upfront design. Andrei
Re: [std.database]
Andrei, It's not an easy one is it. I think it has to be #2. If we just did #1 we would probably alienate the Linux community. #3 is a recipe for maintenance problems. It would be nice if the DB specific drivers could be mixed and matched with whatever we come up with as the standard D interface. I'm more or less finished with populating individual variables and structs and arrays of structs with my mysqld implementation. I'm going to try to merge variants in as seamlessly as possible tomorrow. Steve
Re: [std.database]
On Saturday, October 08, 2011 06:43:29 Steve Teale wrote: > I'd also like to get a feel for the magnitude of the task, so I'd like to > ask what database systems you think should be supported. sqlite, postgres, and mysql are the ones that come to mind for me, though outside of a corporate environment, I can't ever see myself using anything other than sqlite, since it uses a file instead of a server, which makes way more sense when having a database for an individual program. - Jonathan M Davis
Re: [std.database]
On 10/8/11 8:36 AM, Adam Burton wrote: I'd also like to get a feel for the magnitude of the task, so I'd like to ask what database systems you think should be supported. mysql, postgresql, sqllite are the 3 I am aiming at in my personal implementation. I had lunch yesterday with a database expert and discussed the matter with him. He advised that we take a driver-oriented approach in which we define a common API for all databases (modeled at high level after e.g. JDBC to reuse that accumulated experience), and then define a few drivers for popular DBMSs in hope that users interested in supporting additional DBMSs will contribute additional drivers using the initial drivers as a template. Eventually when D and its database API become successful, DBMS providers themselves will write the drivers. It would be great if we could leverage another already widespread driver architecture. There are two that come to mind, and I looked into both: JDBC and ODBC. The first requires either bridging into Java or translating Java driver code to D. The second is widespread on Windows but less so on other OSs. So probably we'll need to define our own drivers. (This also brings dynamic linking as a possibility.) Andrei
Re: [std.database]
On Saturday, October 08, 2011 12:00:37 Andrei Alexandrescu wrote: > 1. If we build a D wrapper for ODBC, then we allow people to write code > for any database that has an ODBC driver. This, assuming we commit to > ODBC as D's standard database interface, would complete the project. > > 2. If we want to go the route of "one std.database API with drivers for > each DBMS" and consider ODBC one of several DBMSs, then we need to > define our own driver architecture, write a few drivers ourselves > (including probably ODBC), and hope that people will add more drivers. > That's a larger project but it unties us from ODBC. > > 3. If we want to go the route of "similar but not identical specialized > APIs per database system" and consider ODBC again only one of the > database systems, then we need to define one specialized API per DBMS > and force users to essentially choose upfront what DBMS they'll use, and > code for it. It's an even larger project and I don't see obvious > advantages for it than less of a need for upfront design. I definitely vote for #2 or #3. One of our projects at work uses it (though not one that I personally work on), and I've never heard good things about it. Supporting it makes good sense, but I wouldn't want us to design Phobos' database solution around it. We should probably explore #2 first, and then if it doesn't work well enough, then at least we have a solid base for the API's being similar with #3. However, given what I've heard about ODBC and its attempts to unify databases, I'm skeptical of how well we'll be able to have a unified DBMS API without harming performance. And from what I understand, it's pretty rare to change DBMSes for a project. You just pick one and use it. And then in the rare case where you have to change, you do the work to do it (and as long as the DB is appropriately modularized with regards to the rest of the program, it doesn't have a hugely negative affect on the rest of the program). So, I question that #2 gains us a whole lot over #3 ultimately (_especially_ if it ends up costing much of anything in terms of performance or usability), but I'm not a DB expert, and I do think that it's at least worth exploring #2 first - if nothing else because it could lead to the APIs for #3 being better unified without harming their performance or usability. - Jonathan M Davis
Re: [std.database]
Microsoft SQL Server is important to cover too. I'm pretty sure ODBC works fine for that (there's ODBC bindings for D already, it's part of the Windows headers) and I wrote a little something for my database.d, but I haven't actually tested it yet! (The project I work on for SQL server has a lot of legacy VB code, so while I want to get D in on it, haven't really gotten into it yet.)
Re: [std.database]
The database API I wrote ages ago is built on ODBC and you're welcome to a copy if it would help. At the time (admittedly 15 years ago) the docs for ODBC were incomplete and wrong in places, so a reference can be handy. Sent from my iPhone On Oct 8, 2011, at 9:49 AM, Steve Teale wrote: > Andrei, > > I had a go at odbcd at about the time I first started on mysqld. I must dig > it out and > get it up to the same state so that I understand ODBC again > > But basically from what you're saying, all we need is the ODBC header files > translated into D. There appear to be driver managers and plenty of ODBC > drivers > for Linux. > > But that does not get us to where I was thinking of. ODBC is not much easier > to > use than the native C apis for the databases. I had thought that ODBC was > just one > of the C database apis that we would have to cover. > > Steve
Re: [std.database]
Jonathan M Davis wrote: > On Saturday, October 08, 2011 12:00:37 Andrei Alexandrescu wrote: >> 1. If we build a D wrapper for ODBC, then we allow people to write code >> for any database that has an ODBC driver. This, assuming we commit to >> ODBC as D's standard database interface, would complete the project. >> >> 2. If we want to go the route of "one std.database API with drivers for >> each DBMS" and consider ODBC one of several DBMSs, then we need to >> define our own driver architecture, write a few drivers ourselves >> (including probably ODBC), and hope that people will add more drivers. >> That's a larger project but it unties us from ODBC. >> >> 3. If we want to go the route of "similar but not identical specialized >> APIs per database system" and consider ODBC again only one of the >> database systems, then we need to define one specialized API per DBMS >> and force users to essentially choose upfront what DBMS they'll use, and >> code for it. It's an even larger project and I don't see obvious >> advantages for it than less of a need for upfront design. > > I definitely vote for #2 or #3. One of our projects at work uses it > (though not one that I personally work on), and I've never heard good > things about it. Supporting it makes good sense, but I wouldn't want us to > design Phobos' database solution around it. > > We should probably explore #2 first, and then if it doesn't work well > enough, then at least we have a solid base for the API's being similar > with #3. However, given what I've heard about ODBC and its attempts to > unify databases, I'm skeptical of how well we'll be able to have a unified > DBMS API without harming performance. And from what I understand, it's > pretty rare to change DBMSes for a project. You just pick one and use it. > And then in the rare case where you have to change, you do the work to do > it (and as long as the DB is appropriately modularized with regards to the > rest of the program, it doesn't have a hugely negative affect on the rest > of the program). So, I question that > #2 gains us a whole lot over #3 ultimately (_especially_ if it ends up > #costing > much of anything in terms of performance or usability), but I'm not a DB > expert, and I do think that it's at least worth exploring #2 first - if > nothing else because it could lead to the APIs for #3 being better unified > without harming their performance or usability. > > - Jonathan M Davis I agree that once a project picks its database it rarely changes but I find most people tend to use common functionality on the application code side and it's the sql which tends to get optimised a lot (there are some traps in the application code that potentially reduce performance like impropper use of mysql prepared statements). I think with #2 we should be able to hit the main db requirements so people can atleast develop database connected applications. The drivers would give us the flexibility to handle more databases at a basic level (at least an odbc driver will let us achieve this quickly until someone wants to provide D driver implementation of a db if it provides advantage of some kind). That being said how do people tend to handle the SQL itself? Unless you are using a library to create the SQL for you, like hibernate, does anyone find you tend to run into SQL syntax variation across different databases? For example MySQL and MSSQL handle results pagination differently (http://forums.asp.net/p/1427310/3186427.aspx#3186427). I know there is ANSI SQL but my understanding is most databases don't implement it well (not all of it or poor performance etc). Seems to me even if you used a common API you are going to need to abstract the different types of databases some how just because of the SQL itself. Even if #2 is completed #3 could still be worked on, if someone is willing. A common API is likely to miss some features from the specific APIs (I've only looked into mysql so this is an assumption). The implmentations from #3 could then be used to implement drivers for #2 allowing those who want flexibility to use #2 but those with more specific requirements, and bothered about performance, can use #3. That is more work than #2 or #3 separately but like I said "if someone is willing". My plan for my personal implementation was to develop specific APIs (#3) for MySql and Postgresql, trying to keep them similar where possible, then look into providing a wrapper api for common/basic functionality (#2). That project was as much about getting to grips with D as a project I intend to use, which is why I chose the long way round to implement it.
Re: [std.database]
Steve Teale Wrote: > I use this title at Andrei's suggestion, and repeat his idea that it be used > as a prefix for discussions as we navigate toward a design. Unless there is > resistance to the idea, I will on the job of implementing whatever we decide > is appropriate. I am retired, and have the time to do it. > > It seems that every man, and possibly his dog, has a private implementation > for at least a favorite DB, so we should have plenty of material to build on. > > At this point I would like to get responses from those who feel they are > likely to contribute to the design through to completion. > > I'd also like to get a feel for the magnitude of the task, so I'd like to ask > what database systems you think should be supported. > > I have started a github account, and will put my mysqld stuff there shortly, > then you can kick me out if you don't like what you see. > > Steve Please refer to the successful experience of the industry, such as ejb, jdbc, jdo, jpa, ado.net, delphi db api, ruby on rails active recorder, orm's hibernate, absorb they the advantages of, to create a powerful and simple for both for enterprise applications database api framework, to achieve this goal will Requires more than cooperation, or you personal proficient in all of the above api, Hoping to start directly from the orm(such as hibernate does not depend on jdbc,or Low dependence ), Improve orm performance. Hope more database experts involved in the D database api design ! thanks all ! dolive
Re: [std.database]
On 2011-10-08 19:00, Andrei Alexandrescu wrote: 1. If we build a D wrapper for ODBC, then we allow people to write code for any database that has an ODBC driver. This, assuming we commit to ODBC as D's standard database interface, would complete the project. 2. If we want to go the route of "one std.database API with drivers for each DBMS" and consider ODBC one of several DBMSs, then we need to define our own driver architecture, write a few drivers ourselves (including probably ODBC), and hope that people will add more drivers. That's a larger project but it unties us from ODBC. 3. If we want to go the route of "similar but not identical specialized APIs per database system" and consider ODBC again only one of the database systems, then we need to define one specialized API per DBMS and force users to essentially choose upfront what DBMS they'll use, and code for it. It's an even larger project and I don't see obvious advantages for it than less of a need for upfront design. Andrei I would say that we declare a high level interface for database drivers. std.database can use this interface to connect to all databases there are drivers for. We then provide driver implementations for this interface for the databases we choose to support. It should also be possible for a user to create his/her own driver implementation for a database we haven't yet implemented or choose not to implement. Driver implementations could be: * MySQL/MariaDB * PostgreSQL * SQLite * ODBC * Oracle * SQL Server The smart thing would probably be to implement ODBC as the first database driver since it would allow most of the common databases to be used. But I don't want ODBC to be the only driver. I have some bad experience with ODBC from work. It can't handle multiple result sets and it's not very good at handle invalid Unicode characters. This might not be true for ODBC in general but it's a problem we have with the implementation we currently use. Also, ODBC adds an extra layer between the application and the database. -- /Jacob Carlborg
Re: [std.database]
On 2011-10-08 23:11, Adam D. Ruppe wrote: Microsoft SQL Server is important to cover too. I'm pretty sure ODBC works fine for that (there's ODBC bindings for D already, it's part of the Windows headers) and I wrote a little something for my database.d, but I haven't actually tested it yet! I have some bad experience with ODBC, connecting to SQL Server, from work. It can't handle multiple result sets and it's not very good at handle invalid Unicode characters. This might not be true for ODBC in general but it's a problem we have with the implementation we currently use. If we don't want to completely role our own implementation I suggest we use freetds directly instead (ODBC uses freetds). I think it would be good to have an ODBC implementation but it should not be the only way to connect to SQL Server. -- /Jacob Carlborg
Re: [std.database]
On 2011-10-08 23:12, Jonathan M Davis wrote: On Saturday, October 08, 2011 12:00:37 Andrei Alexandrescu wrote: 1. If we build a D wrapper for ODBC, then we allow people to write code for any database that has an ODBC driver. This, assuming we commit to ODBC as D's standard database interface, would complete the project. 2. If we want to go the route of "one std.database API with drivers for each DBMS" and consider ODBC one of several DBMSs, then we need to define our own driver architecture, write a few drivers ourselves (including probably ODBC), and hope that people will add more drivers. That's a larger project but it unties us from ODBC. 3. If we want to go the route of "similar but not identical specialized APIs per database system" and consider ODBC again only one of the database systems, then we need to define one specialized API per DBMS and force users to essentially choose upfront what DBMS they'll use, and code for it. It's an even larger project and I don't see obvious advantages for it than less of a need for upfront design. I definitely vote for #2 or #3. One of our projects at work uses it (though not one that I personally work on), and I've never heard good things about it. Supporting it makes good sense, but I wouldn't want us to design Phobos' database solution around it. We have some trouble with ODBC at work too. We should probably explore #2 first, and then if it doesn't work well enough, then at least we have a solid base for the API's being similar with #3. However, given what I've heard about ODBC and its attempts to unify databases, I'm skeptical of how well we'll be able to have a unified DBMS API without harming performance. And from what I understand, it's pretty rare to change DBMSes for a project. You just pick one and use it. And then in the rare case where you have to change, you do the work to do it (and as long as the DB is appropriately modularized with regards to the rest of the program, it doesn't have a hugely negative affect on the rest of the program). So, I question that #2 gains us a whole lot over #3 ultimately (_especially_ if it ends up costing much of anything in terms of performance or usability), but I'm not a DB expert, and I do think that it's at least worth exploring #2 first - if nothing else because it could lead to the APIs for #3 being better unified without harming their performance or usability. - Jonathan M Davis I think it can be quite useful to change DBMSes for a project. For example, start with SQLite because it easy to set up and then move to MySQL or similar. It's especially easy to do with Ruby on Rails on Mac OS X. Rails uses SQLite as the default database and Mac OS X comes bundle with SQLite. There is no extra steps to set up this environment. -- /Jacob Carlborg
Re: [std.database]
Am 08.10.2011 23:11, schrieb Adam D. Ruppe: Microsoft SQL Server is important to cover too. Yeah, it's probably pretty widely used in the Windows world so it should be supported. And Oracle should probably be supported as well. But if we have a generic DB API support for these can be added later (based on ODBC, some product specific API or whatever). I think support for SQLite, MySQL and PostgreSQL would be a good start. Cheers, - Daniel
Re: [std.database]
Steve Teale wrote:
I use this title at Andrei's suggestion, and repeat his idea that it be used
as a prefix for discussions as we navigate toward a design. Unless there is
resistance to the idea, I will on the job of implementing whatever we decide
is appropriate. I am retired, and have the time to do it.
It seems that every man, and possibly his dog, has a private implementation
for at least a favorite DB, so we should have plenty of material to build on.
At this point I would like to get responses from those who feel they are
likely to contribute to the design through to completion.
I'd also like to get a feel for the magnitude of the task, so I'd like to ask
what database systems you think should be supported.
I have started a github account, and will put my mysqld stuff there shortly,
then you can kick me out if you don't like what you see.
1. I think that we should not design this API using the least common
denominator approach. This is to not limit some databases. For example
PostgreSQL has many great features not available in MySQL. That's why I
started with postgres in my ddb project. I think DB API should be
designed to support the most featureful databases and those that have
less features may be easily adapted to that API.
Some PostgreSQL vs MySQL differences to note:
- the former does support arrays and composite types that may be stored
in single column, the latter doesn't
- the former support asynchronous queries, but I guess its not that
important
- the former support async notifications (for example, fired by a
trigger, or when server is shutting down)
2. Compile type mapping of fields should support postgres's composites
and arrays. For example, this is obvious:
execQuery!(int, string)("select 5, 'abc'");
but this is not:
execQuery!(int, string)("select ROW(5, 'abc')"); // valid postgres query
Does it return two columns or one composite column? I already addressed
this ambiguities in ddb, please see documentation on github
(http://pszturmaj.github.com/ddb/db.html). See also 'advanced example'
on that page.
3. I think that compile type mapping of fields should be generalized to
not only support DB API. It could also be used with CSV files or other
tabular data. It may also be extended to support tree structures with
XML (mapping xml to structs/tuples/arrays).
Re: [std.database]
On 10/9/11 7:28 AM, Piotr Szturmaj wrote:
1. I think that we should not design this API using the least common
denominator approach. This is to not limit some databases. For example
PostgreSQL has many great features not available in MySQL. That's why I
started with postgres in my ddb project. I think DB API should be
designed to support the most featureful databases and those that have
less features may be easily adapted to that API.
Some PostgreSQL vs MySQL differences to note:
- the former does support arrays and composite types that may be stored
in single column, the latter doesn't
Yah, Hive also has some really interesting data types, such as
parameterized arrays and maps and JSON data. Our API should be generic
enough to accommodate such types seamlessly.
- the former support asynchronous queries, but I guess its not that
important
- the former support async notifications (for example, fired by a
trigger, or when server is shutting down)
Not sure how we could support such. Ideas would be welcome.
2. Compile type mapping of fields should support postgres's composites
and arrays. For example, this is obvious:
execQuery!(int, string)("select 5, 'abc'");
but this is not:
execQuery!(int, string)("select ROW(5, 'abc')"); // valid postgres query
I think that should be
execQuery!(Tuple!(int, string))("select ROW(5, 'abc')");
Does it return two columns or one composite column? I already addressed
this ambiguities in ddb, please see documentation on github
(http://pszturmaj.github.com/ddb/db.html). See also 'advanced example'
on that page.
3. I think that compile type mapping of fields should be generalized to
not only support DB API. It could also be used with CSV files or other
tabular data. It may also be extended to support tree structures with
XML (mapping xml to structs/tuples/arrays).
Interesting.
Andrei
Re: [std.database]
There was some discussion prior to this thread about the relative virtues
of binding to structs or binding to arrays of Variants. I was thinking
about this, and have experimented with Variants in my trial MySQL
implementation. My conclusions below - do they make sense?
Using Variant to capture the output of prepared queries or provide input,
as opposed to a struct, gets you out of the realm of what must be
determined at compile time, but only at the expense of extra DB server
round trip(s).
If you want to use them in a deterministic way with a database table of
known and stable structure you have to bind them. To bind them you must
be able to determine their type. But with Variants you can't do that
until they are initialized. So, with a struct you must have something
like:
struct MyTableQuery42
{
byte col1;
float col2;
char[] col3;
}
This can be bound for output without doing any explicit initialization of
an instance, since you can write a template function to bind it that
discovers everything you need to know about the struct at compile time.
If using variants for a set of out parameters you must have something
equivalent to:
Variant[3] va;
va[0] = cast(byte) 0;
va[1] = 0.0F;
va[2] = cast(char[]) [];
So you have to have exactly the same information at compile time for the
Variant array as you do for the struct - you still have to specify a set
of types.
The difference is that if you have prepared a statement, you can go to
the server and ask for the relevant metadata. With a struct you can use
this to check if the struct is a match for the query. With an array of
Variants you can make if conform to the query.
However, in a large number of cases, using the struct you won't need to
bother with the metadata, because you 'know' the types of the query
result. You don't have to bother with them for the Variant array either,
but in that case you have to provide a function like
Variant[] MyTableQuery42Init() { ... }
which 'knows' the same stuff.
It's probably true to say that the syntax/semantics of the interface will
suck slightly more in the Variant case than in the struct case.
Steve
Re: [std.database]
Further question. Should we assume in the first instance that we should only attempt to accommodate those DBs that are free or have some free version that may be limited in some way - e.g. the developer version of MS SQL Server. Presumably when D reaches the point of being all-conquering, then Oracle, IBM and so on will chip in. Steve
Re: [std.database]
Further generic question. (Yes, I am listening to the answers too) If some underlying databases don't support the features that our chosen interface requires, do we attempt to synthesize them - presumably at cost to performance, or do we just throw a compile-time exception that basically tells the user to use a lower interface and code it themself? It's important that we establish such boundaries. Otherwise we'll never ever have an alpha version of 0.1. To express a personal opinion, then as a first pass we should do something that is at about the same level as JDBC but without the concessions to DBs like Postgres that have fancy SQL types. When we have decided on an interface, we can always go forward, but going back is embarrassing. Steve
Re: [std.database]
The way I'd do it is:
interface Database {
// support shared functions here, and other stuff useful enough to
// warrant emulation
}
class Postgres : Database {
// implement the interface, of course, but also all other postgres
// specific stuff
}
When you go to use it, if you're happy with the basics, declare
Databases.
If you need something special, use Postgres objects.
Re: [std.database]
Adam Ruppe wrote:
The way I'd do it is:
interface Database {
// support shared functions here, and other stuff useful enough to
// warrant emulation
}
class Postgres : Database {
// implement the interface, of course, but also all other postgres
// specific stuff
}
When you go to use it, if you're happy with the basics, declare
Databases.
If you need something special, use Postgres objects.
+1
Re: [std.database]
Steve Teale wrote: To express a personal opinion, then as a first pass we should do something that is at about the same level as JDBC but without the concessions to DBs like Postgres that have fancy SQL types. I disagree. Doing it this way may introduce difficulties or incompabilities in the future. I think we should design it from the ground up, keeping in mind the all databases. We probably should write a page on a wiki describing the API, without actually implementing anything. Then anyone involved may contribute to its design, so it may evolve into somewhat more thought out API.
Re: [std.database]
> We probably should write a page on a wiki describing the API, without > actually implementing anything. Then anyone involved may contribute to > its design, so it may evolve into somewhat more thought out API. I like that idea! Must find out how to put up a wiki. Steve
Re: [std.database]
Steve Teale wrote: We probably should write a page on a wiki describing the API, without actually implementing anything. Then anyone involved may contribute to its design, so it may evolve into somewhat more thought out API. I like that idea! Must find out how to put up a wiki. Or use existing one: http://www.prowiki.org/wiki4d/wiki.cgi :-)
Re: [std.database]
On 10/9/11 11:54 AM, Adam Ruppe wrote:
The way I'd do it is:
interface Database {
// support shared functions here, and other stuff useful enough to
// warrant emulation
}
class Postgres : Database {
// implement the interface, of course, but also all other postgres
// specific stuff
}
When you go to use it, if you're happy with the basics, declare
Databases.
If you need something special, use Postgres objects.
Makes sense. JDBC does that, too.
Andrei
Re: [std.database]
On 10/9/11 11:15 AM, Steve Teale wrote: If using variants for a set of out parameters you must have something equivalent to: Variant[3] va; va[0] = cast(byte) 0; va[1] = 0.0F; va[2] = cast(char[]) []; So you have to have exactly the same information at compile time for the Variant array as you do for the struct - you still have to specify a set of types. Not at all, fortunately it's much simpler than that. Far as I can tell, a resultset emitted by a DBMS comes in the form of data (either untyped buffers or strings) plus column type information, usually in the form of an enumeration (e.g. 0 for NULL, 1 for int, 2 for double, 3 for string etc). You of course also have the total column count. So all you need to do is use that enum to pour the data into the Variants. See this for example: http://msdn.microsoft.com/en-us/library/ms131297.aspx You can glean all needed information from the resultset after having issued the query. It's probably true to say that the syntax/semantics of the interface will suck slightly more in the Variant case than in the struct case. That's a given. The suckiness won't come, however, in the form of additional trips to the database. Andrei
Re: [std.database]
On 10/9/11 11:40 AM, Steve Teale wrote: Further generic question. (Yes, I am listening to the answers too) If some underlying databases don't support the features that our chosen interface requires, do we attempt to synthesize them - presumably at cost to performance, or do we just throw a compile-time exception that basically tells the user to use a lower interface and code it themself? No. Andrei
Re: [std.database]
On 2011-10-09 18:54, Adam Ruppe wrote:
The way I'd do it is:
interface Database {
// support shared functions here, and other stuff useful enough to
// warrant emulation
}
class Postgres : Database {
// implement the interface, of course, but also all other postgres
// specific stuff
}
When you go to use it, if you're happy with the basics, declare
Databases.
If you need something special, use Postgres objects.
+1
--
/Jacob Carlborg
Re: [std.database]
On 10/09/11 13:22, Andrei Alexandrescu wrote: On 10/9/11 11:40 AM, Steve Teale wrote: Further generic question. (Yes, I am listening to the answers too) If some underlying databases don't support the features that our chosen interface requires, do we attempt to synthesize them - presumably at cost to performance, or do we just throw a compile-time exception that basically tells the user to use a lower interface and code it themself? No. Andrei Sorry, that was awfully unclear. I meant to say the driver shouldn't do little miracles in adapting support from one engine to the next. It's a losing race. It should be fine if certain queries or API calls fail either statically or dynamically. Andrei
Re: [std.database]
On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: 1. I think that we should not design this API using the least common denominator approach. This is to not limit some databases. For example PostgreSQL has many great features not available in MySQL. That's why I started with postgres in my ddb project. I think DB API should be designed to support the most featureful databases and those that have less features may be easily adapted to that API. Haven't common denominator designs been more or less failures in at least one category - gui libraries? Some driver models have succeeded only because a powerful entity forced the issue - like for device drivers for an OS. I suspect that trying to design a common api to popular databases is an expensive and quixotic quest. If it weren't, wouldn't it have happened already?
Re: [std.database]
On 10/7/2011 11:43 PM, Steve Teale wrote: I use this title at Andrei's suggestion, and repeat his idea that it be used as a prefix for discussions as we navigate toward a design. Unless there is resistance to the idea, I will on the job of implementing whatever we decide is appropriate. I am retired, and have the time to do it. Thanks, Steve, for being the champion for this project. I'd like to suggest that, as a first step, simple translations of the C header files for the various databases be added to etc.c. That'll at least enable people who need to use a database now to get started.
Re: [std.database]
On 10/9/11 5:31 PM, Walter Bright wrote: On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: 1. I think that we should not design this API using the least common denominator approach. This is to not limit some databases. For example PostgreSQL has many great features not available in MySQL. That's why I started with postgres in my ddb project. I think DB API should be designed to support the most featureful databases and those that have less features may be easily adapted to that API. Haven't common denominator designs been more or less failures in at least one category - gui libraries? A common database interface is not a common denominator API; more like the opposite. This is not difficult because most differences across database systems lie in their SQL, which is strings from D's perspective. Some driver models have succeeded only because a powerful entity forced the issue - like for device drivers for an OS. I suspect that trying to design a common api to popular databases is an expensive and quixotic quest. If it weren't, wouldn't it have happened already? It has. All database APIs for programming languages do exactly that. Andrei
Re: [std.database]
On Oct 9, 2011, at 3:56 PM, Andrei Alexandrescu wrote: > On 10/9/11 5:31 PM, Walter Bright wrote: >> On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: >>> 1. I think that we should not design this API using the least common >>> denominator >>> approach. This is to not limit some databases. For example PostgreSQL >>> has many >>> great features not available in MySQL. That's why I started with >>> postgres in my >>> ddb project. I think DB API should be designed to support the most >>> featureful >>> databases and those that have less features may be easily adapted to >>> that API. >> >> >> Haven't common denominator designs been more or less failures in at >> least one category - gui libraries? > > A common database interface is not a common denominator API; more like the > opposite. This is not difficult because most differences across database > systems lie in their SQL, which is strings from D's perspective. Assuming that by "database" you mean SQL. Pretty fair assumption, though NoSQL databases (which cover a broad range of designs since there's no standard language yet for key-value DBs, etc) are rapidly gaining popularity. I almost wonder if the base type should be named SqlDatabase instead of Database.
Re: [std.database]
On Sunday, October 09, 2011 16:31:35 Sean Kelly wrote: > On Oct 9, 2011, at 3:56 PM, Andrei Alexandrescu wrote: > > On 10/9/11 5:31 PM, Walter Bright wrote: > >> On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: > >>> 1. I think that we should not design this API using the least common > >>> denominator > >>> approach. This is to not limit some databases. For example > >>> PostgreSQL > >>> has many > >>> great features not available in MySQL. That's why I started with > >>> postgres in my > >>> ddb project. I think DB API should be designed to support the most > >>> featureful > >>> databases and those that have less features may be easily adapted to > >>> that API. > >> > >> Haven't common denominator designs been more or less failures in at > >> least one category - gui libraries? > > > > A common database interface is not a common denominator API; more like > > the opposite. This is not difficult because most differences across > > database systems lie in their SQL, which is strings from D's > > perspective. > Assuming that by "database" you mean SQL. Pretty fair assumption, though > NoSQL databases (which cover a broad range of designs since there's no > standard language yet for key-value DBs, etc) are rapidly gaining > popularity. I almost wonder if the base type should be named SqlDatabase > instead of Database. If we were to do that, then maybe it should just be sql, since it's shorter and just as clear: std.sql.*. - Jonathan M Davis
Re: [std.database]
On 10/9/11 6:31 PM, Sean Kelly wrote: On Oct 9, 2011, at 3:56 PM, Andrei Alexandrescu wrote: On 10/9/11 5:31 PM, Walter Bright wrote: On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: 1. I think that we should not design this API using the least common denominator approach. This is to not limit some databases. For example PostgreSQL has many great features not available in MySQL. That's why I started with postgres in my ddb project. I think DB API should be designed to support the most featureful databases and those that have less features may be easily adapted to that API. Haven't common denominator designs been more or less failures in at least one category - gui libraries? A common database interface is not a common denominator API; more like the opposite. This is not difficult because most differences across database systems lie in their SQL, which is strings from D's perspective. Assuming that by "database" you mean SQL. Pretty fair assumption, though NoSQL databases (which cover a broad range of designs since there's no standard language yet for key-value DBs, etc) are rapidly gaining popularity. I almost wonder if the base type should be named SqlDatabase instead of Database. Good point and good idea. Andrei
Re: [std.database]
> I'd like to suggest that, as a first step, simple translations of the C > header files for the various databases be added to etc.c. That'll at > least enable people who need to use a database now to get started. Walter, I'm sure they're already out there waiting. I have MySQL. Any offers for Postgres and SQLite? Does MS-SQL have a C interface these days I wonder? Steve
Re: [std.database]
On 2011-10-10 00:31, Walter Bright wrote: On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: 1. I think that we should not design this API using the least common denominator approach. This is to not limit some databases. For example PostgreSQL has many great features not available in MySQL. That's why I started with postgres in my ddb project. I think DB API should be designed to support the most featureful databases and those that have less features may be easily adapted to that API. Haven't common denominator designs been more or less failures in at least one category - gui libraries? Some driver models have succeeded only because a powerful entity forced the issue - like for device drivers for an OS. I suspect that trying to design a common api to popular databases is an expensive and quixotic quest. If it weren't, wouldn't it have happened already? At some level we need a common API for the databases, but still be able to get to database specific API's when there is need for that. -- /Jacob Carlborg
Re: [std.database]
On 2011-10-10 06:54, Steve Teale wrote: I'd like to suggest that, as a first step, simple translations of the C header files for the various databases be added to etc.c. That'll at least enable people who need to use a database now to get started. Walter, I'm sure they're already out there waiting. I have MySQL. Any offers for Postgres and SQLite? Does MS-SQL have a C interface these days I wonder? Steve I don't know but FreeTDS has implemented the TDS protocol (which SQL server uses). Either we use FreeTDS (LGPL license) or role our own implementation. -- /Jacob Carlborg
Re: [std.database]
On Sun, 09 Oct 2011 20:31:35 -0300, Sean Kelly wrote: On Oct 9, 2011, at 3:56 PM, Andrei Alexandrescu wrote: On 10/9/11 5:31 PM, Walter Bright wrote: On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: 1. I think that we should not design this API using the least common denominator approach. This is to not limit some databases. For example PostgreSQL has many great features not available in MySQL. That's why I started with postgres in my ddb project. I think DB API should be designed to support the most featureful databases and those that have less features may be easily adapted to that API. Haven't common denominator designs been more or less failures in at least one category - gui libraries? A common database interface is not a common denominator API; more like the opposite. This is not difficult because most differences across database systems lie in their SQL, which is strings from D's perspective. Assuming that by "database" you mean SQL. Pretty fair assumption, though NoSQL databases (which cover a broad range of designs since there's no standard language yet for key-value DBs, etc) are rapidly gaining popularity. I almost wonder if the base type should be named SqlDatabase instead of Database. There is a standard language defined for NoSQL, namely UnQL: http://.unqlspec.org/display/UnQL/Home Roald
Re: [std.database]
On Sat, 08 Oct 2011 17:19:02 +0100, Andrei Alexandrescu wrote: On 10/8/11 8:36 AM, Adam Burton wrote: I'd also like to get a feel for the magnitude of the task, so I'd like to ask what database systems you think should be supported. mysql, postgresql, sqllite are the 3 I am aiming at in my personal implementation. I had lunch yesterday with a database expert and discussed the matter with him. He advised that we take a driver-oriented approach in which we define a common API for all databases (modeled at high level after e.g. JDBC to reuse that accumulated experience) Just a small note, because it's on my mind currently, with regards to JDBC. This is slightly OT, but pertains to the comment about how we should model std.database. JDBC has Statement and PreparedStatement objects, the latter allows you to 'bind' parameters, which is great. But, it requires you actually 'prepare' them as well.. I am not 100% certain, but I believe similar C/C++ code can bind parameters without actually 'preparing' the statement for reuse. We use this in cases where we do not re-use the statement, and want to avoid the excess work of preparing it for re-use. In addition we bind parameters to avoid getting unique query strings, which (I believe) get cached by SQL server.. resulting in thousands of unique queries in the cache, slowing things down. So, my comment is simply to say, assuming I am not talking rubbish, make sure the design allows for binding/not binding parameters with prepared/not-prepared statements. This came up recently because some of our JDBC code was taking 12 minutes to do a select, due to using a PreparedStatement, and changing to a Statement reduced this to <10 sec. But, it means the select is now unique, and we will be introducing more cached queries.. anyone know how to avoid this using Statement? R -- Using Opera's revolutionary email client: http://www.opera.com/mail/
Re: [std.database]
> > You can glean all needed information from the resultset after having > issued the query. > >> It's probably true to say that the syntax/semantics of the interface >> will suck slightly more in the Variant case than in the struct case. > > That's a given. The suckiness won't come, however, in the form of > additional trips to the database. > > > Andrei Maybe in some cases. But at least with MySQL you have to bind before you fetch, and you don't have the meta-data about the columns until after the fetch, so you have to insert a call to mysql_stmt_result_metadata() to set up the Variant types before you bind. I also discovered after some time wasted this morning that the MySQL struct used for date/time is too big for Variant. I had to define a MyVariant taking that struct into consideration. Is Variant supposed to throw if you initialize an instance with a struct that is bigger than maxSize? It didn't - I just got mysterious segfaults when fetch tried to store the result via a null buffer pointer. Anyway, that's working now. >From the comments we're getting, it looks like we need to head toward a generic std.sql (or std.database) that provides basic functionality for most databases. This would be implemented over more capable modules for individual databases that covered the differentiating features as well as what would be needed for the generic case. Kind of between your options 2 and 3. Steve
Re: [std.database]
Here's a sketch of an interface. This is based on my experiments with
MySQL, and as such it is probably mid-level, and not a top level covers-
all interface.
Hopefully it will create a number of discussion points.
// Can interfaces include template functions???
interface SQLDBConnection
{
@property Handle handle();
Handle connect(string host, string user, string password,
string database = null);
T getProperty(T)(string name);
T getProperty(T)(int id);
void setProperty(T)(T property, string name);
void setProperty(T)(T property, int id);
Handle disconnect();
}
// There should possibly be a connection pool as well, and that
// should handle the RAII aspects of connections. Handle is an
// alias to suit the database system.
interface Raw
{
// Delete, insert, update, createXXX, and the like - no result set
rowcount_t exec(string sql);
// Select and such with result set - result set buffered to the
// client to provide a Random Access Range of Rows
rowcount_t execResultSet(string sql);
// Select and such with result set - prepares for sequential
// processing of an Input Range of Rows
void execSequence(string sql);
// Do the range defining methods need to be in the interface?
}
enum ParamDirection
{
ParamIn,
ParamOut,
ParamInOut
}
interface Prepared
{
void createParam(T)(ref T target, ParamDirection pd);
void createInParams(T...)(ref T args)
void createOutParams(T...)(ref T args)
void createVariantParam(ref Variant v, ParamDirection pd);
void createVariantParams(T...)(ref Variant[] va, T);
// If D type arrays are the bound type, it's likely that some
// updating of the bindings will be required when a new value
// is set, since a.ptr and a.length may change. Otherwise
// these operations are no-ops.
void updateInputParam(T)(ref T target);
void updateInParameters(T...)(ref T args);
void updateInArray(Variant[]);
void updateInStruct(S)(ref S s);
// Create a set of in parameters from an array of Variants
void setInArray(ref Variant[] va);
// Create a set of out parameters from an array of Variants
void setOutArray(ref Variant[] va);
// Initialize an array of out Variants to types appropriate for a query
void getTypesForArray(ref MyVariant[] va);
// Create a set of input params from a struct
void setInStruct(S)(ref S s) if (is(S== struct));
// Create a set of out params from a struct
void setOutStruct(S)(ref S s) if (is(S== struct));
prepare(string sql);
// Delete, update, createXXX, and the like - no result set
// returns rows affected;
rowcount_t exec();
// Select and such with result set - result set buffered
// to the client to
// provide a Random Access Range of Rows
rowcount_t execResultSet();
// Select and such with result set - prepares for sequential
// processing of an Input Range of Rows
void execSequence();
// A composite operation prepare, bind, and execute a statement
// to get a single column value into a D variable.
// execScalar(T)(ref T target);
// Do the range defining methods need to be in the interface?
}
interface Row // mmm bit close to Raw
{
// Get the values from a fetched row into a D struct
void rowToStruct(S)(ref S s) if (is(S == struct));
// Get the values from a fetched row into an array of Variants
void rowToStruct(ref Variant[] va);
// Get a column value by index into a D variable from the current row
T getValue(T)(out T target, int index, out bool isnull);
// Get a column value by index into a D variable from the current row
T getValue(T)(out T target, string colName, out bool isnull)
string toString(uint index);
string toString(string colName);
}
interface ResultSet
{
// Get the entire result set into an array of structs/Variants
S[] getAllRows(S)(ref S dummy) if (is(S == struct));
Variant[] getAllRows();
// This should be automated where possible
void free();
}
I can currently do most of this for MySQL, and what I haven't done
is mostly rehashing of what I have.
As an example of how level 2 interfaces may differ from the top-level one
is that in my implementation, chunking is supported for transfer and
disposal of large objects - either auto-chunking, or chunking via a
delegate. That stuff is not shown here.
Steve
Re: [std.database]
Surprising. I read a research paper about a proposed language just a few months ago. I wonder if this is by the same guys. Sent from my iPhone On Oct 10, 2011, at 12:05 AM, "Roald Ribe" wrote: > On Sun, 09 Oct 2011 20:31:35 -0300, Sean Kelly wrote: > >> On Oct 9, 2011, at 3:56 PM, Andrei Alexandrescu wrote: >> >>> On 10/9/11 5:31 PM, Walter Bright wrote: On 10/9/2011 5:28 AM, Piotr Szturmaj wrote: > 1. I think that we should not design this API using the least common > denominator > approach. This is to not limit some databases. For example PostgreSQL > has many > great features not available in MySQL. That's why I started with > postgres in my > ddb project. I think DB API should be designed to support the most > featureful > databases and those that have less features may be easily adapted to > that API. Haven't common denominator designs been more or less failures in at least one category - gui libraries? >>> >>> A common database interface is not a common denominator API; more like the >>> opposite. This is not difficult because most differences across database >>> systems lie in their SQL, which is strings from D's perspective. >> >> Assuming that by "database" you mean SQL. Pretty fair assumption, though >> NoSQL databases (which cover a broad range of designs since there's no >> standard language yet for key-value DBs, etc) are rapidly gaining >> popularity. I almost wonder if the base type should be named SqlDatabase >> instead of Database. > > There is a standard language defined for NoSQL, namely UnQL: > http://.unqlspec.org/display/UnQL/Home > > Roald
Re: [std.database]
I've just been looking at the documentation for the PostgreSQL C api. Wow! It is so different from MySQL, and so clean. No out parameters from queries. That one is not going to be a problem. Steve
Re: [std.database]
On 10/10/11 7:01 AM, Steve Teale wrote: You can glean all needed information from the resultset after having issued the query. It's probably true to say that the syntax/semantics of the interface will suck slightly more in the Variant case than in the struct case. That's a given. The suckiness won't come, however, in the form of additional trips to the database. Andrei Maybe in some cases. But at least with MySQL you have to bind before you fetch, and you don't have the meta-data about the columns until after the fetch, so you have to insert a call to mysql_stmt_result_metadata() to set up the Variant types before you bind. I'm confused. Isn't binding setting values to parameters prior to executing a query? That's a different thing from getting the row of a resultset. So, you first prepare, bind, and execute a query. Then you call mysql_stmt_result_metadata() to get the number of columns in the resultset and their individual types. At that point you get to allocate the Variant[] row appropriately. After that you're ready to iterate rows. I also discovered after some time wasted this morning that the MySQL struct used for date/time is too big for Variant. That's a bug in std.variant. Large structs should be supported automatically by using an indirection and dynamic allocation. Andrei
