Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
Thanks for all the advice. I'm not familiar with PG's COPY, so I would take a look. anyway, I will start small, and make it more functional bit by bit :-) . On Sat, Jul 31, 2010 at 9:35 AM, Stewart Smith stew...@flamingspork.comwrote: On Fri, 30 Jul 2010 08:29:45 -0700, Rob Wultsch wult...@gmail.com wrote: Does Drizzle need all the featues of LOAD DATA INFILE? COPY in pg is much more basic and works fine... probably not fully... and I think COPY would be great to have too. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Sat, Jul 31, 2010 at 2:34 AM, ZQ zim...@gmail.com wrote: Thanks for all the advice. I'm not familiar with PG's COPY, so I would take a look. anyway, I will start small, and make it more functional bit by bit :-) . I have been thinking about this a bit. I have never liked LOAD DATA INFILE. My big gripe is that as far as I know there is no way to get it to work like a normal *nix program in terms out stdin. The same sort of thought applies to SELECT INTO OUTFILE, I want to use stdout. (can one emulate stdin/stdout usage by using a FIFO?). I wonder if there would be some way for two servers to be using the same file as a table through the filesystem storage engine in order to achieve something like what I would like with stdin/stdout. I wonder if something like the pgdump custom format might be desirable in the future. PG uses their version of SELECT INTO OUTFILE (COPY) to create a compressed single file of table(s) (or entire schemas) archive including all the DDL. Individual schems/tables can be restored through the use of the pgrestore utility. Are any decisions being made now that would make it easier have something similar? -- Rob Wultsch wult...@gmail.com ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Fri, Jul 30, 2010 at 5:05 AM, Stewart Smith stew...@flamingspork.comwrote: On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? following the existing code to do LOAD DATA INFILE shouldn't be *too* hard, and some can probably be pretty easily reused. Then I'm afraid some of the code in filesystem storage will be moved to the kernel in drizzle. Finishing the basics of LOAD DATA INFILE is not hard, but to accomplish all the requirements is a little tricky especially for these ENCLOSING/ESCAPED_BY (I have a hard time to figure them out...) --Zimin ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Fri, Jul 30, 2010 at 6:50 AM, ZQ zim...@gmail.com wrote: On Fri, Jul 30, 2010 at 5:05 AM, Stewart Smith stew...@flamingspork.com wrote: On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? following the existing code to do LOAD DATA INFILE shouldn't be *too* hard, and some can probably be pretty easily reused. Then I'm afraid some of the code in filesystem storage will be moved to the kernel in drizzle. Finishing the basics of LOAD DATA INFILE is not hard, but to accomplish all the requirements is a little tricky especially for these ENCLOSING/ESCAPED_BY (I have a hard time to figure them out...) --Zimin Does Drizzle need all the featues of LOAD DATA INFILE? COPY in pg is much more basic and works fine... -- Rob Wultsch wult...@gmail.com ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Fri, 30 Jul 2010 21:50:22 +0800, ZQ zim...@gmail.com wrote: Then I'm afraid some of the code in filesystem storage will be moved to the kernel in drizzle. Finishing the basics of LOAD DATA INFILE is not hard, but to accomplish all the requirements is a little tricky especially for these ENCLOSING/ESCAPED_BY (I have a hard time to figure them out...) Hrm... I think we need enclosed by and escaped by too (for the filesystem_engine). But for starting with LOAD DATA INFILE, perhaps can start out without this option. so then both features are developed in parallel (the engine itself and the code to make it work for LOAD DATA INFILE). To do that, do the LAD DATA INFILE work in a different branch. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Fri, 30 Jul 2010 08:29:45 -0700, Rob Wultsch wult...@gmail.com wrote: Does Drizzle need all the featues of LOAD DATA INFILE? COPY in pg is much more basic and works fine... probably not fully... and I think COPY would be great to have too. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
[Drizzle-discuss] about replacing loadfile with filesystem storage engine
Hi Stewart, You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? Thanks. --Zimin ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? following the existing code to do LOAD DATA INFILE shouldn't be *too* hard, and some can probably be pretty easily reused. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Jul 29, 2010, at 2:04 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? Hi! Basically, I don't think we need two blocks of code doing relatively similar things. I think we can replace the LOAD DATA INFILE with some code that does the following: - create temporary table using filesystem_engine - open target table - open temporary table - read row from temporary table - write row to target table - repeat until done - close tables. I don't think many people are using LOAD DATA INFILE with direct file access on the servers. Thats just a really bad paradigm, especially since the code that does that in MySQL is really flaky (for instance, does not check for errors during read, so if your NFS mount where your source file hiccups, you just go Oh, EOF, return instead of Hey user your data wasn't fully loaded! LOAD DATA LOCAL INFILE is far more common, and has its own set of weirdness. I'd much rather see the LOAD DATA commands dropped in favor of the way postgresql does this, with the COPY command. With COPY, you can basically just say COPY INTO table and it uses the socket as a data stream of text to fill the table. It would be awesome if this command can leverage the code that the filesystem_engine uses, but I don't know if it makes sense to have the engine itself be at the core. ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On 07/29/2010 02:04 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? Hi! Basically, I don't think we need two blocks of code doing relatively similar things. I think we can replace the LOAD DATA INFILE with some code that does the following: - create temporary table using filesystem_engine - open target table - open temporary table - read row from temporary table - write row to target table - repeat until done - close tables. ++ ++ ++ (Imagine me continuing to do this) ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Thu, 29 Jul 2010 14:24:30 -0700, Clint Byrum cl...@fewbar.com wrote: I don't think many people are using LOAD DATA INFILE with direct file access on the servers. Thats just a really bad paradigm, especially since the code that does that in MySQL is really flaky (for instance, does not check for errors during read, so if your NFS mount where your source file hiccups, you just go Oh, EOF, return instead of Hey user your data wasn't fully loaded! LOAD DATA LOCAL INFILE is far more common, and has its own set of weirdness. I think people aren't as well for all sorts of reasons too... of course, our code should not be flaky. One benefit of code reuse here is that we could do wacky things like only have one place where we have to make sure error handling works :) I'd much rather see the LOAD DATA commands dropped in favor of the way postgresql does this, with the COPY command. With COPY, you can basically just say COPY INTO table and it uses the socket as a data stream of text to fill the table. We should probably have something like that too. It would be awesome if this command can leverage the code that the filesystem_engine uses, but I don't know if it makes sense to have the engine itself be at the core. It probably does, we're wanting things in the row format eventually anyway, so having the same bit of code doing the conversion, not so bad. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
