Re: [Drizzle-discuss] Drizzle on MS windows
Hi, pushed the code changes to bzr ( lp:~jobinau/libdrizzle/mingwport ) Requesting your reivew. 2 points need to consider. windows 32 bit compiation defines _WIN32 windows 64 bit compilation defines both _WIN32 *and* _WIN64 so to detect whether it is windows platform we need to check _WIN32 only. final liking needs -lwsock32 -lws2_32 parameters to be passed to link against windows libraries. detailed handweaving steps i used in windows are documented here: http://drizzlenotes.blogspot.com/ Thank you, Jobin. On Sun, Jul 25, 2010 at 12:37 AM, Monty Taylor mord...@inaugust.com wrote: On 07/24/2010 11:11 AM, Jobin Augustine wrote: Happy News is that now libdrizzle fully compiles natively on MS windows. we got it right!. during testing, found few run time issues, but that too ironed out (mainly because the windows error codes retured are different than posix ones) so silly things made me crazy and took lots of time. EXCELLENT. Monty, Can you guide me to push the code back to libdrizzle. Main part of the work is from your branch only. :) Yup. Continuing to chat with you on IRC about this - we'll get this all fixed up. Thanks for all your work! On Sat, Jul 24, 2010 at 8:54 AM, Jobin Augustine jobi...@gmail.com mailto:jobi...@gmail.com wrote: Step 1: Kill Windows Step 2: live happy life. Stewart, You said it correctly. :) Now realizing how bad it is. first i tried to set up Msys (bash for windows) and other mingw build stuffs. but someway windows 7 don't like all these things. windows keeps crashing the configure script. :( IMHO, this confirms Monty's approch is the best for native compilation and in the right direction (Cross compile it from linux) So that we at least have properly working build environment Monty, lp:~mordred/libdrizzle/mingwport gave me a good starting. Robert, the suggested libwinport.lib approch is very similar to Monty's approch. gnulib's poll.c and poll.h can make a object file which an be used to link at the end just like your libwinport.lib. Thank you. Left with no weapons (configure, make etc) in windows, started hand weaving the compilation like: C:\MinGW\bin\gcc.exe -I. -IC:\MinGW\include -c libdrizzle\drizzle.c -o libdrizzle\libdrizzle_la-drizzle.o C:\MinGW\bin\gcc.exe -I. -IC:\MinGW\include -c libdrizzle\conn.c -o libdrizzle\libdrizzle_la-conn.o C:\MinGW\bin\gcc.exe -I. -IC:\MinGW\include -c libdrizzle\conn_uds.c -o libdrizzle\libdrizzle_la-conn_uds.o Thank you, Jobin. On Tue, Jul 20, 2010 at 3:13 AM, Stewart Smith stew...@flamingspork.com mailto:stew...@flamingspork.com wrote: On Sat, 17 Jul 2010 09:59:21 -0500, Monty Taylor mord...@inaugust.com mailto:mord...@inaugust..com wrote: If you rewrite something like the my_socket abstraction that I did for NDB win32 port, socket code should be pretty easy. I would strongly prefer it if we would not have any wrappers such as this. (although I understand that in the case of NDB and MySQL it was required) so the pain point was that you can kinda do this... except for closing sockets. close() will cause a *runtime* error popping up to the end user. it is ass. maybe there's some way to get something that isn't completely stupid.. but at least historically (NDB porting time), there wasn't. Essentially, any time we have the urge to write a my_ anything, we should not. Step 1: Kill Windows Step 2: live happy life. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] Drizzle on MS windows
Excellent. I have a test/define in configure for windows now... I'll get that sorted. On 07/29/2010 04:32 AM, Jobin Augustine wrote: Hi, pushed the code changes to bzr ( lp:~jobinau/libdrizzle/mingwport ) Requesting your reivew. 2 points need to consider. windows 32 bit compiation defines _WIN32 windows 64 bit compilation defines both _WIN32 *and* _WIN64 so to detect whether it is windows platform we need to check _WIN32 only. final liking needs -lwsock32 -lws2_32 parameters to be passed to link against windows libraries. detailed handweaving steps i used in windows are documented here: http://drizzlenotes.blogspot.com/ Thank you, Jobin. On Sun, Jul 25, 2010 at 12:37 AM, Monty Taylor mord...@inaugust.com mailto:mord...@inaugust.com wrote: On 07/24/2010 11:11 AM, Jobin Augustine wrote: Happy News is that now libdrizzle fully compiles natively on MS windows. we got it right!. during testing, found few run time issues, but that too ironed out (mainly because the windows error codes retured are different than posix ones) so silly things made me crazy and took lots of time. EXCELLENT. Monty, Can you guide me to push the code back to libdrizzle. Main part of the work is from your branch only. :) Yup. Continuing to chat with you on IRC about this - we'll get this all fixed up. Thanks for all your work! On Sat, Jul 24, 2010 at 8:54 AM, Jobin Augustine jobi...@gmail.com mailto:jobi...@gmail.com mailto:jobi...@gmail.com mailto:jobi...@gmail.com wrote: Step 1: Kill Windows Step 2: live happy life. Stewart, You said it correctly. :) Now realizing how bad it is. first i tried to set up Msys (bash for windows) and other mingw build stuffs. but someway windows 7 don't like all these things. windows keeps crashing the configure script. :( IMHO, this confirms Monty's approch is the best for native compilation and in the right direction (Cross compile it from linux) So that we at least have properly working build environment Monty, lp:~mordred/libdrizzle/mingwport gave me a good starting. Robert, the suggested libwinport.lib approch is very similar to Monty's approch. gnulib's poll.c and poll.h can make a object file which an be used to link at the end just like your libwinport.lib. Thank you. Left with no weapons (configure, make etc) in windows, started hand weaving the compilation like: C:\MinGW\bin\gcc.exe -I. -IC:\MinGW\include -c libdrizzle\drizzle.c -o libdrizzle\libdrizzle_la-drizzle.o C:\MinGW\bin\gcc.exe -I. -IC:\MinGW\include -c libdrizzle\conn.c -o libdrizzle\libdrizzle_la-conn.o C:\MinGW\bin\gcc.exe -I. -IC:\MinGW\include -c libdrizzle\conn_uds.c -o libdrizzle\libdrizzle_la-conn_uds.o Thank you, Jobin. On Tue, Jul 20, 2010 at 3:13 AM, Stewart Smith stew...@flamingspork.com mailto:stew...@flamingspork.com mailto:stew...@flamingspork.com mailto:stew...@flamingspork.com wrote: On Sat, 17 Jul 2010 09:59:21 -0500, Monty Taylor mord...@inaugust.com mailto:mord...@inaugust.com mailto:mord...@inaugust. mailto:mord...@inaugust..com wrote: If you rewrite something like the my_socket abstraction that I did for NDB win32 port, socket code should be pretty easy. I would strongly prefer it if we would not have any wrappers such as this. (although I understand that in the case of NDB and MySQL it was required) so the pain point was that you can kinda do this... except for closing sockets. close() will cause a *runtime* error popping up to the end user. it is ass. maybe there's some way to get something that isn't completely stupid.. but at least historically (NDB porting time), there wasn't. Essentially, any time we have the urge to write a my_ anything, we should not. Step 1: Kill Windows Step 2: live happy life. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
[Drizzle-discuss] about replacing loadfile with filesystem storage engine
Hi Stewart, You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? Thanks. --Zimin ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
[Drizzle-discuss] Pythian Group: Determining I/O throughput for a system
Pythian Group: Determining I/O throughput for a system
At Kscope this year, I attended a half day in-depth session entitled Data Warehousing Performance Best Practices, given by Maria Colgan of Oracle. In that session, there was a section on how to determine I/O throughput for a system, because in data warehousing I/O per second (iops) is less important than I/O throughput (how much actual data goes through, not just how many reads/writes).
The section contained an Oracle-specific in-database tool, and a standalone tool that can be used on many operating systems, regardless of whether or not a database exists:
If Oracle is installed, run DBMS_RESOURCE_MANAGER.CALIBRATE_IO:
SET SERVEROUTPUT ON
DECLARE
lat INTEGER;
iops INTEGER;
mbps INTEGER;
BEGIN
-- DBMS_RESOURCE_MANAGER.CALIBRATE_IO(DISKS, MAX_LATENCY,iops,mbps,lat);
DBMS_RESOURCE_MANAGER.CALIBRATE_IO (2, 10, iops, mbps, lat);
dbms_output.put_line('max_mbps = ' || mbps);
end;
For us MySQL folks, or even the Drizzle or NoSQL folks, Oracle offers a free standalone tool called Orion. The example given in the slides was:
./orion –run advanced –testname mytest –num_small 0 –size_large 1024 –type rand –simulate contact –write 0 –duration 60 –matrix column
-num_small is 0 because you dont usually do small transactions in a dw.
-type rand for random I/Os because data warehouse queries usually dont do sequential reads
-write 0 no writes, because you do not write often to the dw, that is what the ETL is for.
-duration is in seconds
-matrix column shows you how much you can sustain
I would be interested to see how other folks measure I/O throughput, and maybe even do a side-by-side comparison of different tools. Orion is available for:
Linux (x86, x86-64, Itanium, Power)
Solaris (SPARC64)
AIX (PPC64)
zLinux
HPUX (PA RISC, Itanium)
Windows
I am working on a larger write-up of the session itself, which had many concise descriptions of data warehousing issues, but I thought that this merited its own post.URL: http://www.pythian.com/news/15161/determining-io-throughput-for-a-system/
___
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : drizzle-discuss@lists.launchpad.net
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] Drizzle on MS windows
On Thu, 29 Jul 2010 17:02:55 +0530, Jobin Augustine jobi...@gmail.com wrote: detailed handweaving steps i used in windows are documented here: http://drizzlenotes.blogspot.com/ We should get this on planetdrizzle too... Lee, is that you? -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
[Drizzle-discuss] Jobin Augustine: Drizzle on Windows - Libdrizzle
Jobin Augustine: Drizzle on Windows - Libdrizzle lately I started thinking from a users point of view.Web developers want to try drizzle. but many of them got windows platform for devleopment.So they are not able to connect to drizzle serverso started seaching for options. found that i have 3 options.1. Native compile the source using MinGW.2. Take the help of cygwin to get posix platform3. Coss-compile the source from Linux to windows.Toru, Stewart and Monty responded to my views on drizzle-discuss.Monty was already trying on libdrizzle for sometime.and Toru also voted for libdrizzle first.Monty's approach was to use option 3 with option 1.(later i realized his approach is better and smart). like:./configure --build=x86_64-unknown-linux-gnu --host=i586-mingw32msvceverybody voted for libdrizzle to be available for windows first.That voting makes sence because developer in windows should be able to connect to ser! ver first.My step was to get libdrizzle compiled using cygwin.cygwin gives full Linux system calls in a dll. that is a very big offer for me.i could not deny that.as expected everything went smooth in cygwin.libdrizzle got compiled without any effort. and i am able to verify the connectivity from windows xp box.Next step is very challenging.(The step to compile libdrizzle on windows natively without help of cygwin1.dll)for a non-programmer like me it is a herculean task.Monty's working branch (lp:~mordred/libdrizzle/mingwport)gave a kick start because he already done all background work.after adding few more lines to Monty's work, (drizzle.h and conn.c )I am able to compile files from mingwit was bit challenging to figure out that following defines are must have (i added them to common.h)#define WINVER WindowsXP#define _WIN32_WINNT 0x0501(otherw! ise haunting, "undefined reference to `freeaddrinfo'"and undefined re ference to `getaddrinfo'"" like messages will come when you try to link)My effort is to create static library which can be linked to make executable.since configure and make are giving me trouble on windows, i tried handweave the compilation.to see what is happening in the makelogged all commands executed when libdrizzle is getting compiled with cygwin$make V=1more over go a good guidence from:http://www.adp-gmbh.ch/cpp/gcc/create_lib.htmlStep 1. compiling the library files:all below executed in windows command prompt."C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\drizzle.c -o libdrizzle\libdrizzle_la-drizzle.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\conn.c -o libdrizzle\libdrizzle_la-conn.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\conn_uds.c -o libdrizzle\libdrizzle_la-conn_uds.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\! MinGW\include" -c libdrizzle\handshake.c -o libdrizzle\libdrizzle_la-handshake.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\command.c -o libdrizzle\libdrizzle_la-command.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\query.c -o libdrizzle\libdrizzle_la-query.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\result.c -o libdrizzle\libdrizzle_la-result.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\column.c -o libdrizzle\libdrizzle_la-column.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\row.c -o libdrizzle\libdrizzle_la-row.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\field.c -o libdrizzle\libdrizzle_la-field.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\pack.c -o libdrizzle\libdrizzle_la-pack.o"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\state.c -o libdrizzle\libdrizzle_la-state.o! "C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c libdrizzle\sh a1.c -o libdrizzle\libdrizzle_la-sha1.oStep 2. get the poll implementation on windowscd libdrizzle\win32"C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c poll.c -o poll.ocd ..\..Step 3. Create a single static library (libdrizzle.a)"C:\MinGW\bin\ar.exe" cru libdrizzle/libdrizzle.a libdrizzle/libdrizzle_la-drizzle.o libdrizzle/libdrizzle_la-conn.o libdrizzle/libdrizzle_la-conn_uds.o libdrizzle/libdrizzle_la-handshake.o libdrizzle/libdrizzle_la-command.o libdrizzle/libdrizzle_la-query.o libdrizzle/libdrizzle_la-result.o libdrizzle/libdrizzle_la-column.o libdrizzle/libdrizzle_la-row.o libdrizzle/libdrizzle_la-field.o libdrizzle/libdrizzle_la-pack.o libdrizzle/libdrizzle_la-state.o libdrizzle/libdrizzle_la-sha1.o libdrizzle/win32/poll.oStep 4. Compile user program and link to the above created library."C:\MinGW\bin\gcc.exe" -I. -I"C:\MinGW\include" -c examples/simple.c -o examples/simple.o"C:\Min! GW\bin\gcc.exe" -op -o examples/simple.exe examples/simple.o ./libdrizzle/libdrizzle.a -lwsock32 -lws2_32cd examplesStep 5. Test the user program.simple.exe -d test -h 10.97.93.243 -q "select * from t1"I want to thank you Monty for the huge help and guidance i recived.URL:
Re: [Drizzle-discuss] Drizzle on MS windows
On 07/29/2010 12:22 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 17:02:55 +0530, Jobin Augustine jobi...@gmail.com wrote: pushed the code changes to bzr ( lp:~jobinau/libdrizzle/mingwport ) Requesting your reivew. (one way to do this is via a merge request on launchpad). I'm just about to do that. I merged him in to me and then did build fixes. Anyway, some thoughts: I wonder if the #define WINVER and _WIN32_WINNT defines should be in common.h, or somehow coming out of configure into config.h? They should, however, be behind a #ifdef WIN32 Yup. Did that. around connect() in conn.c, the indenting seems a bit off with the rest of the code. Yup. Fixed. We can use send() and recv() on Unix too. I cannot think of any current reason not to do that and save some #ifdefs in drizzle.c, the WSAStartup call should be only once per process, correct? This isn't guaranteed by drizzle_create()... perhaps we need a library init function (can't remember how to do this with DLLs on Win32) or some kind of condition around it. I did not do anything with restructuring in this way. I'd put a flag around it ( like if (wsa_initted) ) though rather than a library init function ... if we can avoid requiring library users needing to call drizzle_lib_init I'd really like to. Submitting merge req in just a sec... Monty ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? following the existing code to do LOAD DATA INFILE shouldn't be *too* hard, and some can probably be pretty easily reused. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] Drizzle on MS windows
On Thu, 29 Jul 2010 13:46:46 -0700, Monty Taylor mord...@inaugust.com wrote: I did not do anything with restructuring in this way. I'd put a flag around it ( like if (wsa_initted) ) though rather than a library init function ... if we can avoid requiring library users needing to call drizzle_lib_init I'd really like to. You may be able to do a constructor for the library that's automatically called on the equiv of dlopen() (I do this in libeatmydata for example) -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Jul 29, 2010, at 2:04 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? Hi! Basically, I don't think we need two blocks of code doing relatively similar things. I think we can replace the LOAD DATA INFILE with some code that does the following: - create temporary table using filesystem_engine - open target table - open temporary table - read row from temporary table - write row to target table - repeat until done - close tables. I don't think many people are using LOAD DATA INFILE with direct file access on the servers. Thats just a really bad paradigm, especially since the code that does that in MySQL is really flaky (for instance, does not check for errors during read, so if your NFS mount where your source file hiccups, you just go Oh, EOF, return instead of Hey user your data wasn't fully loaded! LOAD DATA LOCAL INFILE is far more common, and has its own set of weirdness. I'd much rather see the LOAD DATA commands dropped in favor of the way postgresql does this, with the COPY command. With COPY, you can basically just say COPY INTO table and it uses the socket as a data stream of text to fill the table. It would be awesome if this command can leverage the code that the filesystem_engine uses, but I don't know if it makes sense to have the engine itself be at the core. ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On 07/29/2010 02:04 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 23:39:42 +0800, ZQ zim...@gmail.com wrote: You once mentioned one thing I can do is to replace load file with this my file system storage engine, do you mean I should abstract some general code from my storage engine and plug it into load file code path? I would like to know what's the potential problem with the original load file functionality? Hi! Basically, I don't think we need two blocks of code doing relatively similar things. I think we can replace the LOAD DATA INFILE with some code that does the following: - create temporary table using filesystem_engine - open target table - open temporary table - read row from temporary table - write row to target table - repeat until done - close tables. ++ ++ ++ (Imagine me continuing to do this) ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] Drizzle on MS windows
On 07/29/2010 02:12 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 13:46:46 -0700, Monty Taylor mord...@inaugust.com wrote: I did not do anything with restructuring in this way. I'd put a flag around it ( like if (wsa_initted) ) though rather than a library init function ... if we can avoid requiring library users needing to call drizzle_lib_init I'd really like to. You may be able to do a constructor for the library that's automatically called on the equiv of dlopen() (I do this in libeatmydata for example) Yes - but it's not even remotely portable (I did some research on this a while ago. You can do this in certain environments for certain usage patterns... but sadly it's wonky enough that unless it's something like libeatmydata, it's probably just not safe. (libdrizzle is gonna get sucked in by all sorts of stuff) In this case, doing the flag check should be _really_ easy. Monty ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] Drizzle on MS windows
On 29. juli 2010, at 23.48, Monty Taylor wrote: On 07/29/2010 02:12 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 13:46:46 -0700, Monty Taylor mord...@inaugust.com wrote: I did not do anything with restructuring in this way. I'd put a flag around it ( like if (wsa_initted) ) though rather than a library init function ... if we can avoid requiring library users needing to call drizzle_lib_init I'd really like to. You may be able to do a constructor for the library that's automatically called on the equiv of dlopen() (I do this in libeatmydata for example) Yes - but it's not even remotely portable (I did some research on this a while ago. You can do this in certain environments for certain usage patterns... but sadly it's wonky enough that unless it's something like libeatmydata, it's probably just not safe. (libdrizzle is gonna get sucked in by all sorts of stuff) In this case, doing the flag check should be _really_ easy. Isn't this something the application that use libdrizzle should do, and not something that the library should do automatically? Other parts of the application may also use sockets Cheers, Trond ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] about replacing loadfile with filesystem storage engine
On Thu, 29 Jul 2010 14:24:30 -0700, Clint Byrum cl...@fewbar.com wrote: I don't think many people are using LOAD DATA INFILE with direct file access on the servers. Thats just a really bad paradigm, especially since the code that does that in MySQL is really flaky (for instance, does not check for errors during read, so if your NFS mount where your source file hiccups, you just go Oh, EOF, return instead of Hey user your data wasn't fully loaded! LOAD DATA LOCAL INFILE is far more common, and has its own set of weirdness. I think people aren't as well for all sorts of reasons too... of course, our code should not be flaky. One benefit of code reuse here is that we could do wacky things like only have one place where we have to make sure error handling works :) I'd much rather see the LOAD DATA commands dropped in favor of the way postgresql does this, with the COPY command. With COPY, you can basically just say COPY INTO table and it uses the socket as a data stream of text to fill the table. We should probably have something like that too. It would be awesome if this command can leverage the code that the filesystem_engine uses, but I don't know if it makes sense to have the engine itself be at the core. It probably does, we're wanting things in the row format eventually anyway, so having the same bit of code doing the conversion, not so bad. -- Stewart Smith ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] Drizzle on MS windows
On 07/29/2010 03:04 PM, Trond Norbye wrote: On 29. juli 2010, at 23.48, Monty Taylor wrote: On 07/29/2010 02:12 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 13:46:46 -0700, Monty Taylor mord...@inaugust.com wrote: I did not do anything with restructuring in this way. I'd put a flag around it ( like if (wsa_initted) ) though rather than a library init function ... if we can avoid requiring library users needing to call drizzle_lib_init I'd really like to. You may be able to do a constructor for the library that's automatically called on the equiv of dlopen() (I do this in libeatmydata for example) Yes - but it's not even remotely portable (I did some research on this a while ago. You can do this in certain environments for certain usage patterns... but sadly it's wonky enough that unless it's something like libeatmydata, it's probably just not safe. (libdrizzle is gonna get sucked in by all sorts of stuff) In this case, doing the flag check should be _really_ easy. Isn't this something the application that use libdrizzle should do, and not something that the library should do automatically? Other parts of the application may also use sockets Hrm. I dunno. Why does windows suck? What's the best practice here? ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
Re: [Drizzle-discuss] Drizzle on MS windows
On 07/29/2010 03:04 PM, Trond Norbye wrote: On 29. juli 2010, at 23.48, Monty Taylor wrote: On 07/29/2010 02:12 PM, Stewart Smith wrote: On Thu, 29 Jul 2010 13:46:46 -0700, Monty Taylor mord...@inaugust.com wrote: I did not do anything with restructuring in this way. I'd put a flag around it ( like if (wsa_initted) ) though rather than a library init function ... if we can avoid requiring library users needing to call drizzle_lib_init I'd really like to. You may be able to do a constructor for the library that's automatically called on the equiv of dlopen() (I do this in libeatmydata for example) Yes - but it's not even remotely portable (I did some research on this a while ago. You can do this in certain environments for certain usage patterns... but sadly it's wonky enough that unless it's something like libeatmydata, it's probably just not safe. (libdrizzle is gonna get sucked in by all sorts of stuff) In this case, doing the flag check should be _really_ easy. Isn't this something the application that use libdrizzle should do, and not something that the library should do automatically? Other parts of the application may also use sockets Ok. I did some reading. It's perfectly fine to call it more than once. It is, in fact, NOT ok to call it from DllMain, so the auto-run bit would be extra bad. So I'm going to vote for leaving this how Jobin had it. If the calling app needs other sockets, they can call WSAStartup and be fine. If the calling app just wants to talk to Drizzle, they can just use the library and also be fine. Done. ___ Mailing list: https://launchpad.net/~drizzle-discuss Post to : drizzle-discuss@lists.launchpad.net Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
