Re: [PATCHES] [HACKERS] large object regression tests
Patch applied. Thanks. --- Jeremy Drake wrote: On Sun, 24 Sep 2006, Jeremy Drake wrote: On Thu, 21 Sep 2006, Tom Lane wrote: I think we could do without the Moby Dick extract too ... I am open to suggestions. I saw one suggestion that I use an image of an elephant, but I suspect that was tongue-in-cheek. I am not very fond of the idea of generating repetitious data, as I think it would be more difficult to determine whether or not the loseek/tell functions put me in the right place in the middle of the file. I just had the idea that I could use one of the existing data files which are used for testing COPY instead of the Moby Dick extract. They are already there, a few of them are pretty good sized, they have data in the file which is not just simple repetition so it would be pretty obvious if the seek function broke, and they are very unlikely to change. I am considering changing the test I put together to use tenk.data as the input file tomorrow and send in what I have again, since I also am doing a test of \lo_import (which also requires a patch to psql I sent in earlier to fix the output of the \lo_* commands to respect the output settings). -- When does summertime come to Minnesota, you ask? Well, last year, I think it was a Tuesday. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] [HACKERS] large object regression tests
Sorry, please disregard. Was means for a Japanese FAQ update. --- Bruce Momjian wrote: Patch applied. Thanks. --- Jeremy Drake wrote: On Sun, 24 Sep 2006, Jeremy Drake wrote: On Thu, 21 Sep 2006, Tom Lane wrote: I think we could do without the Moby Dick extract too ... I am open to suggestions. I saw one suggestion that I use an image of an elephant, but I suspect that was tongue-in-cheek. I am not very fond of the idea of generating repetitious data, as I think it would be more difficult to determine whether or not the loseek/tell functions put me in the right place in the middle of the file. I just had the idea that I could use one of the existing data files which are used for testing COPY instead of the Moby Dick extract. They are already there, a few of them are pretty good sized, they have data in the file which is not just simple repetition so it would be pretty obvious if the seek function broke, and they are very unlikely to change. I am considering changing the test I put together to use tenk.data as the input file tomorrow and send in what I have again, since I also am doing a test of \lo_import (which also requires a patch to psql I sent in earlier to fix the output of the \lo_* commands to respect the output settings). -- When does summertime come to Minnesota, you ask? Well, last year, I think it was a Tuesday. ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Bruce Momjian [EMAIL PROTECTED] EnterpriseDBhttp://www.enterprisedb.com + If your life is a hard drive, Christ can be your backup. + ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] [HACKERS] large object regression tests
On Sun, 24 Sep 2006, Jeremy Drake wrote: On Thu, 21 Sep 2006, Tom Lane wrote: I suggest that instead of testing the server-side lo_import/lo_export functions, perhaps you could test the psql equivalents and write and read a file in psql's working directory. snip In the mean time, I will alter the test to also test the psql backslash commands based on how the copy equivalents are tested, since I had forgotten them and they need to be tested also. I just tried using the \lo_import command in a regression test, and I think I figured out why this will not work: $ make check ... largeobject ... FAILED ... $ cat regression.diffs *** ./expected/largeobject.out Sun Sep 24 19:55:25 2006 --- ./results/largeobject.out Sun Sep 24 19:55:58 2006 *** *** 188,194 (1 row) \lo_import 'results/lotest.txt' ! lo_import 31138 \set newloid :LASTOID -- This is a hack to test that export/import are reversible -- This uses knowledge about the inner workings of large object mechanism --- 188,194 (1 row) \lo_import 'results/lotest.txt' ! lo_import 31199 \set newloid :LASTOID -- This is a hack to test that export/import are reversible -- This uses knowledge about the inner workings of large object mechanism == Yes, that's the large object OID in the output there, and it is different each run (as I expect). If you look at src/bin/psql/large_obj.c line 192, you see: fprintf(pset.queryFout, lo_import %u\n, loid); Which is executed unconditionally whenever the lo_import is successful. While in a normal circumstance, it is quite necessary to know the loid, since it does change each call, in this case it serves to break the diffs, and so I guess it is impossible to use the \lo_import command in a regression test. -- The first time, it's a KLUDGE! The second, a trick. Later, it's a well-established technique! -- Mike Broido, Intermetrics ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] [HACKERS] large object regression tests
On Mon, 25 Sep 2006, Tom Lane wrote: Jeremy Drake [EMAIL PROTECTED] writes: I just tried using the \lo_import command in a regression test, and I think I figured out why this will not work: ... Yes, that's the large object OID in the output there, and it is different each run (as I expect). Right. I'd suggest temporarily setting ECHO off to hide the unpredictable part of the output. There are similar measures taken in many of the contrib tests. I tried this: jeremyd=# \set QUIET jeremyd=# \set ECHO off jeremyd=# BEGIN; jeremyd=# \lo_import results/lotest.txt lo_import 84951 jeremyd=# ROLLBACK; From what I could tell in the code, the message is printed regardless of setting. It looks like the large_obj.c output is missing much of the output settings handling which is in the PrintQueryStatus function in common.c, such as handling quiet mode, and html output. I will try to dig around and try to put together a patch to make it respect the settings like other commands... -- You are old, said the youth, and your programs don't run, And there isn't one language you like; Yet of useful suggestions for help you have none -- Have you thought about taking a hike? Since I never write programs, his father replied, Every language looks equally bad; Yet the people keep paying to read all my books And don't realize that they've been had. ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings
Re: [PATCHES] [HACKERS] large object regression tests
On Sun, 24 Sep 2006, Jeremy Drake wrote: On Thu, 21 Sep 2006, Tom Lane wrote: I think we could do without the Moby Dick extract too ... I am open to suggestions. I saw one suggestion that I use an image of an elephant, but I suspect that was tongue-in-cheek. I am not very fond of the idea of generating repetitious data, as I think it would be more difficult to determine whether or not the loseek/tell functions put me in the right place in the middle of the file. I just had the idea that I could use one of the existing data files which are used for testing COPY instead of the Moby Dick extract. They are already there, a few of them are pretty good sized, they have data in the file which is not just simple repetition so it would be pretty obvious if the seek function broke, and they are very unlikely to change. I am considering changing the test I put together to use tenk.data as the input file tomorrow and send in what I have again, since I also am doing a test of \lo_import (which also requires a patch to psql I sent in earlier to fix the output of the \lo_* commands to respect the output settings). -- When does summertime come to Minnesota, you ask? Well, last year, I think it was a Tuesday. ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [PATCHES] [HACKERS] large object regression tests
Jeremy Drake [EMAIL PROTECTED] writes: I just tried using the \lo_import command in a regression test, and I think I figured out why this will not work: ... Yes, that's the large object OID in the output there, and it is different each run (as I expect). Right. I'd suggest temporarily setting ECHO off to hide the unpredictable part of the output. There are similar measures taken in many of the contrib tests. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] [HACKERS] large object regression tests
On Thu, 21 Sep 2006, Tom Lane wrote: Jeremy Drake [EMAIL PROTECTED] writes: I put together a patch which adds a regression test for large objects, hopefully attached to this message. I would like some critique of it, to see if I have gone about it the right way. Also I would be happy to hear any additional tests which should be added to it. I'd prefer it if we could arrange not to need any absolute paths embedded into the test, because maintaining tests that require such is a real PITA --- instead of just committing the actual test output, one has to reverse-convert it to a .source file. I just copied how the test for COPY worked, since I perceived a similarity in what I needed to do (use external files to load data). I suggest that instead of testing the server-side lo_import/lo_export functions, perhaps you could test the psql equivalents and write and read a file in psql's working directory. I did not see any precedent for that when I was looking around in the existing tests for an example of how to do things. I am not even sure where the cwd of psql is, so I can put an input file there. Could you provide an example of how this might look, by telling me where to put a file in the src/test/regress tree and the path to give to \lo_import? Besides which, shouldn't both the server-side and psql versions be tested? When I was looking at the copy tests, it looked like the server-side ones were tested, and then the psql ones were tested by exporting and then importing data which was originally loaded from the server-side method. Am I correctly interpreting the precedent, or are you suggesting that the precedent be changed? I was trying to stay as close to the copy tests as possible since the functionality is so similar (transferring data to/from files in the filesystem, either via server-side functions which require absolute paths or via psql \ commands (which I forgot about for the lo funcs)). I think we could do without the Moby Dick extract too ... I am open to suggestions. I saw one suggestion that I use an image of an elephant, but I suspect that was tongue-in-cheek. I am not very fond of the idea of generating repetitious data, as I think it would be more difficult to determine whether or not the loseek/tell functions put me in the right place in the middle of the file. Perhaps if there was a way to generate deterministic pseudo-random data, that would work (has to be deterministic so the diffs of the output come out right). Anyone have a good example of seeding a random number generator and generating a bunch of bytea which is deterministic cross-platform? regards, tom lane In the mean time, I will alter the test to also test the psql backslash commands based on how the copy equivalents are tested, since I had forgotten them and they need to be tested also. -- Any sufficiently advanced technology is indistinguishable from a rigged demo. ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] [HACKERS] large object regression tests
On Sun, 24 Sep 2006, Jeremy Drake wrote: On Thu, 21 Sep 2006, Tom Lane wrote: I suggest that instead of testing the server-side lo_import/lo_export functions, perhaps you could test the psql equivalents and write and read a file in psql's working directory. I did not see any precedent for that when I was looking around in the existing tests for an example of how to do things. snip When I was looking at the copy tests, it looked like the server-side ones were tested, and then the psql ones were tested by exporting and then importing data which was originally loaded from the server-side method. I just went back and looked at the tests again. The only time the psql \copy command was used was in the (quite recent IIRC) copyselect test, and then only via stdout (never referring to psql working directory, or to files at all). Did I misunderstand, and you are proposing a completely new way of doing things in the regression tests? I am not particularly fond of the sed substitution stuff myself, but it seems to be the only currently supported/used method in the regression tests... I do think that making the large object test and the copy test consistent would make a lot of sense, since as I said before, the functionality of file access is so similar... -- We demand rigidly defined areas of doubt and uncertainty! -- Vroomfondel ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] [HACKERS] large object regression tests
Jeremy Drake [EMAIL PROTECTED] writes: I put together a patch which adds a regression test for large objects, hopefully attached to this message. I would like some critique of it, to see if I have gone about it the right way. Also I would be happy to hear any additional tests which should be added to it. I'd prefer it if we could arrange not to need any absolute paths embedded into the test, because maintaining tests that require such is a real PITA --- instead of just committing the actual test output, one has to reverse-convert it to a .source file. I suggest that instead of testing the server-side lo_import/lo_export functions, perhaps you could test the psql equivalents and write and read a file in psql's working directory. I think we could do without the Moby Dick extract too ... regards, tom lane ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [HACKERS] large object regression tests
On Tuesday 05 September 2006 02:59, Jeremy Drake wrote: I am considering, and I think that in order to get a real test of the large objects, I would need to load data into a large object which would be sufficient to be loaded into more than one block (large object blocks were 1 or 2K IIRC) so that the block boundary case could be tested. Is there any precedent on where to grab such a large chunk of data from? I was thinking about using an excerpt from a public domain text such as Moby Dick, but on second thought binary data may be better to test things with. A 5 or 6 megapixel JPEG image. Maybe a photograph of an elephant. -- Lamar Owen Director of Information Technology Pisgah Astronomical Research Institute 1 PARI Drive Rosman, NC 28772 (828)862-5554 www.pari.edu ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [HACKERS] large object regression tests
Hi, Jeremy, Jeremy Drake wrote: I am considering, and I think that in order to get a real test of the large objects, I would need to load data into a large object which would be sufficient to be loaded into more than one block (large object blocks were 1 or 2K IIRC) so that the block boundary case could be tested. Is there any precedent on where to grab such a large chunk of data from? You could generate such data on the fly, as part of the test scripts. E. G. a blob of zero bytes, blob of 0xff bytes, a blob of pseudo random data... Markus -- Markus Schaber | Logical TrackingTracing International AG Dipl. Inf. | Software Development GIS Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org signature.asc Description: OpenPGP digital signature
[HACKERS] large object regression tests
I noticed when I was working on a patch quite a while back that there are no regression tests for large object support. I know, large objects are not the most sexy part of the code-base, and I think they tend to be ignored/forgotten most of the time. Which IMHO is all the more reason they should have some regression tests. Otherwise, if someone managed to break them somehow, it is quite likely not to be noticed for quite some time. So in this vein, I have recently found myself with some free time, and a desire to contribute something, and decided this would be the perfect place to get my feet wet without stepping on any toes. I guess what I should ask is, would a patch to add a test for large objects to the regression suite be well received? And, is there any advice for how to go about making these tests? I am considering, and I think that in order to get a real test of the large objects, I would need to load data into a large object which would be sufficient to be loaded into more than one block (large object blocks were 1 or 2K IIRC) so that the block boundary case could be tested. Is there any precedent on where to grab such a large chunk of data from? I was thinking about using an excerpt from a public domain text such as Moby Dick, but on second thought binary data may be better to test things with. My current efforts, and probably the preliminary portion of the final test, involves loading a small amount (less than one block) of text into a large object inline from a sql script and calling the various functions against it to verify that they do what they should. In the course of doing so, I find that it is necessary to stash certain values across statements (large object ids, large object 'handles'), and so far I am using a temporary table to store these. Is this reasonable, or is there a cleaner way to do that? -- Even if you're on the right track, you'll get run over if you just sit there. -- Will Rogers ---(end of broadcast)--- TIP 4: Have you searched our list archives? http://archives.postgresql.org
Re: [HACKERS] large object regression tests
Jeremy Drake [EMAIL PROTECTED] writes:
I noticed when I was working on a patch quite a while back that there are
no regression tests for large object support.
Yeah, this is bad :-(
I am considering, and I think that in order to get a real test of the
large objects, I would need to load data into a large object which would
be sufficient to be loaded into more than one block (large object blocks
were 1 or 2K IIRC) so that the block boundary case could be tested. Is
there any precedent on where to grab such a large chunk of data from?
There's always plain old junk data, eg, repeat('xyzzy', 10).
I doubt that Moby Dick would expose any unexpected bugs ...
... I find that it is necessary to stash certain values across
statements (large object ids, large object 'handles'), and so far I am
using a temporary table to store these. Is this reasonable, or is there a
cleaner way to do that?
I think it's supposed to be possible to use psql variables for that;
if you can manage to test psql variables as well as large objects,
that'd be a double bonus.
regards, tom lane
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
[HACKERS] large object regression tests
Sorry if this gets through more than once, I seem to be having email difficulties... On Tue, 5 Sep 2006, Jeremy Drake wrote: I noticed when I was working on a patch quite a while back that there are no regression tests for large object support. I know, large objects are not the most sexy part of the code-base, and I think they tend to be ignored/forgotten most of the time. Which IMHO is all the more reason they should have some regression tests. Otherwise, if someone managed to break them somehow, it is quite likely not to be noticed for quite some time. So in this vein, I have recently found myself with some free time, and a desire to contribute something, and decided this would be the perfect place to get my feet wet without stepping on any toes. I guess what I should ask is, would a patch to add a test for large objects to the regression suite be well received? And, is there any advice for how to go about making these tests? I am considering, and I think that in order to get a real test of the large objects, I would need to load data into a large object which would be sufficient to be loaded into more than one block (large object blocks were 1 or 2K IIRC) so that the block boundary case could be tested. Is there any precedent on where to grab such a large chunk of data from? I was thinking about using an excerpt from a public domain text such as Moby Dick, but on second thought binary data may be better to test things with. My current efforts, and probably the preliminary portion of the final test, involves loading a small amount (less than one block) of text into a large object inline from a sql script and calling the various functions against it to verify that they do what they should. In the course of doing so, I find that it is necessary to stash certain values across statements (large object ids, large object 'handles'), and so far I am using a temporary table to store these. Is this reasonable, or is there a cleaner way to do that? -- Never make anything simple and efficient when a way can be found to make it complex and wonderful. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
