Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-09 Thread Scott Cain
Oh, and I forgot to mention: it's highly compressed (bzip2 -9) and is 109M. Scott On Tue, 2003-08-05 at 11:01, Scott Cain wrote: > Joe, > > Good idea, since I may not get around to profiling it this week. I > created a dump of the data set I was working with. It is available at > http://www.gm

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-06 Thread Joe Conway
Scott Cain wrote: Oh, and I forgot to mention: it's highly compressed (bzip2 -9) and is 109M. Thanks. I'll grab a copy from home later today and see if I can find some time to poke at it. Joe ---(end of broadcast)--- TIP 1: subscribe and unsubscr

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-06 Thread Scott Cain
Joe, Good idea, since I may not get around to profiling it this week. I created a dump of the data set I was working with. It is available at http://www.gmod.org/string_dump.bz2 Thanks, Scott On Mon, 2003-08-04 at 16:29, Joe Conway wrote: > Is there a sample table schema and dataset available

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Shridhar Daithankar
On 4 Aug 2003 at 12:14, Scott Cain wrote: > I forgot about searching--I suspect that application is why I faced > opposition for shredding in my schema development group. Maybe I should > push that off to the file system and use grep (or BLAST). Otherwise, I > could write a function that would se

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Matt Clark
> > 2. If you want to search for a sequence you'll need to deal with the case > > where it starts in one chunk and ends in another. > > I forgot about searching--I suspect that application is why I faced > opposition for shredding in my schema development group. Maybe I should > push that off to t

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Joe Conway
Scott Cain wrote: On Mon, 2003-08-04 at 11:53, Tom Lane wrote: I find it really, really hard to believe that a crude reimplementation in plpgsql of the TOAST concept could beat the built-in implementation at all, let alone beat it by two orders of magnitude. Either there's something unrealistic abo

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Scott Cain
On Mon, 2003-08-04 at 11:53, Tom Lane wrote: > Scott Cain <[EMAIL PROTECTED]> writes: > > At least this appears to work and is much faster, completing substring > > operations like above in about 0.27 secs (that's about two orders of > > magnitude improvement!) > > I find it really, really hard to

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Scott Cain
On Mon, 2003-08-04 at 11:55, Richard Huxton wrote: > On Monday 04 August 2003 16:25, Scott Cain wrote: > [snip] > > [snip] > > You might want some checks to make sure that smin < smax, otherwise looks like > it does the job in a good clean fashion. Good point--smin < smax generally by virtue of

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Richard Huxton
On Monday 04 August 2003 16:25, Scott Cain wrote: [snip] > In an effort to find the best way to do this operation, I decided to > look at what is my "worst case" scenario: the DNA sequence for human > chromosome 1, which is about 250 million characters long (previous > strings where about 20 millio

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Tom Lane
Scott Cain <[EMAIL PROTECTED]> writes: > At least this appears to work and is much faster, completing substring > operations like above in about 0.27 secs (that's about two orders of > magnitude improvement!) I find it really, really hard to believe that a crude reimplementation in plpgsql of the

Re: [SQL] [PERFORM] EXTERNAL storage and substring on long strings

2003-08-04 Thread Scott Cain
Hello, Note: there is a SQL question way at the bottom of this narrative :-) Last week I asked about doing substring operations on very long strings (>10 million characters). I was given a suggestion to use EXTERNAL storage on the column via the ALTER TABLE ... SET STORAGE command. In one test