[REBOL] online database manipulation Re:(4)
Thanks, Brian... yes, it does help. I'm far more concerned with memory usage than speed, especially if a number of people are running the script at once. Your tip looks very worthwhile! --Ralph > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] > Sent: Friday, April 21, 2000 6:03 PM > To: [EMAIL PROTECTED] > Subject: [REBOL] online database manipulation Re:(3) > > > Hi Ralph! > > Gabrielle wrote: > > > Hello [EMAIL PROTECTED]! > > > > > > On 21-Apr-00, you wrote: > > > > > > r> a: read/lines %auto.db > > > > > > > > > r> a: to-hash a > > > > > > You don't need a hash if you're just using functions like PICK and > > > not FIND, SELECT etc. This conversion could take some time if the > > > db is very big... > > > > > You replied: > > Thanks, Gabriele, but I am searching each line for a specific > > string. I hope I am correct in that converting it to hash > > speeds up such searches. > > Hashes only speed up searching if you arrange the data as in pairs, > [key value key value ...]. It won't help your database directly, > but would help if you used a hash as an index. > > Say %auto.db is a flat file with tab-delimited fields and line- > delimited records. If you wanted a straight name index, and the > name was in the first field, you could do something like this: > > ; Split the database into records > a: read/lines %auto.db > ; Make index as block at first for insert speed > i: make block! 2 * length? a > foreach r a [ > ; Split record into fields > r: parse/all r "^-" > ; Insert first field and record into index > i: insert insert i r/1 r > ] > ; Change index into hash for efficiency > i: make hash! head i > ; Now select the record as often as you want > result: select i value > > but that only makes sense if you are doing many searches in one > transaction, quite unlike what you are doing on your web site. > > Instead, try this: > > value: whatever... > lines: make list! 0 ; List for insertion speed/space > use [a rb re] [ > ; Read the database in text mode > a: read %auto.db > ; Find the value in the database > while [re: find a value] [ ; Use find/any here? > ; Get the beginning of the record > either rb: find/reverse re newline [rb: next rb] [rb: a] > ; Get the end of the record > if none? re: find rb newline [re: tail rb] > ; Set the data to the next record > a: next re > ; Store a copy of the record, split into fields > lines: tail insert lines (parse (copy/part rb re) "^-") > ] > ] > ; Here are the found record lines > lines: head lines > > It may not be as pretty, but it's much faster and uses less > memory as well. You can even add wildcard searching by using > find/any instead of find where marked above, but that might > find patterns that span records. > > I suppose that parse could be used instead, but I don't feel > like figuring out how right now, and it would be about as fast. > > This technique does require you to read the entire database > into memory for every transaction though, just like your first > solution did. If you wanted to use open/lines instead of read > you would be back to doing a find on each line in turn. > > Does this help? > > Brian Hawley >
[REBOL] online database manipulation Re:
Hi, I might consider saving the database natively as a REBOL hash, only updating it as needed from the database using the script you wrote. I would imagine if you could get away with doing this it would save a ton of time. --Ryan [EMAIL PROTECTED] wrote: > Hi REBOLers: > > Among many other REBOLian projects, I am putting my ref book, THE SANDERS > PRICE GUIDE TO AUTOGRAPHS on line. It has over 80,000 prices. Until > REBOL/command is available and I can use mySQL, I've got the info on the > server as a tab-delimited text file. I read it into REBOL via this > technique: > > a: read/lines %auto.db > > a: to-hash a > b: [] > > random/seed now/time > > repeat count 10 [append b pick a (random ((length? a) - 10)) > ] > > You can a test of this in action (getting 10 random lookups every time > Refresh is hit) at http://abooks.com/cgi-bin/randauto.r. > > My question: can anyone suggest a more efficient way to doing this? > > Thanks, > > --Ralph Roberts > REBmeister, > ALEXANDER BOOKS
[REBOL] online database manipulation Re:(3)
Hi Ralph! Gabrielle wrote: > > Hello [EMAIL PROTECTED]! > > > > On 21-Apr-00, you wrote: > > > > r> a: read/lines %auto.db > > > > > > r> a: to-hash a > > > > You don't need a hash if you're just using functions like PICK and > > not FIND, SELECT etc. This conversion could take some time if the > > db is very big... > > You replied: > Thanks, Gabriele, but I am searching each line for a specific > string. I hope I am correct in that converting it to hash > speeds up such searches. Hashes only speed up searching if you arrange the data as in pairs, [key value key value ...]. It won't help your database directly, but would help if you used a hash as an index. Say %auto.db is a flat file with tab-delimited fields and line- delimited records. If you wanted a straight name index, and the name was in the first field, you could do something like this: ; Split the database into records a: read/lines %auto.db ; Make index as block at first for insert speed i: make block! 2 * length? a foreach r a [ ; Split record into fields r: parse/all r "^-" ; Insert first field and record into index i: insert insert i r/1 r ] ; Change index into hash for efficiency i: make hash! head i ; Now select the record as often as you want result: select i value but that only makes sense if you are doing many searches in one transaction, quite unlike what you are doing on your web site. Instead, try this: value: whatever... lines: make list! 0 ; List for insertion speed/space use [a rb re] [ ; Read the database in text mode a: read %auto.db ; Find the value in the database while [re: find a value] [ ; Use find/any here? ; Get the beginning of the record either rb: find/reverse re newline [rb: next rb] [rb: a] ; Get the end of the record if none? re: find rb newline [re: tail rb] ; Set the data to the next record a: next re ; Store a copy of the record, split into fields lines: tail insert lines (parse (copy/part rb re) "^-") ] ] ; Here are the found record lines lines: head lines It may not be as pretty, but it's much faster and uses less memory as well. You can even add wildcard searching by using find/any instead of find where marked above, but that might find patterns that span records. I suppose that parse could be used instead, but I don't feel like figuring out how right now, and it would be about as fast. This technique does require you to read the entire database into memory for every transaction though, just like your first solution did. If you wanted to use open/lines instead of read you would be back to doing a find on each line in turn. Does this help? Brian Hawley
[REBOL] online database manipulation Re:
Remember how excited you were when you described this great new language called Rebol ? And do you remember when your boss asked you if Rebol could talk to the db and you had to just hang your head? Well, things are looking up! Can Rebol/Core really talk to databases today? Yup! I think some of you database folks out there are really going to get a kick out of this. Especially if you have access to ms-sql-server7. My apologies for the rough state of the code. Please share with us all any juicy little improvements made. (See attached file: sqlproxy.r) Remember, it ain't a party until Rebol shows up! Galt Barber <[EMAIL PROTECTED]> p.s. If this is an old trick by now, my apologies; I have been unable to monitor the list for quite some time now... sqlproxy.r
[REBOL] online database manipulation Re:(2)
> Hello [EMAIL PROTECTED]! > > On 21-Apr-00, you wrote: > > r> a: read/lines %auto.db > > > r> a: to-hash a > > You don't need a hash if you're just using functions like PICK and > not FIND, SELECT etc. This conversion could take some time if the > db is very big... > Thanks, Gabriele, but I am searching each line for a specific string. I hope I am correct in that converting it to hash speeds up such searches. To see the beta searchable version of our autograph database, check http://abooks.com/cgi-bin/search.r. ... love REBOL, it makes development fast and easy. Best, --Ralph Roberts
[REBOL] online database manipulation Re:
Thanks, Allen... a good tip! --Ralph > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] > Sent: Thursday, April 20, 2000 9:32 PM > To: [EMAIL PROTECTED] > Subject: [REBOL] online database manipulation > > > Hi REBOLers: > > Among many other REBOLian projects, I am putting my ref book, THE SANDERS > PRICE GUIDE TO AUTOGRAPHS on line. It has over 80,000 prices. Until > REBOL/command is available and I can use mySQL, I've got the info on the > server as a tab-delimited text file. I read it into REBOL via this > technique: > > a: read/lines %auto.db > > > a: to-hash a > b: [] > > random/seed now/time > > repeat count 10 [append b pick a (random ((length? a) - 10)) > ] > > You can a test of this in action (getting 10 random lookups every time > Refresh is hit) at http://abooks.com/cgi-bin/randauto.r. > > My question: can anyone suggest a more efficient way to doing this? > > Thanks, > > --Ralph Roberts > REBmeister, > ALEXANDER BOOKS >
[REBOL] online database manipulation Re:
- Original Message - From: <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, April 21, 2000 11:32 AM Subject: [REBOL] online database manipulation > Hi REBOLers: > > Among many other REBOLian projects, I am putting my ref book, THE SANDERS > PRICE GUIDE TO AUTOGRAPHS on line. It has over 80,000 prices. Until > REBOL/command is available and I can use mySQL, I've got the info on the > server as a tab-delimited text file. I read it into REBOL via this > technique: > Hi Ralph, Just a small tweak for you. If the length of 'a doesn't need to be looked up all the time in the loop, then assign it once. e.g a-length: length? a repeat count 10 [append b pick a (random a-length - 10)) Allen K > a: read/lines %auto.db > > > a: to-hash a > b: [] > > random/seed now/time > > repeat count 10 [append b pick a (random ((length? a) - 10)) > ] > > You can a test of this in action (getting 10 random lookups every time > Refresh is hit) at http://abooks.com/cgi-bin/randauto.r. > > My question: can anyone suggest a more efficient way to doing this? > > Thanks, > > --Ralph Roberts > REBmeister, > ALEXANDER BOOKS
