Re: Reading/Deleting Last Line Of File
Jeff Massung wrote: I'm still new to Rev, but in other languages this is dead simple: ... // nuke everything else at the end of the file trunctate(fp, new_len); ... Done. Now, maybe this isn't as easy in Rev as it is in C and *many* other languages. But it should be [if it isn't]. That is *exactly* the problem - Rev doesn't expose ftruncate() (or ftell() either), so what should be very simple becomes very difficult. In C, or Python, or even shell it is simple, but Rev's incomplete interface to the file system causes the problem. I opened a RQCC report (#1851) in 2004 to request the enhancement of providing a way to get the file position (i.e. ftell()). That report remains Unconfirmed. Given that response, I have not chosen to waste any time entering an enhancement request for truncate. -- Alex. ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Richard Gaskin wrote: That looks similar to what I posted here on the 9th: open file tFile for update seek relative -1000 in file tFile repeat read from file tFile until cr if it is not empty then put it after tBuffer else delete last line of tBuffer write tBuffer to file tFile end if end repeat close file tFile Does that not do what you need? No, it doesn't. If a file is opened for 'update' then any write to the file simply overwrites any existing characters at the appropriates position(s), and leaves everything following that unchanged. There is no EOF implied by a write in update mode. There is an EOF inserted after a write if the file is opened in 'append' mode - but then you cannot read from it. (Actually, there is another additional reason - each time you do a 'read' that updates the file pointer, so the subsequent write would begin at the file position immediately *after* the last read, not where the read had been done from; you would have needed another seek before write (or do a 'write at x') to have a chance, but you'd still need Rev to provide access to truncate() after your write.) -- Alex. ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jim Bufalini wrote: Just one thing Alex, you need to: put URLDecode(the detailed files) into t in case the file name has, for example, a space in it. Thanks Jum, I didn't spot that. But then, if the file name had a comma in it, I'd be caught out. I think what I needed was to URLEncode(pFile) to use in the filter : on deletelastline pFile constant K = 1000 put the detailed files into t filter t with URLEncode(pFile) ,* put item 2 of t into tFileLength open file pFile for read seek to tFileLength-K in file pFile read from file pFile until end close file pFile put the number of chars in the last line of it into tNum open file pFile for append write empty to file pFile at (tFileLength-tNum-3) close file pFile end deletelastline -- Alex. ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Alex Tweedly wrote: Richard Gaskin wrote: That looks similar to what I posted here on the 9th: open file tFile for update seek relative -1000 in file tFile repeat read from file tFile until cr if it is not empty then put it after tBuffer else delete last line of tBuffer write tBuffer to file tFile end if end repeat close file tFile Does that not do what you need? No, it doesn't. If a file is opened for 'update' then any write to the file simply overwrites any existing characters at the appropriates position(s), and leaves everything following that unchanged. There is no EOF implied by a write in update mode. Good catch. Seems my instinct to use append in my first version of that was closer to the mark. But FWIW, I tried your version and it seemed to leave the file unchanged - is this a user error on my end, or does it have a limitation I overlooked? on deletelastline pFile constant K = 1000 put the detailed files into t filter t with URLEncode(pFile) ,* put item 2 of t into tFileLength open file pFile for read seek to tFileLength-K in file pFile read from file pFile until end close file pFile put the number of chars in the last line of it into tNum open file pFile for append write empty to file pFile at (tFileLength-tNum-3) close file pFile end deletelastline -- Richard Gaskin Fourth World Rev training and consulting: http://www.fourthworld.com Webzine for Rev developers: http://www.revjournal.com revJournal blog: http://revjournal.com/blog.irv ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Richard Gaskin wrote: But FWIW, I tried your version and it seemed to leave the file unchanged If your original file ends in a cr then Alex's code would end in absolutely no change. ;-) Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jim Bufalini wrote: Richard Gaskin wrote: But FWIW, I tried your version and it seemed to leave the file unchanged If your original file ends in a cr then Alex's code would end in absolutely no change. ;-) Thanks, Jim. I tried it both ways; no change to the file's contents, although the mod date gets updated so I know it's trying. -- Richard Gaskin Fourth World Rev training and consulting: http://www.fourthworld.com Webzine for Rev developers: http://www.revjournal.com revJournal blog: http://revjournal.com/blog.irv ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jim Bufalini wrote: Richard Gaskin wrote: But FWIW, I tried your version and it seemed to leave the file unchanged If your original file ends in a cr then Alex's code would end in absolutely no change. ;-) That's not what my testing showed (or appeared to show). Here's the short version of the test script, and the corresponding output note the first example has a single final CR, the second has no CR and the third one has multiple CRs, and each is correctly truncated. (I missed a few test cases here - empty file, single line of data, etc. but it is only reasonably tested) The test stack can be downloaded from http://www.tweedly.org/deletelastline.rev (btw - I don't currently have access to a mac, so it's only tested on Windows. Could that cause the different result for Richard ?) on mouseUp put empty into field F put abcdefg CR into t put 1234567 CR after t put t into URL (file:a,b.txt) put abcdefg CR into t put 1234567 after t put t into URL (file:b.txt) put abcdefg CR into t put 1234567 CR CR after t put t into URL (file:c.txt) putfile a,b.txt deletelastline a,b.txt putfile a,b.txt putfile b.txt deletelastline b.txt putfile b.txt putfile c.txt deletelastline c.txt putfile c.txt end mouseUp on putfile p put file p CR after field F put URL (file: P) after field F put endoffile CR after field F end putfile giving file a,b.txt abcdefg 1234567 endoffile file a,b.txt abcdefg endoffile file b.txt abcdefg 1234567endoffile file b.txt abcdefendoffile file c.txt abcdefg 1234567 endoffile file c.txt abcdefg 1234567 endoffile -- Alex. ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Richard Gaskin wrote: Jim Bufalini wrote: Richard Gaskin wrote: But FWIW, I tried your version and it seemed to leave the file unchanged If your original file ends in a cr then Alex's code would end in absolutely no change. ;-) Thanks, Jim. I tried it both ways; no change to the file's contents, although the mod date gets updated so I know it's trying. I strongly suspect it only works on Windows :-( Sorry. I don't have either a Mac or Linux box to try it on, however I did just try it on the on-rev server (as an .irev script) and it also appears to leave the file unchanged there. In fact, it gives an error (error in offset expression) on the seek, because the test file is less than 1000 chars - an error which didn't get flagged up on Windows. I changed the line from seek tFileLength-K in file pFile to if tFileLength-k 0 then seek tFileLength-K in file pFile to avoid the error. But it then leaves the file unchanged. I suspect the *trick* of opening a file for append, then seeking back into the middle of the file before doing a write is OS-dependent. It's certainly not documented (either way), and I was surprised when Jim said that it could be done. Intuitively, 'append' should mean that the existing file is unchangable - all you can do is add to it. Playing around some more, it appears that the results of write to file in these odd cases are not always what you'd expect. I would have expected the following two fragments to be equivalent seek to 2 in file b.txt write x to file b.txt and write x to file b.txt at 2 to be equivalent, but they are not. The latter truncates the file after the write, but the former does not (in Windows). Could you try this in your multi-line msg box and let me know what it does on Mac ? put abcdefg CR into URL file:b.txt put URL file:b.txt open file b.txt for append write x to file b.txt at 2 close file b.txt put URL file:b.txt after msg on Win, it produces abcdefg abx Thanks -- Alex. ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Alex Tweedly wrote: I suspect the *trick* of opening a file for append, then seeking back into the middle of the file before doing a write is OS-dependent. It's certainly not documented (either way), and I was surprised when Jim said that it could be done. Intuitively, 'append' should mean that the existing file is unchangable - all you can do is add to it. Playing around some more, it appears that the results of write to file in these odd cases are not always what you'd expect. I would have expected the following two fragments to be equivalent seek to 2 in file b.txt write x to file b.txt and write x to file b.txt at 2 to be equivalent, but they are not. The latter truncates the file after the write, but the former does not (in Windows). I have never used the *seek* that both you and Richard had in your code examples for write. I've never even used seek for a read, preferring to keep track of offsets myself. This is why, in the example that I gave back on the 9th (and tested at the time), I used *open for append - write at offset - close*. I also have never written an empty, but some truncated data. So, I just assumed and it sounded logical that seeking and then writing an empty was the equivalent of what I knew worked. I guess not. Good to know. In any case, I do know that if you open for append and write at least one char at an offset, it does truncate the file at the end of that char. And I know this works on at least PC and Mac. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Alex Tweedly wrote: I strongly suspect it only works on Windows :-( Sorry. I don't have either a Mac or Linux box to try it on, however I did just try it on the on-rev server (as an .irev script) and it also appears to leave the file unchanged there. I wonder if it has to do with the OS-dependent line-ending conversion that the engine does when it writes to text files. If you open the file as binfile instead of plain file you'll avoid that conversion. It might work differently that way. Playing around some more, it appears that the results of write to file in these odd cases are not always what you'd expect. I would have expected the following two fragments to be equivalent seek to 2 in file b.txt write x to file b.txt and write x to file b.txt at 2 That looks right. The write command always begins writing at the postion after the pointer. But when you specify an exact postion in particular, it writes where you tell it. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
J. Landman Gay wrote: Alex Tweedly wrote: I strongly suspect it only works on Windows :-( Sorry. I don't have either a Mac or Linux box to try it on, however I did just try it on the on-rev server (as an .irev script) and it also appears to leave the file unchanged there. I wonder if it has to do with the OS-dependent line-ending conversion that the engine does when it writes to text files. If you open the file as binfile instead of plain file you'll avoid that conversion. It might work differently that way. I'll withdraw this, it's wrong. Whatever OS your script is on, the file it writes will will have matching line endings. So nope. If I finish all the other stuff I need to do this weekend I'll try some of the tests everyone posted. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
On Sat, Feb 13, 2010 at 8:25 AM, Alex Tweedly a...@tweedly.net wrote: Could you try this in your multi-line msg box and let me know what it does on Mac ? put abcdefg CR into URL file:b.txt put URL file:b.txt open file b.txt for append write x to file b.txt at 2 close file b.txt put URL file:b.txt after msg on Win, it produces abcdefg abx Alex, on OS X.6.2, MBP, Rev 4.0 abcdefg abcdefg x :-( ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Kay C Lan wrote: On Sat, Feb 13, 2010 at 8:25 AM, Alex Tweedly a...@tweedly.net wrote: Could you try this in your multi-line msg box and let me know what it does on Mac ? put abcdefg CR into URL file:b.txt put URL file:b.txt open file b.txt for append write x to file b.txt at 2 close file b.txt put URL file:b.txt after msg on Win, it produces abcdefg abx Alex, on OS X.6.2, MBP, Rev 4.0 abcdefg abcdefg x I did some quick tests and it looks like the write command is broken on OS X. Endian issue with Intel Macs maybe. Don't open a file for write until they fix it, you can lose data. I'll file a bug report. The reason I just noticed this is because I was testing some of these suggestions. The version of open file we should be using, the one which truncates contents, is the open for write command. However, no matter at what position I write to the file, the entire contents were replaced. Don't go there if you're on a new Mac. You won't lose data with other types of open file, but the write position may be off. It sounds like it works okay on Windows, and I suspect Linux is okay too. I remember using open file for write on my PPC machine and had no trouble, which is why I suspect an Intel issue. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Kay C Lan wrote: On Sat, Feb 13, 2010 at 8:25 AM, Alex Tweedly a...@tweedly.net wrote: Could you try this in your multi-line msg box and let me know what it does on Mac ? put abcdefg CR into URL file:b.txt put URL file:b.txt open file b.txt for append write x to file b.txt at 2 close file b.txt put URL file:b.txt after msg on Win, it produces abcdefg abx Alex, on OS X.6.2, MBP, Rev 4.0 abcdefg abcdefg x I get what Kay got on OS X 10.5.8, which is the correct behavior. Append is supposed to force any writes to the end of the existing file. I tested some more with the open for write command, which should do what Alex was after -- truncating the text at the character offset, and adding any new data at that position. This fails on both Windows and Mac. Bug report has been submitted. For details, see: http://quality.runrev.com/qacenter/show_bug.cgi?id=8614 -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jeff Massung wrote: Warren, I've read through most of these suggestions, but I'm surprised that the obvious hasn't been suggested yet (that I've seen): skip everything... Hasn't been suggested because it won't work. put the length of url file:myfile.txt into tEnd open file myfile.txt for text update seek to tEnd in file myfile.txt From here just back up a reasonable number of characters... say 200. Find the last CR character, nuke everything else and close the file. Didn't find one? Try backing up another 200, etc. It'll be a whole lot faster. ... nuke everything else ?? How ? You've opened the file for update, so anything that you write to the file *overwrites* any existing characters at the same position(s), and leaves everything following that unchanged. So there is no way to shorten a file if opened in update mode. I *think* the best you can do is 1. find file length (best to use the detailed files) 2. open for read, and read some chunk at the end (200, 1000, whatever ) (open for read is more efficient than open for update) 3. calculate the number of bytes to be deleted off the end of the file 4. close the file 5. open the file for append 6. write empty to the file at the appropriate point Edge case to concern yourself with would be if your file happens to end with a CR and maybe you want to ignore those cases. But that should be easy enough to solve. Easiest (and I think best) way to resolve that corner case is to preserve the trailing CR if it's there to begin with, and not leave one if it wasn't there to start with. This happens to involve no extra work at all, hence it's the easiest one :-) I decided it was time to email some (reasonably tested) revTalk rather than just ideas (NB assumes that the defaultfolder has already been set to the appropriate place) on deletelastline pFile constant K = 1000 put the detailed files into t filter t with pFile ,* put item 2 of t into tFileLength open file pFile for read seek to tFileLength-K in file pFile read from file pFile until end close file pFile put the number of chars in the last line of it into tNum open file pFile for append write empty to file pFile at (tFileLength-tNum-3) close file pFile end deletelastline -- Alex. ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Alex Tweedly wrote: Jeff Massung wrote: Warren, I've read through most of these suggestions, but I'm surprised that the obvious hasn't been suggested yet (that I've seen): skip everything... Hasn't been suggested because it won't work. put the length of url file:myfile.txt into tEnd open file myfile.txt for text update seek to tEnd in file myfile.txt From here just back up a reasonable number of characters... say 200. Find the last CR character, nuke everything else and close the file. Didn't find one? Try backing up another 200, etc. It'll be a whole lot faster. ... nuke everything else ?? How ? You've opened the file for update, so anything that you write to the file *overwrites* any existing characters at the same position(s), and leaves everything following that unchanged. So there is no way to shorten a file if opened in update mode. I *think* the best you can do is 1. find file length (best to use the detailed files) 2. open for read, and read some chunk at the end (200, 1000, whatever ) (open for read is more efficient than open for update) 3. calculate the number of bytes to be deleted off the end of the file 4. close the file 5. open the file for append 6. write empty to the file at the appropriate point Edge case to concern yourself with would be if your file happens to end with a CR and maybe you want to ignore those cases. But that should be easy enough to solve. Easiest (and I think best) way to resolve that corner case is to preserve the trailing CR if it's there to begin with, and not leave one if it wasn't there to start with. This happens to involve no extra work at all, hence it's the easiest one :-) I decided it was time to email some (reasonably tested) revTalk rather than just ideas (NB assumes that the defaultfolder has already been set to the appropriate place) on deletelastline pFile constant K = 1000 put the detailed files into t filter t with pFile ,* put item 2 of t into tFileLength open file pFile for read seek to tFileLength-K in file pFile read from file pFile until end close file pFile put the number of chars in the last line of it into tNum open file pFile for append write empty to file pFile at (tFileLength-tNum-3) close file pFile end deletelastline -- Alex. Just one thing Alex, you need to: put URLDecode(the detailed files) into t in case the file name has, for example, a space in it. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
On Thu, Feb 11, 2010 at 5:49 PM, Alex Tweedly a...@tweedly.net wrote:
Jeff Massung wrote:
Warren,
I've read through most of these suggestions, but I'm surprised that the
obvious hasn't been suggested yet (that I've seen): skip everything...
Hasn't been suggested because it won't work.
put the length of url file:myfile.txt into tEnd
open file myfile.txt for text update
seek to tEnd in file myfile.txt
From here just back up a reasonable number of characters... say 200.
Find
the last CR character, nuke everything else and close the file. Didn't
find
one? Try backing up another 200, etc. It'll be a whole lot faster.
... nuke everything else ?? How ?
I'm still new to Rev, but in other languages this is dead simple:
void truncate_huge_file(const char* filename)
{
FILE* fp = fopen(filename,wb);
char bytes[200];
size_t new_len;
// move to the end of the file, read 200 bytes
fseek(fp, -200, SEEK_END);
fread(bytes, 1, 200, fp);
// .. TODO: scan for end of line, repeat as needed
// .. TODO: seek SEEK_CUR past what should still be there
new_len = ftell(fp);
// nuke everything else at the end of the file
trunctate(fp, new_len);
fclose(fp);
}
Done.
Now, maybe this isn't as easy in Rev as it is in C and *many* other
languages. But it should be [if it isn't].
Jeff M.
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jeff Massung wrote:
I'm still new to Rev, but in other languages this is dead simple:
void truncate_huge_file(const char* filename)
{
FILE* fp = fopen(filename,wb);
char bytes[200];
size_t new_len;
// move to the end of the file, read 200 bytes
fseek(fp, -200, SEEK_END);
fread(bytes, 1, 200, fp);
// .. TODO: scan for end of line, repeat as needed
// .. TODO: seek SEEK_CUR past what should still be there
new_len = ftell(fp);
// nuke everything else at the end of the file
trunctate(fp, new_len);
fclose(fp);
}
Done.
Now, maybe this isn't as easy in Rev as it is in C and *many* other
languages. But it should be [if it isn't].
That looks similar to what I posted here on the 9th:
open file tFile for update
seek relative -1000 in file tFile
repeat
read from file tFile until cr
if it is not empty then
put it after tBuffer
else
delete last line of tBuffer
write tBuffer to file tFile
end if
end repeat
close file tFile
Does that not do what you need?
--
Richard Gaskin
Fourth World
Rev training and consulting: http://www.fourthworld.com
Webzine for Rev developers: http://www.revjournal.com
revJournal blog: http://revjournal.com/blog.irv
___
use-revolution mailing list
use-revolution@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Thanks for all the suggestions. I will look into trying some of them to see what works best for me. The file currently contains 105,750,304 records. I will report back my findings. Thanks, Warren On Tue, Feb 9, 2010 at 11:58 PM, J. Landman Gay jac...@hyperactivesw.com wrote: Jim Bufalini wrote: Jacque wrote: Worth a test anyway. I've used it on files that were several megs in size without a problem, but they weren't super huge, just kinda big. Hi Jacque, I made the suggestion I did because at 100 Million records plus 100 million? Yes, well...I think I read the zeros wrong. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
You mean you'll report back when you only have 105,750,303 records ;-) On Wed, Feb 10, 2010 at 9:06 PM, Warren Kuhl warrenk...@gmail.com wrote: Thanks for all the suggestions. I will look into trying some of them to see what works best for me. The file currently contains 105,750,304 records. I will report back my findings. Thanks, Warren ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Warren, I've read through most of these suggestions, but I'm surprised that the obvious hasn't been suggested yet (that I've seen): skip everything... put the length of url file:myfile.txt into tEnd open file myfile.txt for text update seek to tEnd in file myfile.txt From here just back up a reasonable number of characters... say 200. Find the last CR character, nuke everything else and close the file. Didn't find one? Try backing up another 200, etc. It'll be a whole lot faster. Edge case to concern yourself with would be if your file happens to end with a CR and maybe you want to ignore those cases. But that should be easy enough to solve. HTH, Jeff M. On Wed, Feb 10, 2010 at 7:06 AM, Warren Kuhl warrenk...@gmail.com wrote: Thanks for all the suggestions. I will look into trying some of them to see what works best for me. The file currently contains 105,750,304 records. I will report back my findings. Thanks, Warren ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jeff, This line: put the length of url file:myfile.txt into tEnd Loads the entire file into memory in order to get its length. - Brian ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
On Feb 10, 2010, at 11:26 PM, Brian Yennie wrote: Jeff, This line: put the length of url file:myfile.txt into tEnd Loads the entire file into memory in order to get its length. That's a joke, right? :-( A freakin' OS call could get that just by touching an I-Node. Please, God, tell me Rev was smart enough to do it the right way? Jeff M.___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
It's more a product of the syntax than anything. url file:myfile.txt is a container, and the length operates on it as such. URL containers are pretty darn handy in general, but this is one potential gotcha. You can use the detailed files to get information about a file, including its size in bytes. So actually your suggestion should work fine, just needs a little tweaking. It would just be more like: put filelength(myfile.txt) into tEnd open file myfile.txt for text update seek to tEnd in file myfile.txt function filelength tFile // use the detailed files to get the file size end filelength No need to get all flustered =) - Brian On Feb 10, 2010, at 11:26 PM, Brian Yennie wrote: Jeff, This line: put the length of url file:myfile.txt into tEnd Loads the entire file into memory in order to get its length. That's a joke, right? :-( A freakin' OS call could get that just by touching an I-Node. Please, God, tell me Rev was smart enough to do it the right way? Jeff M.___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolutio ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
doesn't that work?: delete line -1 of file myTextFile Till Am 09.02.2010 um 14:52 schrieb Warren Kuhl: I have a large text file (100,000,000+ records). I need to write a utility that removes the last record of the file. Is there anyway to just remove the last record without reading through the complete file with RunRev? Thanks, Warren ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Warren, I have a large text file (100,000,000+ records). I need to write a utility that removes the last record of the file. Is there anyway to just remove the last record without reading through the complete file with RunRev? This is totally untested but logically you could: 1. First get the size of the file on disk and approximate a character offset that should be only a few lines from the end of the file. 2. Then Open the file for read and read from file starting at the char offset you calculated to EOF into a var and delete the last line of the var. 3. Then Close the file and Open it again for write. 4. Now write the var to file at the same starting offset you read from. The reason you don't use open for update is that you want to open the file for write, so that whatever you write to the file replaces everything from the offset to the EOF. Update will replace chars and leave the last record at the end. What I don't know here is if internally the read/write from file at start still has to go through the entire file to calculate the starting offset, so I don't know how long this will take on a file of your size. But, at least, doing it this way won't require reading the whole file into memory and should result in the last line being deleted from the file in the quickest way available. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jim, I will give this a try. I was trying to go down the same path by getting the number of records of the file...then trying to determine if I could just read the last line and removing the EOF. Is there anyway to specifiy a READ to a certain line of a file without reading through the whole file? Warren On Tue, Feb 9, 2010 at 8:34 AM, Jim Bufalini j...@visitrieve.com wrote: Warren, I have a large text file (100,000,000+ records). I need to write a utility that removes the last record of the file. Is there anyway to just remove the last record without reading through the complete file with RunRev? This is totally untested but logically you could: 1. First get the size of the file on disk and approximate a character offset that should be only a few lines from the end of the file. 2. Then Open the file for read and read from file starting at the char offset you calculated to EOF into a var and delete the last line of the var. 3. Then Close the file and Open it again for write. 4. Now write the var to file at the same starting offset you read from. The reason you don't use open for update is that you want to open the file for write, so that whatever you write to the file replaces everything from the offset to the EOF. Update will replace chars and leave the last record at the end. What I don't know here is if internally the read/write from file at start still has to go through the entire file to calculate the starting offset, so I don't know how long this will take on a file of your size. But, at least, doing it this way won't require reading the whole file into memory and should result in the last line being deleted from the file in the quickest way available. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Warren, I will give this a try. I was trying to go down the same path by getting the number of records of the file...then trying to determine if I could just read the last line and removing the EOF. Is there anyway to specifiy a READ to a certain line of a file without reading through the whole file? The read from file command allows you start at a character offset and then read to a string, or for a number of bytes/chars/words/items (see dictionary) or to EOF. The write command will replace from the starting character offset and move the EOF for you. So, in your case, I don't think you have to be exact in your read. Just read in close to the end to EOF and write from the same offset and let the write command handle the EOF for you. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jim, Thanks for the explanationI will give this a try. Thanks! Warren On Tue, Feb 9, 2010 at 9:08 AM, Jim Bufalini j...@visitrieve.com wrote: Warren, I will give this a try. I was trying to go down the same path by getting the number of records of the file...then trying to determine if I could just read the last line and removing the EOF. Is there anyway to specifiy a READ to a certain line of a file without reading through the whole file? The read from file command allows you start at a character offset and then read to a string, or for a number of bytes/chars/words/items (see dictionary) or to EOF. The write command will replace from the starting character offset and move the EOF for you. So, in your case, I don't think you have to be exact in your read. Just read in close to the end to EOF and write from the same offset and let the write command handle the EOF for you. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
I don't think this will work, Jim. If you open a file for write, then it erases the entire content of the file first as the docs say (emphasis added)... The file to write to must be opened first with the open file command, and the mode the file was opened in must be write, append, or update. If the file is not open or is open read-only, the result function is set to File is not open for write.. If the file was opened in write mode, the write to file command completely *replaces the file contents from the start.* For example, if the file originally contains ABC, and you write 1 to it, after the write to file command is executed the file contains 1. And as you say, open for update does not allow you to truncate the file. The standard C library has a way to do it, and there is a Unix shell command - but not afaict a Windows shell utility to do it. -- Alex Jim Bufalini wrote: Warren, I have a large text file (100,000,000+ records). I need to write a utility that removes the last record of the file. Is there anyway to just remove the last record without reading through the complete file with RunRev? This is totally untested but logically you could: 1. First get the size of the file on disk and approximate a character offset that should be only a few lines from the end of the file. 2. Then Open the file for read and read from file starting at the char offset you calculated to EOF into a var and delete the last line of the var. 3. Then Close the file and Open it again for write. 4. Now write the var to file at the same starting offset you read from. The reason you don't use open for update is that you want to open the file for write, so that whatever you write to the file replaces everything from the offset to the EOF. Update will replace chars and leave the last record at the end. What I don't know here is if internally the read/write from file at start still has to go through the entire file to calculate the starting offset, so I don't know how long this will take on a file of your size. But, at least, doing it this way won't require reading the whole file into memory and should result in the last line being deleted from the file in the quickest way available. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Piggy-backing here: I have a large text file (100,000,000+ records). I need to write a utility that removes the last record of the file. Is there anyway to just remove the last record without reading through the complete file with RunRev? It sounds like a big file, but Rev is pretty good about managing memory and swapping to disk. I'd be tempted to try this and see how it works: delete line -1 of url (file:mytext.txt) Worth a test anyway. I've used it on files that were several megs in size without a problem, but they weren't super huge, just kinda big. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Would the seek command help? Presumably it would move the file pointer to the specified point without having to load the entire file into RAM. If you knew your lines were shorter than a given length, say 1000 chars, you could so something like: open file tFile for append seek relative -1000 in file tFile repeat read from file tFile until cr if it is not empty then put it after tBuffer else delete last line of tBuffer write tBuffer to file tFile end if end repeat close file tFile This is off the top of my head to test before using on real data. ;) -- Richard Gaskin Fourth World Rev training and consulting: http://www.fourthworld.com Webzine for Rev developers: http://www.revjournal.com revJournal blog: http://revjournal.com/blog.irv ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
See why you shouldn't trust off-the-cuff code? Where I wrote: open file tFile for append ...I meant to write: open file tFile for update -- Richard Gaskin Fourth World Rev training and consulting: http://www.fourthworld.com Webzine for Rev developers: http://www.revjournal.com revJournal blog: http://revjournal.com/blog.irv ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Alex, It does work, just not exactly how I said to do it. ;-) You have to open the file for append*, instead of *write* and then write at a byte offset. So: open file filename for append write This is the EOF to file fileName at close file filename I just tested on an approximately 10kb text file and if is 5000, the This is the EOF is written starting at byte 5000 and the file is terminated at the end of that string and the file data before byte 5000 is retained in the file and the file changes to an approximately a 5KB file, so the EOF is moved and the file is truncated. I really shouldn't dash these answers off from memory. ;-) Aloha from Hawaii, Jim Bufalini Alex Tweedly wrote: I don't think this will work, Jim. If you open a file for write, then it erases the entire content of the file first as the docs say (emphasis added)... The file to write to must be opened first with the open file command, and the mode the file was opened in must be write, append, or update. If the file is not open or is open read-only, the result function is set to File is not open for write.. If the file was opened in write mode, the write to file command completely *replaces the file contents from the start.* For example, if the file originally contains ABC, and you write 1 to it, after the write to file command is executed the file contains 1. And as you say, open for update does not allow you to truncate the file. The standard C library has a way to do it, and there is a Unix shell command - but not afaict a Windows shell utility to do it. -- Alex Jim Bufalini wrote: Warren, I have a large text file (100,000,000+ records). I need to write a utility that removes the last record of the file. Is there anyway to just remove the last record without reading through the complete file with RunRev? This is totally untested but logically you could: 1. First get the size of the file on disk and approximate a character offset that should be only a few lines from the end of the file. 2. Then Open the file for read and read from file starting at the char offset you calculated to EOF into a var and delete the last line of the var. 3. Then Close the file and Open it again for write. 4. Now write the var to file at the same starting offset you read from. The reason you don't use open for update is that you want to open the file for write, so that whatever you write to the file replaces everything from the offset to the EOF. Update will replace chars and leave the last record at the end. What I don't know here is if internally the read/write from file at start still has to go through the entire file to calculate the starting offset, so I don't know how long this will take on a file of your size. But, at least, doing it this way won't require reading the whole file into memory and should result in the last line being deleted from the file in the quickest way available. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Jacque wrote: Worth a test anyway. I've used it on files that were several megs in size without a problem, but they weren't super huge, just kinda big. Hi Jacque, I made the suggestion I did because at 100 Million records plus, that's an awfully big read into memory just to delete the last line, so I figured it's better to work with the file on disk. I just remembered the params for open file wrong and Alex caught it. Aloha from Hawaii, Jim Bufalini ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
RE: Reading/Deleting Last Line Of File
Hey Richard, Well now we presumably have three different ways to do it. Sounds like a job for Mr. Gaskin to test which is the fastest. I'd offer to test but am plumb out of 100 MM record data files. LOL. Aloha from Hawaii, Jim Bufalini -Original Message- From: use-revolution-boun...@lists.runrev.com [mailto:use-revolution- boun...@lists.runrev.com] On Behalf Of Richard Gaskin Sent: Tuesday, February 09, 2010 2:10 PM To: How to use Revolution Subject: Re: Reading/Deleting Last Line Of File See why you shouldn't trust off-the-cuff code? Where I wrote: open file tFile for append ...I meant to write: open file tFile for update -- Richard Gaskin Fourth World Rev training and consulting: http://www.fourthworld.com Webzine for Rev developers: http://www.revjournal.com revJournal blog: http://revjournal.com/blog.irv ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
Re: Reading/Deleting Last Line Of File
Jim Bufalini wrote: Jacque wrote: Worth a test anyway. I've used it on files that were several megs in size without a problem, but they weren't super huge, just kinda big. Hi Jacque, I made the suggestion I did because at 100 Million records plus 100 million? Yes, well...I think I read the zeros wrong. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com ___ use-revolution mailing list use-revolution@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-revolution
