Re: A performance question
In that case, I'll have to thank Bram for fixing my problem before I even asked him to do so! Thanks Gary, when I get a chance I'll download vim 7. To those of you who provided links to work-around scripts etc., thank you for your help. If any of you are having trouble with large files I'd recommend doing what I _didn't_ do: make sure you're using the most recent version of vim before looking for other solutions. You may not need to reduce vim's capabilities in order to work with large files, either! Cheers, Max On Tue, 22 May 2007, Gary Johnson wrote: It turns out that this Red Hat installation also has vim 6.3.82 in /usr/bin/vim, so I tried that, too. /usr/bin/vim -u NONE two_million_lines 50% :.,$d 2 minutes 30 seconds! Eureka! According to the System Monitor CPU bar color, that was almost all User time, whereas with vim 7.1, it was a more balanced mix of User and Kernel time. (Kudos to Bram for such a performance improvement from vim 6 to 7!)
Re: A performance question
Hmm, interesting. I've noticed before that the CPU is pegged when I'm deleting, but I don't think my machine's behavior is due to CPU load; the machine has two CPUs, I'm typically the only (serious) user, as "top" has confirmed is the case now, and I get the same behavior whether I'm running another large job or not. My other large job takes about 1 Gb leaving almost 2 Gb of memory free, so I don't think I'm running out of physical memory, either. Given the difference between your results and mine, I finally checked my software versions, which are old: Red Hat 3.4.6, vim 6.3.82. Unfortunately I don't have permission to update this system, and the administrator hasn't been willing to do so in the past. I went looking for release notes for vim, but the announcements I found didn't go into detail about what bugs were fixed in which version. Can someone point me in the right direction? Thanks. --Max On Tue, 22 May 2007, Gary Johnson wrote: Now, does having the undo facility available _necessarily_ mean deleting a large chunk of a file takes so long, or can that be added to the list of desired performance enhancements? Not in my experience. In both experiments I reported earlier I hadn't done anything special with 'undolevels' and checking them now shows "undolevels=1000". I repeated the experiment on the Linux system staring vim as vim -u NONE two_million_lines ":.,$d" took 13 seconds. I did notice that the CPU was railed at 100% during that time, so loading of your CPU by other tasks may have an effect, as might the actual physical memory available to vim. ":set undolevels=-1" did reduce the time to 10 seconds. Regards, Gary -- Gary Johnson | Agilent Technologies [EMAIL PROTECTED] | Mobile Broadband Division | Spokane, Washington, USA
Re: A performance question
Thanks, Andy; the black hole register is a new idea to me. Unfortunately,
":.,$d _" to the black hole register appears to take the same amount of
time as ":.,$d" itself. "set undolevels=-1" speeds it up, but "set
undolevels=0" does not; this suggests to me that the problem isn't related
to how many undo buffers are around, just that the undo facility is
available at all.
Honestly, the 3 minutes it takes has to involve a significant amount of
waste, such as timing out for some system resource; reading the 2 million
line file into memory doesn't take that long in the first place, and the
delete is the first change I make to the file, so there isn't a stack of
buffers filled with undo information to start with.
Max
On Tue, 22 May 2007, Andy Wokula wrote:
A.J.Mechelynck schrieb:
Robert M Robinson wrote:
First, thanks very much for creating VIM! I have been using it on Linux
systems for years, and now use it via cygwin at home as well. I vastly
prefer VIM to EMACS, especially at home. I learned vi on a VAX/VMS
system long ago (a friend of mine had ported it), when our computer
science department was loading so many people on the VAXen that EDT was
rendered unusably slow. I still like VIM largely because I can do so
much with so little effort in so little time.
That brings me to my question. I have noticed that when editing large
files (millions of lines), deleting a large number of lines (say,
hundreds of thousands to millions) takes an unbelieveably long time in
VIM--at least on my systems. This struck me as so odd, I looked you up
(for the first time in all my years of use) so I could ask why!
Seriously, going to line 1 million of a 2 million line file and typing
the command ":.,$d" takes _minutes_ on my system (Red Hat Linux on a
2GHz Athlon processor (i686), 512kb cache, 3 Gb memory), far longer than
searching the entire 2 million line file for a single word
(":g/MyQueryName/p"). Doing it this way fits way better into my usual
workflow than using "head -n 100", because of course I'm using a
regular expression search to determine that I
want to truncate my file at line 100 in the first place.
I looked in the archive, and couldn't see that this issue had been
raised before. Is there any chance it can get added to the list of
performance enhancement requests?
Thanks,
Max Robinson, PhD
I think this is just "part of how Vim behaves".
When you edit a file, Vim holds the whole file in memory (IIUC). When you
delete a million lines, Vim frees (i.e., releases to the OS) the memory
those lines were using. That takes some time.
Best regards,
Tony.
What about the numbered registers?
:h "1
After freeing the lines, they are copied to "1 .
And the content of "1 is shifted to "2 (before, of course)
And so on, until register "9.
To avoid the copies, the blackhole register can be used:
:.,$d _
If there are copies, registeres can be cleared by hand:
:let @1 = ""
:let @2 = ""
...
:let @9 = ""
This also takes time, but frees the memory.
--
Regards,
Andy
Re: A performance question
":set undolevels=-1" caused my test to run in less than 15 sec, with no other options fiddled with. Thanks Tim, now I have a work-around! Now, does having the undo facility available _necessarily_ mean deleting a large chunk of a file takes so long, or can that be added to the list of desired performance enhancements? Max On Tue, 22 May 2007, Tim Chase wrote: The issue of editing large files comes up occasionally. A few settings can be tweaked to vastly improve performance. Notably, the 'undolevels' setting can be reduced to -1 or 0 for improved performance. If your lines are long, it can also help to disable syntax highlighting as well. You can drop in on one such thread here:
Re: A performance question
":set syntax?" replies "syntax=". I don't think it's syntax highlighting. I've used that with C and Prolog code before; I gave it up because it was too slow. I'm editing text output from one of my programs; truncating the output of a day-long run to match a run in progress for testing purposes, hunting down rare bugs. Max On Tue, 22 May 2007, Tim Chase wrote: Do you have syntax highlighting enabled? That can really slow vim down. Well, I don't mean to. ":set" says this: It can be toggled via :syntax on and :syntax off To see what flavor of syntax highlighting you currently have, you can query the 'syntax' setting: :set syntax? -tim
Re: A performance question
I just tried deleting 1133093 lines of a 1133093+1133409 line file, after typing ":syntax off". It took about 3 minutes. Max On Tue, 22 May 2007, Tim Chase wrote: Do you have syntax highlighting enabled? That can really slow vim down. Well, I don't mean to. ":set" says this: It can be toggled via :syntax on and :syntax off To see what flavor of syntax highlighting you currently have, you can query the 'syntax' setting: :set syntax? -tim
Re: A performance question
Well, I don't mean to. ":set" says this: -- autoindent helplang=en scroll=11 t_Sb=Esc[4%dm backspace=2 history=50 ttyfast t_Sf=Esc[3%dm cscopetag hlsearchttymouse=xterm cscopeverbose ruler viminfo='20,"50 cscopeprg=/usr/bin/cscope fileencoding=utf-8 fileendcodings=utf-8,latin1 formatoptions=tcql -- So, do I have syntax highlighting enabled? The t_Sb and t_Sf look suspiciously like formatting commands, but I confess I'm not conversant on vim options. Thanks, Max On Tue, 22 May 2007, Gary Johnson wrote: Do you have syntax highlighting enabled? That can really slow vim down. I created and opened a file as follows: while true do echo '123456789012345678901234567890123456789012345678901234567890' done | head -200 > two_million_lines vim two_million_lines Then within vim executed: 50% :.,$d Using vim 7.1 under Cygwin and Windows XP on a 3.6 GHz Pentium with 2 GB of RAM: 9 seconds. Using vim 7.1 under Red Hat Enterprise Linux WS release 4 on a 2.8 GHz Pentium with 500 MB RAM: 16 seconds. Regards, Gary -- Gary Johnson | Agilent Technologies [EMAIL PROTECTED] | Mobile Broadband Division | Spokane, Washington, USA
Re: A performance question
Thanks, Tim. I'll look at the options you recommended--and those you didn't, so I may not need to ask next time. :) Cheers, Max On Tue, 22 May 2007, Tim Chase wrote: The issue of editing large files comes up occasionally. A few settings can be tweaked to vastly improve performance. Notably, the 'undolevels' setting can be reduced to -1 or 0 for improved performance. If your lines are long, it can also help to disable syntax highlighting as well. You can drop in on one such thread here: http://www.nabble.com/Re%3A-editing-large-file-p3665161.html or the associated vim-tip at http://www.vim.org/tips/tip.php?tip_id=611 Another option might be to use a stream-oriented tool such as sed to edit your file: sed '10q' < infile.txt > outfile.txt Fortunately, Vim has oodles of knobs to twiddle, so you can monkey with 'undolevels', 'swapfile', and the 'bufhidden' setting, as well as turning off sytnax highlighting, all of which can improve the performance of vim under uncommon load. This struck me as so odd, I looked you up (for the first time in all my years of use) so I could ask why! Welcome aboard...the list is friendly, informative, on-topic, and an all-round example of what a mailing-list should be. :) -tim
