Re: Vim freezes system ?!
I did th3 follwing: With a program, which generates random
numbers in different formats, I created a file, which consists
of _one_ line of 2097152 characters ("0"-"9","A"-"F").
To split the line into lines of 72 characters each, I started
vim and let it read the file.
I postioned the cursor at position 0 and entered the following
in normal mode:
qq72i0q
Then I did a
[EMAIL PROTECTED]
After only 10 or 15 (guessed) executions of the macro the
system freezes while constantly swapping (?) and became
unuseable and did no longer respond.
While I don't know what went hooey on your machine, a couple ideas:
1) turn off syntax highlighting. Vim seems to wheeze when
syntax-highlighting long lines.
2) To split the line like you describe, you might try
:s/.\{72}/&\r/g
(adjust the 72 for the number of characters you want on each
line). I don't know if it's any faster, but it can hardly be
worse ;)
3) you might try cranking back your 'undolevels'. I'm not sure
it would help a great deal, but if the other two don't have a
deep impact, it might help.
HTH,
-tim
Re: Vim freezes system ?!
On 4/6/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
Hi,
I did th3 follwing: With a program, which generates random numbers in
different formats, I created a file, which consists of _one_ line of
2097152 characters ("0"-"9","A"-"F").
To split the line into lines of 72 characters each, I started vim and
let it read the file.
I postioned the cursor at position 0 and entered the following in
normal mode:
qq72i0q
Then I did a
[EMAIL PROTECTED]
After only 10 or 15 (guessed) executions of the macro the system
freezes while constantly swapping (?) and became unuseable and did no
longer respond.
Even the mouse pointer was nearly unmoveable...
After heavily and constantly trying I managed to kill the X-session
and to 'killall -9 vim' from the console to get back my computer.
Hello Meino "the vim killer" Cramer,
I tried your scenario. You need to add 'set ul=-1' to disable undoes, and
'set lz' to disable excess redraws. Even then, vim goes rather slow at
this task.
Indeed, vim grows to >1000MB vm/rss size
size in matter of one minute without ul=-1 (, and growing very fast. )
To make editing of 2MB-long line manageable, I did:
vim -u NONE file
:set ul=-1 : disable undos
:set lz
NOte that 2MB is already "large file". You are now in the area where you want
take "large file" precautions wrt to performance.
This makes vim stay stable in memory size (although still rather large, 632mb).
But it's still slow at splitting 2mb-long lines. It took vim ~4 minutes to
split ~100 lines. So, for a whole 2mb data, it will take vim about
2 hours. You better try this kind of task with perl or C. Or just add
sensible newlines when generating your data.
Yakov
BTW your disk size is irrelevant.
What is relevant is your RAM size and swap size.
Re: Vim freezes system ?!
The problem I have with "my kind of splitting a line" is not that it does not work -- it is the "deadly side effect" it caused. My opinion is, that it should not be possible to "kill a system" (...too big words, I know, but...) by simply submitting a sub-optimal command to a text editor. As you killed it, there's no way to know if the process would have been seen to completion. *nix-like OSes shouldn't allow you to drag the system to its knees without affording other processes (such as X) timeslices of the CPU. You didn't mention how much memory your system has, but the specs are otherwise well within tolerances. I suspect you may have found a pessimal (opposite of "optimal") solution to your problem and that Vim+Gentoo would have happily carried out your instructions, even if it involved heaps of memory and hours of CPU burn. Additionally, as your line(s) got shorter, I suspect the time to process each subsequent line would diminish. So no, in general, it shouldn't "kill a system", but it sounded like your system was doing exactly what you asked of it. X responded (albeit slowly) and you were able to get to a console and start bustin' heads. That's kudos to an OS that can be functional even under heavy load. It also sounded like Vim was doing what you asked of it (though perhaps not optimally), which caused it chug on the CPU. It would be akin to writing a Python program that spun in a tight infinite loop: while True: pass and dragged your system to its knees...it's not Python's problem per-se, as it's your script/commands that are the problem. Thus, I recommend a "better" algorithm, such as a ":s" as suggested before. Just my perspective. -tim
Re: Vim freezes system ?!
On 4/6/07, Yakov Lerner <[EMAIL PROTECTED]> wrote: ... It took vim ~4 minutes to split ~100 lines. Typo here. This should have been: It took vim ~4 minutes to split ~1000 lines.
Re: Vim freezes system ?!
On 4/6/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: Hi Yakov "the vim healer" Lerner :) memory size is 2GB Dual channel RAM, swap is 1G... For me it is not a problem to fail with one task (splitting a long line)...there are always better solutions to discover on UNIX systems... :)) The problem I see is, that it is that easy to stop a system by submitting the "wrong command" to a text editor... Aha, so you didn't try to edit a file. You tried to find a scenario to push vim beyond it's and your system's memory capacities. OK, you succeeded. Yakov
Re: Vim freezes system ?!
On 4/6/07, Yakov Lerner <[EMAIL PROTECTED]> wrote:
On 4/6/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I did th3 follwing: With a program, which generates random numbers in
> different formats, I created a file, which consists of _one_ line of
> 2097152 characters ("0"-"9","A"-"F").
>
> To split the line into lines of 72 characters each, I started vim and
> let it read the file.
>
> I postioned the cursor at position 0 and entered the following in
> normal mode:
>
> qq72i0q
>
> Then I did a
>
> [EMAIL PROTECTED]
>
> After only 10 or 15 (guessed) executions of the macro the system
> freezes while constantly swapping (?) and became unuseable and did no
> longer respond.
>
> Even the mouse pointer was nearly unmoveable...
>
> After heavily and constantly trying I managed to kill the X-session
> and to 'killall -9 vim' from the console to get back my computer.
Hello Meino "the vim killer" Cramer,
I tried your scenario. You need to add 'set ul=-1' to disable undoes, and
'set lz' to disable excess redraws. Even then, vim goes rather slow at
this task.
Indeed, vim grows to >1000MB vm/rss size
size in matter of one minute without ul=-1 (, and growing very fast. )
The thing I find strange here is that values of 'maxmem', 'maxmemtot' were:
:set mm? mmt?
maxmem=643272
maxmemtot=643272
, yet vim grew past x2.5 times that limits (with default 'ul')
without messages.
Is this expected behaviour ?
Yakov
Re: Vim freezes system ?!
[EMAIL PROTECTED] wrote:
Hi,
I did th3 follwing: With a program, which generates random numbers in
different formats, I created a file, which consists of _one_ line of
2097152 characters ("0"-"9","A"-"F").
To split the line into lines of 72 characters each, I started vim and
let it read the file.
I postioned the cursor at position 0 and entered the following in
normal mode:
qq72i0q
Then I did a
[EMAIL PROTECTED]
After only 10 or 15 (guessed) executions of the macro the system
freezes while constantly swapping (?) and became unuseable and did no
longer respond.
Even the mouse pointer was nearly unmoveable...
After heavily and constantly trying I managed to kill the X-session
and to 'killall -9 vim' from the console to get back my computer.
I am using an up-to-date version of Gentoo and vim (not gvim). My
system runs an AMD 64 X2 3800+ CPU and uses a Seagate 200GB harddisk
(dma enabled). It needs a lot of load to bring the system to its
knees.
May be it was "wrong" what I did in the sense of "there are better
UNIX tools to reformat such a file" or "better commands in vim to
accomplish this",...
...but in a critical moment it may be that I would have lost my work
(other open applikations) due to the need of killing X.
Keep editing! ;)
mcc
Try Ctrl-Alt-F2 to get to a text console without killing X. Then enter a login
name and password.
On Linux, you normally have six text consoles just waiting there for you to
log in, even while X is running. Ctrl-Alt-F1 (/dev/tty1) normally has the last
page of your boot sequence log, so you may (or may not, after all) want to
leave that one in peace. Ctrl-Alt-F2 to Ctrl-Alt-F6 are the other five
(/dev/tty2 to /dev/tty6). /dev/tty7 is what the X server video output goes to
(and, IIUC, where its keyboard input comes from), so you hit Ctrl-Alt-F7 to go
back to X after having used a text terminal. (The X server display input is
normally called :0 ).
There are more possible terminals (/dev/tty8 to /dev/tty12, maybe), but since
there is no getty (or, more often, mingetty) program listening there, you
can't log into them. You _can_, if you wish, redirect a program's sysout
and/or syserr to them, and look at the latest page of output by means of the
appropriate Ctrl-Alt-Fn keychord. For instance, /dev/tty10 (Ctrl-Alt-F10) is
usually the syslog listing, at least in the Linux distros that I use.
Best regards,
Tony.
--
hundred-and-one symptoms of being an internet addict:
164. You got out to buy software, instead of going out for a beer.
Re: Vim freezes system ?!
Yakov Lerner wrote:
On 4/6/07, Yakov Lerner <[EMAIL PROTECTED]> wrote:
On 4/6/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I did th3 follwing: With a program, which generates random numbers in
> different formats, I created a file, which consists of _one_ line of
> 2097152 characters ("0"-"9","A"-"F").
>
> To split the line into lines of 72 characters each, I started vim and
> let it read the file.
>
> I postioned the cursor at position 0 and entered the following in
> normal mode:
>
> qq72i0q
>
> Then I did a
>
> [EMAIL PROTECTED]
>
> After only 10 or 15 (guessed) executions of the macro the system
> freezes while constantly swapping (?) and became unuseable and did no
> longer respond.
>
> Even the mouse pointer was nearly unmoveable...
>
> After heavily and constantly trying I managed to kill the X-session
> and to 'killall -9 vim' from the console to get back my computer.
Hello Meino "the vim killer" Cramer,
I tried your scenario. You need to add 'set ul=-1' to disable undoes, and
'set lz' to disable excess redraws. Even then, vim goes rather slow at
this task.
Indeed, vim grows to >1000MB vm/rss size
size in matter of one minute without ul=-1 (, and growing very fast. )
The thing I find strange here is that values of 'maxmem', 'maxmemtot' were:
:set mm? mmt?
maxmem=643272
maxmemtot=643272
, yet vim grew past x2.5 times that limits (with default 'ul')
without messages.
Is this expected behaviour ?
Yakov
I have
maxmapdepth=1000
maxmempattern=1000
maxmem=249
maxmemtot=249
which strikes me as strange (even though the last three are in KB) since I
never set them and the docs say maxmem is usually at least 256 and maxmemtot
at least 2048. And I don't have a tiny bitty box: my total installed RAM is 2 GB.
Best regards,
Tony.
--
"We don't care. We don't have to. We're the Phone Company."
RE: Vim freezes system ?!
> -Original Message-
> From: A.J.Mechelynck [mailto:[EMAIL PROTECTED]
> Sent: 07 April 2007 05:46
> To: Yakov Lerner
> Cc: vim@vim.org; Meino Christian Cramer; Bram Moolenaar
> Subject: Re: Vim freezes system ?!
>
> Yakov Lerner wrote:
> > On 4/6/07, Yakov Lerner <[EMAIL PROTECTED]> wrote:
> >> On 4/6/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> >> > Hi,
> >> >
> >> > I did th3 follwing: With a program, which generates random
> numbers in
> >> > different formats, I created a file, which consists of _one_ line
> of
> >> > 2097152 characters ("0"-"9","A"-"F").
> >> >
> >> > To split the line into lines of 72 characters each, I started vim
> and
> >> > let it read the file.
> >> >
> >> > I postioned the cursor at position 0 and entered the following in
> >> > normal mode:
> >> >
> >> > qq72i0q
> >> >
> >> > Then I did a
> >> >
> >> > [EMAIL PROTECTED]
> >> >
> >> > After only 10 or 15 (guessed) executions of the macro the system
> >> > freezes while constantly swapping (?) and became unuseable and
> did no
> >> > longer respond.
> >> >
> >> > Even the mouse pointer was nearly unmoveable...
> >> >
> >> > After heavily and constantly trying I managed to kill the X-
> session
> >> > and to 'killall -9 vim' from the console to get back my computer.
> >>
> >> Hello Meino "the vim killer" Cramer,
> >> I tried your scenario. You need to add 'set ul=-1' to disable
> undoes, and
> >> 'set lz' to disable excess redraws. Even then, vim goes rather slow
> at
> >> this task.
> >>
> >> Indeed, vim grows to >1000MB vm/rss size
> >> size in matter of one minute without ul=-1 (, and growing very fast.
> )
> >
> > The thing I find strange here is that values of 'maxmem', 'maxmemtot'
> were:
> >
> >:set mm? mmt?
> > maxmem=643272
> > maxmemtot=643272
> >
> > , yet vim grew past x2.5 times that limits (with default 'ul')
> > without messages.
> > Is this expected behaviour ?
> >
> > Yakov
> >
>
> I have
>maxmapdepth=1000
>maxmempattern=1000
>maxmem=249
>maxmemtot=249
>
> which strikes me as strange (even though the last three are in KB)
> since I
> never set them and the docs say maxmem is usually at least 256 and
> maxmemtot
> at least 2048. And I don't have a tiny bitty box: my total installed
> RAM is 2 GB.
Just to add more to the confusion, on my box (RedHat Enterprise Linux,
512Mb Ram) I have
maxmapdepth=1000
maxmempattern=1000
maxmem=257676
maxmemtot=257676
Which tells me the last two can hardly be in kb and all over I don't
understand what really they mean.
---Zdenek
smime.p7s
Description: S/MIME cryptographic signature
Re: Vim freezes system ?!
Zdenek Sekera wrote: [...] Just to add more to the confusion, on my box (RedHat Enterprise Linux, 512Mb Ram) I have maxmapdepth=1000 maxmempattern=1000 maxmem=257676 maxmemtot=257676 Which tells me the last two can hardly be in kb and all over I don't understand what really they mean. ---Zdenek According to the help, they _are_ in KB and can reach up to two million (i.e., 2 gig). I sure wonder according to what Vim sets themat startup. Why do you think they can't be in KB? The above would mean slightly over 256 meg, which sounds "reasonable" for a 512MB box (more reasonable at least, than the "249 K" which I see). Best regards, Tony. -- "Hit any key to continue" does _not_ mean you should hit the power switch.
RE: Vim freezes system ?!
> -Original Message- > From: A.J.Mechelynck [mailto:[EMAIL PROTECTED] > Sent: 10 April 2007 09:10 > To: Zdenek Sekera > Cc: Yakov Lerner; vim@vim.org; Meino Christian Cramer; Bram Moolenaar > Subject: Re: Vim freezes system ?! > > Zdenek Sekera wrote: > [...] > > Just to add more to the confusion, on my box (RedHat Enterprise > Linux, > > 512Mb Ram) I have > > > > maxmapdepth=1000 > > maxmempattern=1000 > > maxmem=257676 > > maxmemtot=257676 > > > > Which tells me the last two can hardly be in kb and all over I don't > > understand what really they mean. > > > > ---Zdenek > > > > > > According to the help, they _are_ in KB and can reach up to two million > (i.e., > 2 gig). I sure wonder according to what Vim sets themat startup. > > Why do you think they can't be in KB? The above would mean slightly > over 256 > meg, which sounds "reasonable" for a 512MB box (more reasonable at > least, than > the "249 K" which I see). Arrgh, I guess the batteries in my mental calculator are flat after Easter vac :-). But what do these numbers really mean? You are right that mine look more acceptable than yours. Yours are really strange. ---Zdenek smime.p7s Description: S/MIME cryptographic signature
Re: Vim freezes system ?!
Zdenek Sekera wrote: -Original Message- From: A.J.Mechelynck [mailto:[EMAIL PROTECTED] Sent: 10 April 2007 09:10 To: Zdenek Sekera Cc: Yakov Lerner; vim@vim.org; Meino Christian Cramer; Bram Moolenaar Subject: Re: Vim freezes system ?! Zdenek Sekera wrote: [...] Just to add more to the confusion, on my box (RedHat Enterprise Linux, 512Mb Ram) I have maxmapdepth=1000 maxmempattern=1000 maxmem=257676 maxmemtot=257676 Which tells me the last two can hardly be in kb and all over I don't understand what really they mean. ---Zdenek According to the help, they _are_ in KB and can reach up to two million (i.e., 2 gig). I sure wonder according to what Vim sets themat startup. Why do you think they can't be in KB? The above would mean slightly over 256 meg, which sounds "reasonable" for a 512MB box (more reasonable at least, than the "249 K" which I see). Arrgh, I guess the batteries in my mental calculator are flat after Easter vac :-). But what do these numbers really mean? You are right that mine look more acceptable than yours. Yours are really strange. ---Zdenek Maybe there has been an arithmetic wraparound in my case? I have 2 gig of RAM installed, which is slightly /more/ than 2 million KB. Best regards, Tony. -- Never be led astray onto the path of virtue.
Re: Vim freezes system ?!
On 4/10/07, A.J.Mechelynck <[EMAIL PROTECTED]> wrote: Zdenek Sekera wrote: > >> -Original Message- >> From: A.J.Mechelynck [mailto:[EMAIL PROTECTED] >> Sent: 10 April 2007 09:10 >> To: Zdenek Sekera >> Cc: Yakov Lerner; vim@vim.org; Meino Christian Cramer; Bram Moolenaar >> Subject: Re: Vim freezes system ?! >> >> Zdenek Sekera wrote: >> [...] >>> Just to add more to the confusion, on my box (RedHat Enterprise >> Linux, >>> 512Mb Ram) I have >>> >>> maxmapdepth=1000 >>> maxmempattern=1000 >>> maxmem=257676 >>> maxmemtot=257676 >>> >>> Which tells me the last two can hardly be in kb and all over I don't >>> understand what really they mean. >>> >>> ---Zdenek >>> >>> >> According to the help, they _are_ in KB and can reach up to two million >> (i.e., >> 2 gig). I sure wonder according to what Vim sets themat startup. >> >> Why do you think they can't be in KB? The above would mean slightly >> over 256 >> meg, which sounds "reasonable" for a 512MB box (more reasonable at >> least, than >> the "249 K" which I see). > > Arrgh, I guess the batteries in my mental calculator are flat after > Easter vac :-). > But what do these numbers really mean? You are right that mine look > more acceptable than yours. Yours are really strange. > > ---Zdenek Maybe there has been an arithmetic wraparound in my case? I have 2 gig of RAM installed, which is slightly /more/ than 2 million KB. I tried it on the machine with 8GB of RAM. The 'mmt' values are 1024. By the vimhelp, these values are in KB, as Tony said. Another strange thing is: these "low" values do not seem to have any effect on vim rmemory usage. Nor when they are "correct" (like 643272 on 1.3 G ram machine), neither when they are suspicously low (like 1024 on 8G ram machine). -- Yakov "look, these numbers are suspicously low" Lerner
Re: Vim freezes system ?!
Yakov Lerner wrote:
> On 4/6/07, Yakov Lerner <[EMAIL PROTECTED]> wrote:
> > On 4/6/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> > > Hi,
> > >
> > > I did th3 follwing: With a program, which generates random numbers in
> > > different formats, I created a file, which consists of _one_ line of
> > > 2097152 characters ("0"-"9","A"-"F").
> > >
> > > To split the line into lines of 72 characters each, I started vim and
> > > let it read the file.
> > >
> > > I postioned the cursor at position 0 and entered the following in
> > > normal mode:
> > >
> > > qq72i0q
> > >
> > > Then I did a
> > >
> > > [EMAIL PROTECTED]
> > >
> > > After only 10 or 15 (guessed) executions of the macro the system
> > > freezes while constantly swapping (?) and became unuseable and did no
> > > longer respond.
> > >
> > > Even the mouse pointer was nearly unmoveable...
> > >
> > > After heavily and constantly trying I managed to kill the X-session
> > > and to 'killall -9 vim' from the console to get back my computer.
> >
> > Hello Meino "the vim killer" Cramer,
> > I tried your scenario. You need to add 'set ul=-1' to disable undoes, and
> > 'set lz' to disable excess redraws. Even then, vim goes rather slow at
> > this task.
> >
> > Indeed, vim grows to >1000MB vm/rss size
> > size in matter of one minute without ul=-1 (, and growing very fast. )
>
> The thing I find strange here is that values of 'maxmem', 'maxmemtot' were:
>
> :set mm? mmt?
> maxmem=643272
> maxmemtot=643272
>
> , yet vim grew past x2.5 times that limits (with default 'ul')
> without messages.
> Is this expected behaviour ?
The 'maxmem' and 'maxmemtot' are for the text that is kept in memory.
Above these limits text is put in the swap file.
Text that is kept for undo is always in memory and there is no limit for
the amount, there is only 'undolevels'. See ":help limits".
I think the problem with the way lines are split here is that you go in
and out of Insert mode for every change. This causes the current line
to be saved for undo every time. With a function that gets the line and
inserts every 72 chararacters it's probably a lot faster.
--
It doesn't really matter what you are able to do if you don't do it.
(Bram Moolenaar)
/// Bram Moolenaar -- [EMAIL PROTECTED] -- http://www.Moolenaar.net \\\
///sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\download, build and distribute -- http://www.A-A-P.org///
\\\help me help AIDS victims -- http://ICCF-Holland.org///
Re: Vim freezes system ?!
I did th3 follwing: With a program, which generates random numbers in
different formats, I created a file, which consists of _one_ line of
2097152 characters ("0"-"9","A"-"F").
To split the line into lines of 72 characters each, I started vim and
let it read the file.
I postioned the cursor at position 0 and entered the following in
normal mode:
qq72i0q
Then I did a
[EMAIL PROTECTED]
After only 10 or 15 (guessed) executions of the macro the system
freezes while constantly swapping (?) and became unuseable and did no
longer respond.
>
> I think the problem with the way lines are split here is that you go in
> and out of Insert mode for every change. This causes the current line
> to be saved for undo every time. With a function that gets the line and
> inserts every 72 chararacters it's probably a lot faster.
Or even a quality :s command instead of a function:
:%s/.\{72}/&r/g
I suspect this one command will run rather quickly, not have to
guess at the number of iterations (1000 in the above example),
and won't thrash the undo history.
> It doesn't really matter what you are able to do if you don't do it.
> (Bram Moolenaar)
Thanks for the kick in the pants...I've got a pet project on the
back burner that just keep sitting there. It's got a lot of
potential, but if I don't do it, it doesn't matter how great the
project is :) Your motivation is much appreciated (whether you
intended it or not).
-tim
Re: Vim freezes system ?!
Tim Chase wrote:
Or even a quality :s command instead of a function:
:%s/.\{72}/&r/g
Thanks - I occasionally need to split a line into equal-length blocks
and your command is excellent. Your posts are very valuable.
However, for anyone trying this, 'r' should be '\r'.
:%s/.\{72}/&\r/g
John
Re: Vim freezes system ?!
>> Or even a quality :s command instead of a function:
>> :%s/.\{72}/&r/g
>
> Thanks - I occasionally need to split a line into equal-length blocks
> and your command is excellent. Your posts are very valuable.
>
> However, for anyone trying this, 'r' should be '\r'.
>
> :%s/.\{72}/&\r/g
Whooops...good catch. I got it right in my response to the OP,
but missed the "\" in my reply to Bram. If you end up with the
letter "r" every 72 characters instead of a newline, you used my
messed up version. :)
I used this regularly at my last job where I'd occasionally have
serial-data dumps and have to break the file into 128-byte
packet-sized pieces...invaluable to be able to have one command
that quickly sliced my file into 128-character lines. :)
-tim
