RE: Long text file to edit.. (fwd)

[Carlos Marcos Kakihara]

The file is an ascii text dumped from a database, and
it have something about 3 million of registers. I have 1GB of
RAM and 24GB of disk space in a Sun, and I'll try to use vim
and jed to edit it.
If it doesn't work, I'll try to split it.

Thank's for the answers!

-- Forwarded message --
Date: Sun, 19 Jul 1998 00:23:59 +0100 (BST)
From: [EMAIL PROTECTED]
To: Carlos Marcos Kakihara <[EMAIL PROTECTED]>
Cc: Debian List 
Subject: RE: Long text file to edit..

On 17-Jul-98 Carlos Marcos Kakihara wrote:
>   I want to edit a 700MB text file. vi tells that the file is
> too long, and xemacs tells that "maximum buffer size something.." :)
>   There is a way to view this file?

Ehh?? Are you sure it's 700MB?? And are you sure it's a pure plain text file?
And, if so, have you thought about what that corresponds to? (And the same
question to the other contributors to this thread).

The average page of a typical book contains about 2K characters (or 8K for a
very big page in small type like old editions of the Encyclopaedia Britannica).
So you're thinking of !editing! the equivalent of 350K normal pages, i.e. about
the content of about 700 thickish paperbacks or about 100 volumes like the E.B.
???

Doing this by eye and hand, or even merely reading it, is not a realistic
project, I think, unless you have several years ahead of you. At <= 80
char/line, there are >= 9 million lines. I just timed a "cat" of 8k lines at 22
seconds. Just to "cat" such a file would take over 22000 seconds, or > 6 hours.
"There is a way to view this file?" you ask: even if you can scan it at 0.1
seconds per line, you are in for 1,000,000 seconds or about 30 9-hour days;
at 2 secs/line, it would take about 2 years, or more.

What sort of file is this?

Maybe it is an ASCII dump of a database, and you need to make a few known
changes to it. In that case, try awk or perl and pipe the file through a script.
It may take many hours to work from one end to the other, but at least that is
a practical proposition, allowing you to get on with something else and keep
your sanity. Assuming you have the disk space, of course.

If you need to reflect on each line before deciding what to do about it, then
surely there is no realistic prospect of getting the job done. The above
questions need an answer before any sensible advice can be given about how to
handle such a file.

Good luck, and best wishes,
Ted.


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

RE: Long text file to edit..

[Nelson Posse Lago]

On Sun, 19 Jul 1998 [EMAIL PROTECTED] wrote:

> On 17-Jul-98 Carlos Marcos Kakihara wrote:
> >   I want to edit a 700MB text file.
> 
> Ehh?? Are you sure it's 700MB?? And are you sure it's a pure plain text file?
> And, if so, have you thought about what that corresponds to?

How about this possibility: it's a PostScript file for a Imagesetter
(700Mb is not uncommon in this scenario). For some reason, you want to
check the paths of the EPS files that were embedded into it, and possibly
change some of them (say, for OPI post-processing). Sure you may grep
away, find the ones you need to change and write a script, but it *should*
be easier to open the file with an editor, search for the relevant lines  
and, on some of them, make a correction.

See ya,
Nelson
[EMAIL PROTECTED]
That's Internet!

"I'm a bitch, I'm a lover,
I'm a child, I'm a mother, 
I'm a sinner, I'm a saint..." - Gee, where is that woman!!


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

Re: Long text file to edit..

[Alexander]

Hi...

The problem with beav is it's mostly built as a hex editor. Of course, I
haven't used it or any emacs-style editor much. They're so, ugh,
unfriendly! :) (ok, ok, I'm a weenie, I admit it.)

Alex

On Fri, 17 Jul 1998, Nelson Posse Lago wrote:

> Date: Fri, 17 Jul 1998 12:11:13 -0300 (EST)
> From: Nelson Posse Lago <[EMAIL PROTECTED]>
> To: Debian List 
> Subject: Re: Long text file to edit..
> Resent-Date: 17 Jul 1998 15:12:31 -
> Resent-From: debian-user@lists.debian.org
> Resent-cc: recipient list not shown: ;
> 
> On Fri, 17 Jul 1998, Carlos Marcos Kakihara wrote:
> 
> > I want to edit a 700MB text file. vi tells that the file is
> > too long, and xemacs tells that "maximum buffer size something.." :)
> > There is a way to view this file?
> 
> I never tried, but you may want to check the "beav" or the "le" editors;
> the descriptions say they should be able to do that.
> 
> See ya,
> Nelson
> [EMAIL PROTECTED]
> That's Internet!
> 
> "The future? I've seen the future: It's a 46 years old
> virgin singing 'I am a hotdog'" - From "Demolition Man"
> 
> 
> --  
> Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null
> 
> 



--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

Re: Long text file to edit..

[E.L. Meijer \(Eric\)]

> 
> On Sun, Jul 19, 1998 at 12:23:59AM +0100, Ted Harding wrote:
> > On 17-Jul-98 Carlos Marcos Kakihara wrote:
> > >   I want to edit a 700MB text file. vi tells that the file is
> > > too long, and xemacs tells that "maximum buffer size something.." :)
> > >   There is a way to view this file?
> > 
> > Ehh?? Are you sure it's 700MB?? And are you sure it's a pure plain text 
> > file?
> > And, if so, have you thought about what that corresponds to? (And the same
> > question to the other contributors to this thread).
> 
> Well, maybe not to actually edit, but certainly to view and search. A log
> file with a high debug level for example, used to catch an occasional and
> unpredictable oops that may happen once only every few days.

If this is the case you better cut the file in pieces while it is
produced, and throw away bits that contain no oopses.  Anyway, the ideal
tool to handle this would be grep:

grep oops huge_file   # prints all lines with `oops'
grep -10 oops huge_file   # prints all lines with `oops', and 10 lines
  # before and after each match
grep -A 10 oops huge_file # prints all lines with `oops', and 10 lines
  # after each match
grep -B 10 oops huge_file # prints all lines with `oops', and 10 lines
  # before each match

HTH,
Eric

-- 
 E.L. Meijer ([EMAIL PROTECTED])  | tel. office +31 40 2472189
 Eindhoven Univ. of Technology | tel. lab.   +31 40 2475032
 Lab. for Catalysis and Inorg. Chem. (TAK) | tel. fax+31 40 2455054


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

Re: Long text file to edit..

[Ted Harding]

On 19-Jul-98 Pann McCuaig wrote:
> On Sun, Jul 19, 1998 at 12:23:59AM +0100, Ted Harding wrote:
>> On 17-Jul-98 Carlos Marcos Kakihara wrote:
>> >   I want to edit a 700MB text file. vi tells that the file is
>> > too long, and xemacs tells that "maximum buffer size something.." :)
>> >   There is a way to view this file?
>> 
>> Ehh?? Are you sure it's 700MB?? And are you sure it's a pure plain text
>> file?
>> And, if so, have you thought about what that corresponds to? (And the same
>> question to the other contributors to this thread).
> 
> Well, maybe not to actually edit, but certainly to view and search. A log
> file with a high debug level for example, used to catch an occasional and
> unpredictable oops that may happen once only every few days.

I wasn't saying a 700MB file was impossible nor that you wouldn't need to
search it for some interesting item.

I WAS demonstrating that there's no practical hope of doing such a "view and
search" job by eye and hand.

If you're searching, and know what sort of thing you're looking for, then you
might hand it over to grep, maybe "with context" as in e.g. "grep -i -n -5 ..."
which would do a case-insensitive search for a pattern, prefix each output line
with the line number in the original file, and also output the five lines
preceding and the five lines following each line matching the pattern. Redirect
this output into a file and then, if the "unpredictable oops" is sufficiently
rare, you should have a file with not too many KB in it that you can handle by
conventional means ("vim", "less", ... ).

(Note that if you have a comprehensive list of "non-oops"es, e.g. predictable
debug stuff reporting satisfactory working which you expect to get, then you
can reverse the grep so as to output NON-matching lines with the option -v; and
you can take the pattern -- which in this case might be complex -- from a file
by using the "-f filename" option. If the task is like this, then "man grep" is
a good starting point.)

But if you don't know what sort of thing you're looking for, then you haven't a
hope. It would, literally and without exaggeration, take YEARS. Surely that is
clear!

Best wishes,
Ted.


E-Mail: (Ted Harding) <[EMAIL PROTECTED]>
Date: 19-Jul-98   Time: 09:53:31



--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

Re: Long text file to edit..

[Pann McCuaig]

On Sun, Jul 19, 1998 at 12:23:59AM +0100, Ted Harding wrote:
> On 17-Jul-98 Carlos Marcos Kakihara wrote:
> >   I want to edit a 700MB text file. vi tells that the file is
> > too long, and xemacs tells that "maximum buffer size something.." :)
> >   There is a way to view this file?
> 
> Ehh?? Are you sure it's 700MB?? And are you sure it's a pure plain text file?
> And, if so, have you thought about what that corresponds to? (And the same
> question to the other contributors to this thread).

Well, maybe not to actually edit, but certainly to view and search. A log
file with a high debug level for example, used to catch an occasional and
unpredictable oops that may happen once only every few days.

Cheers,
 Pann


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

RE: Long text file to edit..

[Ted Harding]

On 17-Jul-98 Carlos Marcos Kakihara wrote:
>   I want to edit a 700MB text file. vi tells that the file is
> too long, and xemacs tells that "maximum buffer size something.." :)
>   There is a way to view this file?

Ehh?? Are you sure it's 700MB?? And are you sure it's a pure plain text file?
And, if so, have you thought about what that corresponds to? (And the same
question to the other contributors to this thread).

The average page of a typical book contains about 2K characters (or 8K for a
very big page in small type like old editions of the Encyclopaedia Britannica).
So you're thinking of !editing! the equivalent of 350K normal pages, i.e. about
the content of about 700 thickish paperbacks or about 100 volumes like the E.B.
???

Doing this by eye and hand, or even merely reading it, is not a realistic
project, I think, unless you have several years ahead of you. At <= 80
char/line, there are >= 9 million lines. I just timed a "cat" of 8k lines at 22
seconds. Just to "cat" such a file would take over 22000 seconds, or > 6 hours.
"There is a way to view this file?" you ask: even if you can scan it at 0.1
seconds per line, you are in for 1,000,000 seconds or about 30 9-hour days;
at 2 secs/line, it would take about 2 years, or more.

What sort of file is this?

Maybe it is an ASCII dump of a database, and you need to make a few known
changes to it. In that case, try awk or perl and pipe the file through a script.
It may take many hours to work from one end to the other, but at least that is
a practical proposition, allowing you to get on with something else and keep
your sanity. Assuming you have the disk space, of course.

If you need to reflect on each line before deciding what to do about it, then
surely there is no realistic prospect of getting the job done. The above
questions need an answer before any sensible advice can be given about how to
handle such a file.

Good luck, and best wishes,
Ted.


E-Mail: (Ted Harding) <[EMAIL PROTECTED]>
Date: 19-Jul-98   Time: 00:23:59



--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

Re: Long text file to edit..

[Dave Jones]

Try "vim" maybe?

from 

"Vim allows you to edit *all* files, and yet recognizes when you do not
have the permission to edit a file,
showing "[RO]" for "read-only" files. [980101]. "

Have never tried a file that big myself.

Go to  and look for version 5.1

Good luck!
Dave Jones




--
> From: Carlos Marcos Kakihara <[EMAIL PROTECTED]>
> To: Debian List 
> Subject: Long text file to edit..
> Date: Friday, July 17, 1998 9:46 AM
> 
> 
>   I want to edit a 700MB text file. vi tells that the file is
> too long, and xemacs tells that "maximum buffer size something.." :)
>   There is a way to view this file?
> 
> Carlos Marcos Kakihara (bacate)
> 
> Escola de Engenharia de Piracicaba (EEP)
> Departamento de Informatica
> 
> e-mail:
> 
> [EMAIL PROTECTED]
> [EMAIL PROTECTED]
> 
> 
> --  
> Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] <
/dev/null
> 


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

Re: Long text file to edit..

[Keith Beattie]

Carlos Marcos Kakihara wrote:
> 
>   I want to edit a 700MB text file. vi tells that the file is
> too long, and xemacs tells that "maximum buffer size something.." :)
>   There is a way to view this file?
> 

If you simply want to view the file and not edit it, I'd try something
like "more" which I believe just starts reading the file from the
beginning and doesn't care how big it is in total size.

If you want to edit it, and it is just text, what about splitting
it up into more manageable chunks using something like head or tail?

This could actually get quite tricky if you don't have enough space to
hold each peice, which is probably why vi and xemacs are complaining
in the first place!

Just some thoughts...
Keith


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

Re: Long text file to edit..

[Nelson Posse Lago]

On Fri, 17 Jul 1998, Carlos Marcos Kakihara wrote:

>   I want to edit a 700MB text file. vi tells that the file is
> too long, and xemacs tells that "maximum buffer size something.." :)
>   There is a way to view this file?

I never tried, but you may want to check the "beav" or the "le" editors;
the descriptions say they should be able to do that.

See ya,
Nelson
[EMAIL PROTECTED]
That's Internet!

"The future? I've seen the future: It's a 46 years old
virgin singing 'I am a hotdog'" - From "Demolition Man"


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null

RE: Long text file to edit..

[Lewis, James M.]

If vi said lines were too long, then you can not use vi to see all of the file.
If vi ran out of tmp space, then try setting TMPDIR env var to point to a place
that has up to ?? amount of space.  I can't remember if vi needs double the
original file space or not.  That would mean either 700M or 1.4G...  Be ready
to wait a while for vi to bring up the file.
  
The same trick might work for emacs but I don't
know much about the limitations of emacs (I haven't hit any yet).

jim

--
From:   Carlos Marcos Kakihara[SMTP:[EMAIL PROTECTED]
Sent:   Friday, July 17, 1998 10:46 AM
To: Debian List
Cc: The recipient's address is unknown.
Subject:Long text file to edit..


I want to edit a 700MB text file. vi tells that the file is
too long, and xemacs tells that "maximum buffer size something.." :)
There is a way to view this file?

Carlos Marcos Kakihara (bacate)

Escola de Engenharia de Piracicaba (EEP)
Departamento de Informatica

e-mail:

[EMAIL PROTECTED]
[EMAIL PROTECTED]


--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null



--  
Unsubscribe?  mail -s unsubscribe [EMAIL PROTECTED] < /dev/null