Thanks for all the responses. sed did the trick in about 10 minutes (on a multi-threaded dual Xeon box w/ 2GB of RAM [though it probably only used a single thread]).
It is an interesting problem, and it seems like there would be a better solution widely available. I see how the solution could be a part of the file system; does anyone know if any available file system can do this? Perhaps Sun's self-proclaimed glorious ZFS? Also, FYI, it's not a flat file database, but it is a database dump that I need to import, and the first 300 lines are SQL commands to create tables that I already have, hence the desire to remove the lines. Thanks again, Grant On 6/28/06, James Washer <[EMAIL PROTECTED]> wrote:
whoops, my first response went direct to Brian, so I'll copy it here: debugfs? How's that going to help if after chopping off the first 300 lines, the data is not block aligned? You can debugfs all night long and you'll not get around that problem. IMPOSSIBLE I say! - jim On Wed, 28 Jun 2006 18:41:44 -0700 Brian Chrisman <[EMAIL PROTECTED]> wrote: > Hey.. not 'impossible'.. just *really annoying*... ie, firing up debugfs > or whatever they're using these days to manipulate filesystem internals. :) > Sorry, but the impossible in all caps is just begging for > technicalities.. :-) > > -Brian > > James Washer wrote: > > >If he wanted to lop off the first xxx bytes, without "touching" the remainder of the file... then it's IMPOSSIBLE under linux/unix. > > > >You mention flat-file-database... it's that's the case, I hope he can stop updates while removing the first 300 lines... else, the problem is just a bit harder (in fact, it can become impossible if his data base engine does not use some form of file locking.) > > - jim > > > >On Wed, 28 Jun 2006 17:50:14 -0700 (PDT) > >Sebastian Smith <[EMAIL PROTECTED]> wrote: > > > > > > > >>I think what Grant meant by "in place" was that he didn't want to read the > >>entire file, just lop off the first 300 lines -- perhaps he can correct > >>me. I'm guessing storage space isn't an issue. Regardless, I don't know > >>of a way to do that. > >> > >>You really need to get away from those flat file databases Grant ;) > >> > >> > >>On Wed, 28 Jun 2006, James Washer wrote: > >> > >> > >> > >>>it's the "working in place" that makes this difficult, else there are countless ways to do this, including the simple perl > >>> > >>> perl -ne 'next unless $. >300;print' > >>> > >>>On Wed, 28 Jun 2006 17:26:47 -0700 > >>>"Brandon Mitchell" <[EMAIL PROTECTED]> wrote: > >>> > >>> > >>> > >>>>This is an interesting problem, and seems to be revealing a little bit > >>>>about what type of user/admin is in each of us. > >>>> > >>>>So where's Nick with an Emacs Lisp macro for this task? :P > >>>> > >>>>On 6/28/06, Anna <[EMAIL PROTECTED]> wrote: > >>>> > >>>> > >>>>>find out which byte terminates the first 300 lines. maybe... > >>>>> > >>>>>~$ BYTES=$(head -300 nameofbigfile.txt | wc -c) > >>>>> > >>>>>then use that info with dd skip the first part of the input file... > >>>>> > >>>>>~$ dd if=nameofbigfile.txt of=truncatedversion.pl ibs=$BYTES skip=1 > >>>>> > >>>>>one of many ways, I'm sure. I think this way should be pretty fast > >>>>>because it works on a line by line basis for just a small part of the > >>>>>file. The rest, with dd, is done in larger pieces. > >>>>> > >>>>>- Anna > >>>>> > >>>>> > >>>>>On Wed, Jun 28, 2006 at 01:01:03PM -0700, Grant Kelly wrote: > >>>>> > >>>>> > >>>>>>Alright unix fans, who can answer this the best? > >>>>>> > >>>>>>I have a text file, it's about 2.3 GB. I need to delete the first 300 > >>>>>>lines, and I don't want to have to load the entire thing into an > >>>>>>editor. > >>>>>> > >>>>>>I'm trying `sed '1,300d' inputfile > output file` but it's taking a > >>>>>>long time (and space) to output everything to the new file. > >>>>>> > >>>>>>There has got to be a better way, a way that can do this in-place... > >>>>>> > >>>>>> > >>>>>>Grant > >>>>>> > >>>>>>_______________________________________________ > >>>>>>RLUG mailing list > >>>>>>[email protected] > >>>>>>http://lists.rlug.org/mailman/listinfo/rlug > >>>>>> > >>>>>> > >>>>>_______________________________________________ > >>>>>RLUG mailing list > >>>>>[email protected] > >>>>>http://lists.rlug.org/mailman/listinfo/rlug > >>>>> > >>>>> > >>>>> > >>>>-- > >>>>If UNIX doesn't have the solution you have the wrong problem. > >>>>UNIX is simple, but it takes a genius to understand it's simplicity. > >>>> > >>>>_______________________________________________ > >>>>RLUG mailing list > >>>>[email protected] > >>>>http://lists.rlug.org/mailman/listinfo/rlug > >>>> > >>>> > >>>> > >>>_______________________________________________ > >>>RLUG mailing list > >>>[email protected] > >>>http://lists.rlug.org/mailman/listinfo/rlug > >>> > >>> > >>> > >>_______________________________________________ > >>RLUG mailing list > >>[email protected] > >>http://lists.rlug.org/mailman/listinfo/rlug > >> > >> > >> > > > >_______________________________________________ > >RLUG mailing list > >[email protected] > >http://lists.rlug.org/mailman/listinfo/rlug > > > > > > > _______________________________________________ > RLUG mailing list > [email protected] > http://lists.rlug.org/mailman/listinfo/rlug > _______________________________________________ RLUG mailing list [email protected] http://lists.rlug.org/mailman/listinfo/rlug
_______________________________________________ RLUG mailing list [email protected] http://lists.rlug.org/mailman/listinfo/rlug
