How about using python/perl/ruby, designed precisely for this type of
routine data munging, to pipe the processed output into an R dataframe?
msci <- read.table(pipe("python steve/python/msci.py"), header=T, as.is=T)
Iteratively, you could deliver the python output in chunks, something like:
msci <- read.table(pipe("python steve/python/msci.py 1 500000"), header=T,
as.is=T)
msci <- rbind(msci, read.table(pipe("python steve/python/msci.py 500001
1000000"), header=T, as.is=T))
etc.
Steve Miller
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Jason Barnhart
Sent: Wednesday, September 13, 2006 11:52 AM
To: Gabor Grothendieck; Anupam Tyagi
Cc: r-help@stat.math.ethz.ch
Subject: Re: [R] Reading fixed column format
Another possibility:
1) Split the original file into smaller chunks of xx,xxx of rows.
2) Process each file using read.fwf saving the requisite variables.
(If necessary, save each intermediate matrix/data.frame to disk
to conserve space)
3) 'rbind' the results.
Not exactly elegant but it works.
----- Original Message -----
From: "Gabor Grothendieck" <[EMAIL PROTECTED]>
To: "Anupam Tyagi" <[EMAIL PROTECTED]>
Cc: <r-help@stat.math.ethz.ch>
Sent: Wednesday, September 13, 2006 7:21 AM
Subject: Re: [R] Reading fixed column format
> On 9/13/06, Anupam Tyagi <[EMAIL PROTECTED]> wrote:
>> Gabor Grothendieck <ggrothendieck <at> gmail.com> writes:
>>
>> > C:\bin>cut -c2-3,6-8 a.dat
>> > 23678
>> > 23678
>> > 23678
>>
>> Thanks. I think this will work. How do I redirect the output to a file on
>> windows?
>
> Same as on UNIX
>
> cut -c2-3,6-8 a.dat > a2.dat
>
>> Is there simple way to convert the cut command to a script on windows,
>
> Using notepad or other text editor put it in file a.bat and then
> issue this command from the console
>
> a.bat
>
> Note that you could process it multiple time if you like:
>
> cut -c6-8 a.dat > a2.dat
> cut -c2-3 a2.dat > a3.dat
>
> produces the same thing but uses 2 passes and so keeps each line shorter.
> Be sure you do it from the tail end forward as shown above to avoid having
> to recalculate the positions.
>
>> because the entire command may not fit on one line? Anupam.
>>
>
> ______________________________________________
> R-help@stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[[alternative HTML version deleted]]
______________________________________________
R-help@stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
