It easier than that. I forgot I can do it entirely within R:
setwd("/temp/csv")
files <- Sys.glob("daily*csv")
output <- file('Rcombined.csv', 'w')
for (i in files){
cat(i, '\n') # write out file processing
input <- readLines(i)
input <- input[-1L] # delete header
writeLines(input, output)
}
close(output)
On Sat, Nov 3, 2012 at 4:56 PM, jim holtman <jholt...@gmail.com> wrote:
> These are not commands, but programs you can use. Here is a file copy
> program in "perl" (I spelt it wrong in the email); This will copy all
> the files that have "daily" in their names. It also skips the first
> line of each file assuming that it is the header.
>
> perl can be found on most systems. www.activestate.com has a
> version that runs under Windows and that is what I am using.
>
>
> chdir "/temp/csv"; # my directory with files
> @files = glob "daily*csv"; # get files to copy (daily......csv)
> open OUTPUT, ">combined.csv"; # output file
> # loop for each file
> foreach $file (@files) {
> print $file, "\n"; # print file being processed
> open INPUT, "<" . $file;
> # assume that the first line is a header, so skip it
> $header = <INPUT>;
> @all = <INPUT>; # read rest of the file
> close INPUT;
> print OUTPUT @all; # append to the output
> }
> close OUTPUT;
>
> Here is what was printed on the console:
>
>
> C:\Users\Owner>perl copyFiles.pl
> daily.BO.csv
> daily.C.csv
> daily.CL.csv
> daily.CT.csv
> daily.GC.csv
> daily.HO.csv
> daily.KC.csv
> daily.LA.csv
> daily.LN.csv
> daily.LP.csv
> daily.LX.csv
> daily.NG.csv
> daily.S.csv
> daily.SB.csv
> daily.SI.csv
> daily.SM.csv
>
> Which was a list of all the files copied.
>
> On Sat, Nov 3, 2012 at 4:08 PM, Benjamin Caldwell
> <btcaldw...@berkeley.edu> wrote:
>> Jim,
>>
>> Where can I find documentation of the commands you mention?
>> Thanks
>>
>>
>>
>>
>>
>> On Sat, Nov 3, 2012 at 12:15 PM, jim holtman <jholt...@gmail.com> wrote:
>>>
>>> A faster way would be to use something like 'per', 'awk' or 'sed'.
>>> You can strip off the header line of each CSV (if it has one) and then
>>> concatenate the files together. This is very efficient use of memory
>>> since you are just reading one file at a time and then writing it out.
>>> Will probably be a lot faster since no conversions have to be done.
>>> Once you have the one large file, then you can play with it (load it
>>> if you have enough memory, or load it into a database).
>>>
>>> On Sat, Nov 3, 2012 at 11:37 AM, Jeff Newmiller
>>> <jdnew...@dcn.davis.ca.us> wrote:
>>> > On the absence of any data examples from you per the posting guidelines,
>>> > I will refer you to the help files for the melt function in the reshape2
>>> > package. Note that there can be various mixtures of wide versus long...
>>> > such as a wide file with one date column and columns representing all
>>> > stock
>>> > prices and all trade volumes. The longest format would be what melt gives
>>> > (date, column name, and value) but an in-between format would have one
>>> > distinct column each for dollar values and volume values with a column
>>> > indicating ticker label and of course another for date.
>>> >
>>> > If your csv files can be grouped according to those with similar column
>>> > "types", then as you read them in you can use cbind( csvlabel="somelabel",
>>> > csvdf) to distinguish it and then rbind those data frames together to
>>> > create
>>> > an intermediate-width data frame. When dealing with large amounts of data
>>> > you will want to minimize the amount of reshaping you do, but it would
>>> > require knowledge of your data and algorithms to say any more.
>>> >
>>> > ---------------------------------------------------------------------------
>>> > Jeff Newmiller The ..... ..... Go
>>> > Live...
>>> > DCN:<jdnew...@dcn.davis.ca.us> Basics: ##.#. ##.#. Live
>>> > Go...
>>> > Live: OO#.. Dead: OO#.. Playing
>>> > Research Engineer (Solar/Batteries O.O#. #.O#. with
>>> > /Software/Embedded Controllers) .OO#. .OO#.
>>> > rocks...1k
>>> >
>>> > ---------------------------------------------------------------------------
>>> > Sent from my phone. Please excuse my brevity.
>>> >
>>> > Benjamin Caldwell <btcaldw...@berkeley.edu> wrote:
>>> >
>>> >>Jeff,
>>> >>If you're willing to educate, I'd be happy to learn what wide vs long
>>> >>format means. I'll give rbind a shot in the meantime.
>>> >>Ben
>>> >>On Nov 2, 2012 4:31 PM, "Jeff Newmiller" <jdnew...@dcn.davis.ca.us>
>>> >>wrote:
>>> >>
>>> >>> I would first confirm that you need the data in wide format... many
>>> >>> algorithms are more efficient in long format anyway, and rbind is way
>>> >>more
>>> >>> efficient than merge.
>>> >>>
>>> >>> If you feel this is not negotiable, you may want to consider sqldf.
>>> >>Yes,
>>> >>> you need to learn a bit of SQL, but it is very well integrated into
>>> >>R.
>>> >>>
>>>
>>> >> >>---------------------------------------------------------------------------
>>> >>> Jeff Newmiller The ..... ..... Go
>>> >>Live...
>>> >>> DCN:<jdnew...@dcn.davis.ca.us> Basics: ##.#. ##.#. Live
>>> >>> Go...
>>> >>> Live: OO#.. Dead: OO#..
>>> >>Playing
>>> >>> Research Engineer (Solar/Batteries O.O#. #.O#. with
>>> >>> /Software/Embedded Controllers) .OO#. .OO#.
>>> >>rocks...1k
>>> >>>
>>>
>>> >> >>---------------------------------------------------------------------------
>>> >>> Sent from my phone. Please excuse my brevity.
>>> >>>
>>> >>> Benjamin Caldwell <btcaldw...@berkeley.edu> wrote:
>>> >>>
>>> >>> >Dear R help;
>>> >>> >I'm currently trying to combine a large number (about 30 x 30) of
>>> >>large
>>> >>> >.csvs together (each at least 10000 records). They are organized by
>>> >>> >plots,
>>> >>> >hence 30 X 30, with each group of csvs in a folder which corresponds
>>> >>to
>>> >>> >the
>>> >>> >plot. The unmerged csvs all have the same number of columns (5). The
>>> >>> >fifth
>>> >>> >column has a different name for each csv. The number of rows is
>>> >>> >different.
>>> >>> >
>>> >>> >The combined csvs are of course quite large, and the code I'm
>>> >>running
>>> >>> >is
>>> >>> >quite slow - I'm currently running it on a computer with 10 GB ram,
>>> >>> >ssd,
>>> >>> >and quad core 2.3 ghz processor; it's taken 8 hours and it's only
>>> >>75%
>>> >>> >of
>>> >>> >the way through (it's hung up on one of the largest data groupings
>>> >>now
>>> >>> >for
>>> >>> >an hour, and using 3.5 gigs of RAM.
>>> >>> >
>>> >>> >I know that R isn't the most efficient way of doing this, but I'm
>>> >>not
>>> >>> >familiar with sql or C. I wonder if anyone has suggestions for a
>>> >>> >different
>>> >>> >way to do this in the R environment. For instance, the key function
>>> >>now
>>> >>> >is
>>> >>> >merge, but I haven't tried join from the plyr package or rbind from
>>> >>> >base.
>>> >>> >I'm willing to provide a dropbox link to a couple of these files if
>>> >>> >you'd
>>> >>> >like to see the data. My code is as follows:
>>> >>> >
>>> >>> >
>>> >>> >#multmerge is based on code by Tony cookson,
>>> >>> >
>>> >>>
>>>
>>> >> >>http://www.r-bloggers.com/merging-multiple-data-files-into-one-data-frame/
>>> >>> ;
>>> >>> >The function takes a path. This path should be the name of a folder
>>> >>> >that
>>> >>> >contains all of the files you would like to read and merge together
>>> >>and
>>> >>> >only those files you would like to merge.
>>> >>> >
>>> >>> >multmerge = function(mypath){
>>> >>> >filenames=list.files(path=mypath, full.names=TRUE)
>>> >>> >datalist = try(lapply(filenames,
>>> >>> >function(x){read.csv(file=x,header=T)}))
>>> >>> >try(Reduce(function(x,y) {merge(x, y, all=TRUE)}, datalist))
>>> >>> >}
>>> >>> >
>>> >>> >#this function renames files using a fixed list and outputs a .csv
>>> >>> >
>>> >>> >merepk <- function (path, nf.name) {
>>> >>> >
>>> >>> >output<-multmerge(mypath=path)
>>> >>> >name <- list("x", "y", "z", "depth", "amplitude")
>>> >>> >try(names(output) <- name)
>>> >>> >
>>> >>> >write.csv(output, nf.name)
>>> >>> >}
>>> >>> >
>>> >>> >#assumes all folders are in the same directory, with nothing else
>>> >>there
>>> >>> >
>>> >>> >merge.by.folder <- function (folderpath){
>>> >>> >
>>> >>> >foldernames<-list.files(path=folderpath)
>>> >>> >n<- length(foldernames)
>>> >>> >setwd(folderpath)
>>> >>> >
>>> >>> >for (i in 1:n){
>>> >>> >path<-paste(folderpath,foldernames[i], sep="\\")
>>> >>> > nf.name <- as.character(paste(foldernames[i],".csv", sep=""))
>>> >>> >merepk (path,nf.name)
>>> >>> > }
>>> >>> >}
>>> >>> >
>>> >>> >folderpath <- "yourpath"
>>> >>> >
>>> >>> >merge.by.folder(folderpath)
>>> >>> >
>>> >>> >
>>> >>> >Thanks for looking, and happy friday!
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >*Ben Caldwell*
>>> >>> >
>>> >>> >PhD Candidate
>>> >>> >University of California, Berkeley
>>> >>> >
>>> >>> > [[alternative HTML version deleted]]
>>> >>> >
>>> >>> >______________________________________________
>>> >>> >R-help@r-project.org mailing list
>>> >>> >https://stat.ethz.ch/mailman/listinfo/r-help
>>> >>> >PLEASE do read the posting guide
>>> >>> >http://www.R-project.org/posting-guide.html
>>> >>> >and provide commented, minimal, self-contained, reproducible code.
>>> >>>
>>> >>>
>>> >
>>> > ______________________________________________
>>> > R-help@r-project.org mailing list
>>> > https://stat.ethz.ch/mailman/listinfo/r-help
>>> > PLEASE do read the posting guide
>>> > http://www.R-project.org/posting-guide.html
>>> > and provide commented, minimal, self-contained, reproducible code.
>>>
>>>
>>>
>>> --
>>> Jim Holtman
>>> Data Munger Guru
>>>
>>> What is the problem that you are trying to solve?
>>> Tell me what you want to do, not how you want to do it.
>>
>>
>
>
>
> --
> Jim Holtman
> Data Munger Guru
>
> What is the problem that you are trying to solve?
> Tell me what you want to do, not how you want to do it.
--
Jim Holtman
Data Munger Guru
What is the problem that you are trying to solve?
Tell me what you want to do, not how you want to do it.
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
