Oblivious to the problems that Barry notes, I have used pdftotext,
from Xpdf at http://www.foolabs.com/xpdf/download.html
without apparent problem; this under MacOS X. For my purposes,
I need to retain the CTRL/Fs that indicate page throws. Other
converters that I have investigate seem not to retain such
information. I use it with command line options thus:
pdftotext -layout -eol unix rnotes.pdf rnotes.txt
Given a pdf for a manuscript, I have an R function that can then
create an index of functions that appear with their opening and
closing parentheses. (It may pick up a few strays, which can be
weeded out.)
The only issue I've found has been for use with the listings package.
In the LaTeX source, I set the option 'columns' to 'fullflexible', in order
to generate a pdf file that does not have the unwanted hidden spaces,
which are then carried across to the text file.
{\lstset{language=R, xleftmargin=2pt,
basicstyle=\ttfamily,
columns=fullflexible, % Omit for final manuscript
showstringspaces=false}}
{}
The setting 'columns=fullflexible' messes up the formatting
in places where I use tabbing, so that I need to change
'columns' back to the default for the final pdf.
Here is the function that does most of the work. Like the function
that follows, it could no doubt be greatly tidied up:
locatefun <-
function(txlines){
idtxt <- "\\.?[a-zA-Z][a-zA-Z0-9]*(\\.[a-zA-Z]+)*\\("
z <- regexpr("\014", txlines)
z[z>0] <- 1
z[z<=0] <- 0
page <- cumsum(z)+1
k <- 0
findfun <- function(tx){
mn <- t(sapply(tx, function(x)
{m <- regexpr(idtxt, x);
c(m, attr(m, "match.length"))}))
mn[,2] <- mn[,1]+mn[,2]
rownames(mn) <- paste(1:dim(mn)[1])
mn
}
for(i in 1:100){
mn <- findfun(txlines)
if(all(mn[,1]==-1))break
here <- mn[,1]>0
page <- page[here]
txlines <- txlines[here]
mn <- mn[here, , drop=FALSE]
m1 <- regexpr("\\(", txlines)-1
tx1 <- substring(txlines,mn[,1],m1)
if(i==1)xy <- data.frame(nam=I(tx1), page=page) else
xy <- rbind(xy, data.frame(nam=I(tx1), page=page))
txlines <- substring(txlines,mn[,2])
here2 <- nchar(txlines)>0
txlines <- txlines[here2]
page <- page[here2]
if(length(txlines)==0)break
}
zz <- !xy[,1]%in% c("","al","Pr","T", "F", "n","P", "y", "A",
"transformation", "left","f","site.","a","b", "II",
"ARCH", "ARMA", "MA")
xy <- xy[zz,]
nam <- xy$nam
ch <- substring(nam,1,1)
nam[ch%in%c("="," ",",")] <- substring(nam[ch%in%c("="," ",",")],2)
xy$nam <- nam
ord <- order(xy[,2])
xy[ord,]
}
Here is the function that calls findfun:
makeFunIndex <-
function(sourceFile="rnotes.txt",
frompath="~/_notes/rnotes/", fileout=NULL,
availfun=funpack,
offset=0){
## pdftotext -layout -eol unix rnotes.pdf rnotes.txt
len <- nchar(sourceFile)
lfrom <- nchar(frompath)
if(substring(frompath, lfrom, lfrom)=="/")frompath <-
substring(frompath, 1, lfrom-1)
if(is.null(fileout)){
if (substring(sourceFile,len - 3, len-3) == ".")
fnam <- substring(sourceFile, 1, len - 4) else fnam <-
sourceFile
fileout <- paste(fnam, ".fdx", sep = "")
fdxfile <- paste(fileout, sep="/")
fndfile <- paste(fnam, ".fnd", sep = "")
}
sourceFile <- paste(frompath, sourceFile, sep="/")
print(paste("Send output to", fndfile))
tx <- readLines(sourceFile, warn=FALSE)
entrymat <- locatefun(tx)
backn <- regexpr("\\n",entrymat[,1],fixed=TRUE)
entrymat <- entrymat[backn < 0,]
entrymat[,2] <- entrymat[,2] - offset
entrymat[,1] <- gsub("_","\\_",entrymat[,1], fixed=TRUE)
nmatch <- match(entrymat[,1], availfun[,2], nomatch=0)
use <- nmatch > 0
print("Unmatched functions:")
print(unique(entrymat[!use,1]))
entrymat[use,1] <- paste(entrymat[use,1], " ({\\em ",
availfun[nmatch,1], "})", sep="")
funentries <- paste("\\indexentry ", "{", entrymat[,1],"}{",
entrymat[,2], "}",sep="")
write(funentries, fdxfile)
system(paste("makeindex -o", fndfile, fdxfile))
}
John Maindonald email: john.maindon...@anu.edu.au
phone : +61 2 (6125)3473 fax : +61 2(6125)5549
Centre for Mathematics & Its Applications, Room 1194,
John Dedman Mathematical Sciences Building (Building 27)
Australian National University, Canberra ACT 0200.
http://www.maths.anu.edu.au/~johnm
On 10/01/2010, at 10:00 PM, r-help-requ...@r-project.org wrote:
> From: Barry Rowlingson <b.rowling...@lancaster.ac.uk>
> Date: 10 January 2010 12:47:01 AM AEDT
> To: David Kane <d...@kanecap.com>
> Cc: r-help@r-project.org
> Subject: Re: [R] parsing pdf files
>
>
> On Sat, Jan 9, 2010 at 1:11 PM, David Kane <d...@kanecap.com> wrote:
>> I have a pdf file that I would like to parse into R:
>>
>> http://www.williams.edu/Registrar/geninfo/faculty.pdf
>>
>> For now, I open the file in Acrobat by hand, then save it "as text"
>> and then use readLines(). That works fine but a) I am concerned that
>> some information may be lost and b) I may be doing this a lot, so I
>> would rather have R grab the information from the pdf file directly.
>>
>> So: is there something like readPDF() for R?
>
> What could it do that saving as text from Acrobat couldn't do? Here's
> the problem - PDF is a page description format, it's not designed to
> be read back. There's no guarantee that the letters on the page appear
> in the PDF in the same order as they seem on the page. The page could
> have all the letter 'a's, then the 'b's and so on, positioned in their
> right places to make up words. To reconstruct the words you'd have to
> spot where the letters were being placed, and then figure out the
> breaks and make up the words. Good luck making the sentences.
>
> Most PDFs aren't that perverse, and you can often get sensible text
> out of them. But then you run into font encodings and graphics and
> column layouts and stuff. Any effort put into writing a readPDF()
> would have to be redone every time someone tried to read a PDF :)
>
> On Linux/Unix there's a bunch of command line tools for trying to do
> this kind of thing with PDF files - see pdftotext for example. You
> could run that from R with system() and then read the text with
> readLines. But there's absolutely no guarantees this will work.
> Windows/Mac versions (did you say what your platform was?) of the
> command line tools may be available.
>
> The real answer is to get the original data in a format with some
> kind of semantics that R could read, for example a CSV or some nice
> XML format.
>
> Barry
>
> --
> blog: http://geospaced.blogspot.com/
> web: http://www.maths.lancs.ac.uk/~rowlings
> web: http://www.rowlingson.com/
> twitter: http://twitter.com/geospacedman
> pics: http://www.flickr.com/photos/spacedman
>
[[alternative HTML version deleted]]
______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
