Tim -
Many thanks for your help! I'm going to try to provide some additional
information to see about teasing out the cause of this particular problem.
To start, I should toss out a little system information:
DSpace v. 1.7.2
Java version "1.6.0_26"
PostgresSQL 8.4.7
All on the same RedHat box.
The next important detail is that I should have removed the "manifestOnly"
option from the command as I've experienced the error with manifestOnly set
to true as well as left out (defaulted to false). As a side note, this use
of manifestOnly was more of an afterthought (I'd noted it's experimental
nature), and I'm not certain that it actually does quite what I was looking
for. My purpose is setting up instances with data so that I can fully test
themes I'm building for various institutions, but I'm not really needing
"content files" (bitstreams), just the communities, sub-communities,
collections, item-pages, and *references* to content files (having files or
not doesn't really matter as much, but it'd be preferable to leave out the
asset store). From the description in your email, it does sound indeed as
though setting the manifestOnly option to "true" would be good for my
use-case *as long as* the feature was stable (which, as it has been said,
it's not as of yet). But again, the manifestOnly option is not a priority
(for me) at this point.
It sounds as though the next piece of the puzzle might be the database
connection. The postgres service that I'm using for the instance(s) is on
the same machine as the dspace instance(s). I think that eliminates network
troubles as a potential cause (not that there couldn't be something else
going on with the postgres service).
Next up would be a little more detail concerning the behavior of the
attempted command. The error does not happen immediately, it does run for a
while before erroring out. I'm not sure about how to get exact statistics
for how much of the process is being completed, but I can give some info.
The instance I'm attempting to pull from has 28 communities and 82
sub-communities (all but a few have items). I'm not sure how many
collections but it should have between 6,000 to 8,000 items. I've run the
command numerous times and seem to semi-randomly get between 55mb and
125mb's worth of data. That ranges between 2, 6, or more communities
depending on the run (with collection and item numbers varying widely). In
attempting to do some more detailed troubleshooting I delved into a DEBUG
level of dspace logs. Unfortunately the DEBUG level only gave me one
additional line from which to investigate the issue:
2011-08-19 14:05:51,512 DEBUG net.sf.ehcache.CacheManager @ CacheManager
already shutdown
The rest of the log file didn't show anything relevant. From there I was
going to try to take a look at the postgres logs but 1) I don't have read
access to them, 2) They don't have "query level" information (not sure how
necessary that would be), and 3) My sysadmin seemed to think that it
wouldn't worth checking at this point. He did, however, suggest some other
things to check on which I have. We have several (around 10) tomcat
instances serving dspace installations on this server. Half have no data and
are never used. The other half have descent stores of info (communities,
collections, items, bitstreams) but are very infrequently accessed. One
instance is large and has more regular (but hard for me to define in volume)
usage. I ended up changing all the instances to have a db.maxidle value of 5
(rather than the default -1 [unlimited]). This seemed to enable the AIP
export process to run longer before it errored out. After this I ended up
shutting down all the tomcat instances save for the large one and also
increased the db.maxconnections to 400 (just to give it a whirl). This
seemed to allow the AIP export to go even longer before erroring out.
Another interesting (to me at least) thing is that I also tried changing the
db.maxidle down to a value of 1. The trial here lasted a long time, got a
lot of data, but ended due to exceeding the db.maxwait interval (I'd set it
to 10000 milliseconds) for the idle thread. I'm not sure what all exactly
this would tell us, but at least some of it will be relevant/helpful info.
One other thing that I discovered while tweaking with the database settings
was that running the AIP export while tailing the log file showed that
the org.postgresql.util.PSQLException
popped up at least a few times for each AIP export I'd attempted to do.
"Is it a specific object in DSpace that causes the error?"
This is a great question. I'm not sure how to answer it though (would
probably need a "query level" of postgres logs?). I can say that for certain
this was the case for an earlier error (not one that related to postgres
issues) that we'd encountered with the AIP tool. The cause of the earlier
error was that we'd somehow managed to have a couple items in the instance
that did not have titles. After making sure that the items were removed or
given dc.title (pretty sure that was the exact field) values, that error was
resolved. If there are specific things to look for within the postgres logs,
I'm sure I can get my sysadmin to give me access so that I can look through
them. It seems to me that this might have more to do with postgres than
dspace, but it also seems like a reasonable possibility that dspace is
opening database calls that remain idle and aren't completing, or something
of the like. I'm not sure. Any suggestions or avenues to explore for further
information toward troubleshooting this issue would be greatly appreciated.
Many Thanks Tim! (or others)
- Patrick E.
------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
_______________________________________________
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
