I think I may have found the source of the problem.
In org.akubraproject.fs.FSBlob, this is what happens when an object gets moved:
if (!file.renameTo(other)) {
if (!file.exists())
throw new MissingBlobException(getId());
throw new IOException("Rename failed for an unknown reason.");
}
file.renameTo() is a java.io.File method that may or may not move files across
symlinks and filesystems, depending on your platform (I'm running Solaris 10,
so go figure):
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4073756
Googling turns up a rich history of problems with the java.io.File renameTo()
method when other filesystems and symlinks are involved. Best practice these
days seems to be to use java.nio.file instead:
http://download.oracle.com/javase/tutorial/essential/io/legacy.html
Chris: shall I open to a bug report in the Akubra JIRA issue tracker?
-- Scott
On 10/05/10, "Scott Prater" <pra...@wisc.edu> wrote:
> Another item of information: Ingesting FOXML files works without a problem,
> induicating that this may be a problem with the copy/replace functionality?
>
> Also, I simplified the process somewhat: really, the new data is spread
> across three different filesystems, with the symlinks occuring at the
> top-level hash character directories:
>
> /fedora/data/objects/0 -> /fedora/db01/objects/0
> /fedora/data/objects/1 -> /fedora/db02/objects/1
> etc.
>
> I mention this, as it occurs to me that perhaps Akubra, as part of its safe
> move/rename function, is attempting to create temporary hard links across
> filesystems? Just a guess...
>
> -- Scott
>
> On 10/05/10, "Scott Prater" <pra...@wisc.edu> wrote:
>
> > We encountered a worrisome problem today after filling up a filesystem and
> > being forced to move our data to another filesystem. After the move and
> > rebuild, we can no longer add datastreams to objects.
> >
> > I'm running Fedora 3.4, with the default Akubra low-level storage module.
> >
> > Here's what happened:
> >
> > 1. filesystem /fedora/data filled up, Fedora (and Tomcat) hung.
> > 2. I shut down Fedora, Tomcat, Postgres.
> > 3. I moved the directory /fedora/data/objects to another filesystem,
> > /fedora/db01/objects.
> > 4. I moved the directory /fedora/data/datastreams to another filesystem,
> > /fedora/db01/datastreams.
> > 5. I made symlinks from the original locations to the new locations:
> > /fedora/data/objects -> /fedora/db01/objects
> > /fedora/data/datastreams -> /fedora/db01/datastreams
> > 6. I started up Postgres, and ran fedora-rebuild.sh twice, once to rebuild
> > the database and once to rebuild the resource index. The rebuild completed
> > successfully.
> > 7. I then started up Tomcat + Fedora, and I was able to view all the
> > objects.
> > 8. I then did a test ingest of an object, and a stub object was
> > successfully created, with a DC datastream. I verified that I could find
> > the object and the datastream on the filesystem (both through the direct
> > path and through the symlinked path). I verified that permsissions were
> > correct at every directory level, and at the file level.
> >
> > I then try to add a datastream to the test object, and I receive this error
> > (truncated):
> >
> > ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal
> > server error
> > javax.ws.rs.WebApplicationException:
> > org.fcrepo.server.errors.GeneralException: Unable to add or modify object
> > (commit canceled)
> > [...]
> > Caused by: java.io.IOException: Rename failed for an unknown reason.
> > at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151)
> > [akubra-fs-0.3.jar:na]
> >
> > I've tested this with the REST API, through the web admin interface, and
> > through the java GUI admin tool, and the results are the same in all three
> > cases. This error occurs regardless of whether I'm adding a datastream to
> > a newly-created object or attempting to add a datastream to an older object.
> >
> > This may properly be an Akubra matter, but I thought I would bring it up
> > here, first. Any suggestions or help would be greatly appreciated.
> >
> > thanks,
> >
> > -- Scott
> >
> > Following is the full stack trace:
> >
> > ERROR 2010-10-05 14:51:16.424 [http-9090-1] (WebApplicationImpl) Internal
> > server error
> > javax.ws.rs.WebApplicationException:
> > org.fcrepo.server.errors.GeneralException: Unable to add or modify object
> > (commit canceled)
> > at
> > org.fcrepo.server.rest.BaseRestResource.handleException(BaseRestResource.java:168)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:583)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.rest.DatastreamResource.addDatastream(DatastreamResource.java:358)
> > [fcrepo-server-3.4.jar:na]
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > [na:1.6.0_21]
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > [na:1.6.0_21]
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > [na:1.6.0_21]
> > at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21]
> > at
> > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:175)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:67)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:163)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:71)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:111)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:63)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:689)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:647)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:638)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:309)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:425)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at
> > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:590)
> > [jersey-bundle-1.0.3.1.jar:1.0.3.1]
> > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> > [servlet-api.jar:na]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > [catalina.jar:6.0.29]
> > at
> > org.fcrepo.server.security.servletfilters.FilterRestApiFlash.doFilter(FilterRestApiFlash.java:66)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > [catalina.jar:6.0.29]
> > at
> > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > [catalina.jar:6.0.29]
> > at
> > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > [catalina.jar:6.0.29]
> > at
> > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > [catalina.jar:6.0.29]
> > at
> > org.fcrepo.server.security.servletfilters.FilterSetup.doFilter(FilterSetup.java:235)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> > [catalina.jar:6.0.29]
> > at
> > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
> > [tomcat-coyote.jar:6.0.29]
> > at
> > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
> > [tomcat-coyote.jar:6.0.29]
> > at
> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
> > [tomcat-coyote.jar:6.0.29]
> > at java.lang.Thread.run(Thread.java:619) [na:1.6.0_21]
> > Caused by: org.fcrepo.server.errors.GeneralException: Unable to add or
> > modify object (commit canceled)
> > at
> > org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1438)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.storage.SimpleDOWriter.commit(SimpleDOWriter.java:508)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.management.DefaultManagement.addDatastream(DefaultManagement.java:572)
> > [fcrepo-server-3.4.jar:na]
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > [na:1.6.0_21]
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > [na:1.6.0_21]
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > [na:1.6.0_21]
> > at java.lang.reflect.Method.invoke(Method.java:597) [na:1.6.0_21]
> > at
> > org.fcrepo.server.messaging.NotificationInvocationHandler.invoke(NotificationInvocationHandler.java:68)
> > [fcrepo-server-3.4.jar:na]
> > at $Proxy0.addDatastream(Unknown Source) [na:na]
> > at
> > org.fcrepo.server.management.ManagementModule.addDatastream(ManagementModule.java:227)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.rest.DatastreamResource.addOrUpdateDatastream(DatastreamResource.java:467)
> > [fcrepo-server-3.4.jar:na]
> > ... 50 common frames omitted
> > Caused by: org.fcrepo.common.FaultException: java.io.IOException: Rename
> > failed for an unknown reason.
> > at
> > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:363)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.safeOverwrite(AkubraLowlevelStorage.java:306)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replace(AkubraLowlevelStorage.java:244)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.replaceObject(AkubraLowlevelStorage.java:150)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorageModule.replaceObject(AkubraLowlevelStorageModule.java:79)
> > [fcrepo-server-3.4.jar:na]
> > at
> > org.fcrepo.server.storage.DefaultDOManager.doCommit(DefaultDOManager.java:1334)
> > [fcrepo-server-3.4.jar:na]
> > ... 60 common frames omitted
> > Caused by: java.io.IOException: Rename failed for an unknown reason.
> > at org.akubraproject.fs.FSBlob.moveTo(FSBlob.java:151)
> > [akubra-fs-0.3.jar:na]
> > at
> > org.akubraproject.map.IdMappingBlob.moveTo(IdMappingBlob.java:69)
> > [akubra-map-0.3.jar:na]
> > at
> > org.fcrepo.server.storage.lowlevel.akubra.AkubraLowlevelStorage.rename(AkubraLowlevelStorage.java:361)
> > [fcrepo-server-3.4.jar:na]
> > ... 65 common frames omitted
> > --
> > Scott Prater
> > Library, Instructional, and Research Applications (LIRA)
> > Division of Information Technology (DoIT)
> > University of Wisconsin - Madison
> > pra...@wisc.edu
------------------------------------------------------------------------------
Beautiful is writing same markup. Internet Explorer 9 supports
standards for HTML5, CSS3, SVG 1.1, ECMAScript5, and DOM L2 & L3.
Spend less time writing and rewriting code and more time creating great
experiences on the web. Be a part of the beta today.
http://p.sf.net/sfu/beautyoftheweb
_______________________________________________
Fedora-commons-users mailing list
Fedora-commons-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fedora-commons-users
