Hi Andy,
Thanks for your reply. I tried your 2nd Option.
The StoreConnection.release(location) option didn't seem to work.
I'm assuming the location should point to the main TopLevel Directory
/etc/fuseki/databases/201906031724/
├── Data-0001
│ ├── GOSP.bpt
│ ├── GOSP.dat
| | ...
│ └── tdb.lock
└── tdb.lock
I gave "/etc/fuseki/databases/201906031724" as location input
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
LOGGER.info("location {} is presence in StoreConnection.cache
is {}", lockFilename, StoreConnection.isSetup(location));
StoreConnection.release(location);
return ExecUtil.restartJetty();
}
INFO location /etc/fuseki/databases/201906031724/tdb.lock is presence in
StoreConnection.cache is false
This option failed with the same original error. The storeConnection has a
cache, and this location was not in that cache().
=============
So next I tried the following option
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
LOGGER.info("location of the lock file is {}", lockFilename);
StoreConnection storeConnection =
StoreConnection.connectCreate(location);
storeConnection.getLock().unlock();
return ExecUtil.restartJetty();
}
This time I got the following error
org.apache.jena.dboe.base.file.AlreadyLocked: Failed to get a lock:
file='/etc/fuseki/databases/201906031727/tdb.lock': Lock already held
at
org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:187)
at
org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
at
org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:87)
at
org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:61)
at
org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:52)
========
Then I tried to use the ProcessFileLock directly
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
String lockFilename = location.getPath(Names.TDB_LOCK_FILE);
ProcessFileLock fileLock = ProcessFileLock.create(lockFilename);
LOGGER.info("Lock exists for {} is {}", lockFilename,
fileLock.isLockedHere());
fileLock.unlock();
return ExecUtil.restartJetty();
}
INFO Lock exists for /etc/fuseki/databases/201906031755/tdb.lock is true
Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to get a
lock: file='/etc/fuseki/databases/201906031755/Data-0001/tdb.lock':
held by process 8211
at
org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
at
org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
at
org.apache.jena.tdb2.sys.StoreConnection.make(StoreConnection.java:87)
at
org.apache.jena.tdb2.sys.StoreConnection.connectCreate(StoreConnection.java:61)
at
org.apache.jena.tdb2.sys.DatabaseOps.createSwitchable(DatabaseOps.java:97)
at org.apache.jena.tdb2.sys.DatabaseOps.create(DatabaseOps.java:78)
at
org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:90)
at
org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
at
java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at
org.apache.jena.tdb2.sys.DatabaseConnection.make(DatabaseConnection.java:76)
at
org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:61)
at
org.apache.jena.tdb2.sys.DatabaseConnection.connectCreate(DatabaseConnection.java:52)
This time the error was for the lock file which is inside the Data-0001
folder instead.
=====
When I tried using the unlock on both the lock file explicitly, it worked.
I also tried the TDBInternal.expel(dataset) and it worked
private boolean postProcessingWebServer(String path) {
Location location = Location.create(path);
DatabaseConnection databaseConnection =
DatabaseConnection.connectCreate(location);
TDBInternal.expel(databaseConnection.getDatasetGraph());
return ExecUtil.restartJetty();
}
I'm going ahead with the TDBInternal.expel option for now.
Regards
Amit
On Sat, Jun 1, 2019 at 5:59 AM Andy Seaborne <a...@apache.org> wrote:
> Hi Amit,
>
> You are correct - when used in a JVM, the dataset remains manged by the
> JVM after loading. This is intentional - the app may want to use the
> dataset some more in Java code.
>
> You can:
> 1/ Fork the process as a separate OS process using java.util.ProcessBuilder
>
> 2/ Force it out of the running JVM with
> TDBInternal.expel(dataset)
> or
> StoreConenction.release(location)
>
> 3/ Work with the dataset in Fuseki - you can POST data to it
>
> 4/ Not use a war file but use Fuseki main, and run your program and
> Fuseki in the same JVM.
>
> Symbolic links make no difference - they are only involved in file
> system path naming. Once opened, the OS pathname isn't involved.
>
> Andy
>
> On 31/05/2019 21:23, Amit Kumar wrote:
> > Hi,
> > I'm having issue an issue with using Fuseki as a war file inside jetty.
> We
> > have a java application which looks for new data and loads it onto
> fuseki.
> > When it is doing it for the first time, it has to create a new index, so
> we
> > call the tdb2.tdbloader class directly with appropriate parameters. Here
> is
> > the relevant code snippet
> >
> > import tdb2.tdbloader;
> >
> > ...
> >
> > public static List<String> TDBLoaderParams = Arrays.asList("-q",
> >> "--loader", "parallel", "--loc");
> >> ...
> >> List<String> tdbParams = new ArrayList<>(TDBLoaderParams);
> >> tdbParams.add(dstPath);
> >> tdbParams.addAll(srcPaths);
> >> try {
> >> ...
> >> tdbloader.main(tdbParams.toArray(new String[tdbParams.size()]));
> >> }
> >> ...
> >> ...
> >>
> >
> >
> > This works as expected and create a new index at the desired location.
> Now,
> > the problem is the same java process has to apply other feeds onto the
> same
> > graph, so we move the softlink pointing to the fuseki database and
> execute
> > a jetty restart command by calling a shell action using Java
> ProcessBuilder
> > process.
> > The jetty though fails to restart with the following error:
> >
> >>
> >> org.apache.jena.assembler.exceptions.AssemblerException: caught: Failed
> to
> >> get a lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock':
> >> held by process 35918 at
> >>
> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.openBySpecificType(AssemblerGroup.java:165)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerGroup$PlainAssemblerGroup.open(AssemblerGroup.java:144)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerGroup$ExpandingAssemblerGroup.open(AssemblerGroup.java:93)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:39)
> >> at
> >>
> org.apache.jena.assembler.assemblers.AssemblerBase.open(AssemblerBase.java:35)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.getDataset(FusekiConfig.java:353)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.buildDataService(FusekiConfig.java:303)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.buildDataAccessPoint(FusekiConfig.java:292)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.readConfiguration(FusekiConfig.java:275)
> >> at
> >>
> org.apache.jena.fuseki.build.FusekiConfig.readConfigurationDirectory(FusekiConfig.java:254)
> >> ... ... Caused by: org.apache.jena.dboe.DBOpEnvException: Failed to get
> a
> >> lock: file='/home/amit/fuseki/databases/201905302214/tdb.lock': held by
> >> process 35918 at
> >>
> org.apache.jena.dboe.base.file.ProcessFileLock.lockOperation(ProcessFileLock.java:199)
> >> at
> >>
> org.apache.jena.dboe.base.file.ProcessFileLock.lockEx(ProcessFileLock.java:125)
> >> at
> >>
> org.apache.jena.tdb2.sys.DatabaseConnection.buildForCache(DatabaseConnection.java:88)
> >> at
> >>
> org.apache.jena.tdb2.sys.DatabaseConnection.lambda$make$0(DatabaseConnection.java:76)
> >> at
> >>
> java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
> >>
> >
> > So the Jetty Server fails to restart while the jvm that created the index
> > is still active, even though it has completed the index building part. It
> > restarts fine once the loader JVM dies. It looks like the parallel
> loader
> > class is not releasing the lock() properly. FYI, I'm using 3.11 Jena
> >
> > From apache jena github
> >
> https://github.com/apache/jena/blob/66d14eda44dd0f37a9df31ed27c55c26600ff5f8/jena-db/jena-dboe-base/src/main/java/org/apache/jena/dboe/base/file/ProcessFileLock.java#L184-L218
> >
> >
> >
> > Regards
> > Amit Kumar
> >
>
