Hi Guy, (1) I have no experience with PostgresSQL versions beyond 9.3, but I doubt you would have problems. (2) If you are using multiple processes, even if there's only one agents process, you must use synchronization. I would recommend Zookeeper; file-system-based synchronization is deprecated. (3) Windows has many ways of interfering with file-based sync, including path-length issues. I have seen Windows fail to unlock files and need a reboot to release the lock. This is one reason why file-system-based locking is deprecated.
Thanks, Karl On Mon, Mar 6, 2017 at 4:03 AM, Standen Guy <guy.stan...@uk.fujitsu.com> wrote: > Hi Karl, > > Thanks for that I will try version 2.6. Whilst moving to > MCF 2.6 I would potentially like to upgrade my backend PostgreSQL version > from 9.3.5. > > 1) Do you have a recommendation for which PostgreSQL to use with MCF 2.6 > e.g. PostgreSQL 9.3.16 or PostgreSQL 9.6.2? > > 2) For a production system on a single server running a single MCF agents > process would you recommend the file based synchronisation locking or > zookeeper based synchronisation locking. With the file based > synchronisation locking mechanism I have sometimes seen errors of the form : > > 'D:\Apps\ManifoldCF\apache-manifoldcf-2.0.1\multiprocess- > file-example\.\.\syncharea\475\708\lock-_POOLTARGET__OUTPUTCONNECTORPOOL_Solr > COLL1 osp_unstruct.lock' failed: Access is denied’ ( I have ensured that > the SYNCHAREA is not scanned by AV or Indexed by Windows Search and all > MCF processes run as the same user) > > What could cause these errors? > > > > Many Thanks, > > > > Guy > > > > *From:* Karl Wright [mailto:daddy...@gmail.com] > *Sent:* 03 March 2017 17:35 > > *To:* user@manifoldcf.apache.org > *Subject:* Re: Advice on which PostgreSQL to use with ManifoldCF 2.6 > > > > Hi Guy: > > > > It is expected that sometimes database deadlock will develop, and the > transaction will need to be retried. There is code in MCF for doing this: > > > > >>>>>> > > if (sqlState != null && sqlState.equals("40001")) > > return new ManifoldCFException(message,e, > ManifoldCFException.DATABASE_TRANSACTION_ABORT); > > <<<<<< > > > > I suspect that your version of MCF is old enough so that this particular > error and the associated retry are not taking place. Upgrading to 2.6 will > definitely help there. > > > > Karl > > > > > > On Fri, Mar 3, 2017 at 11:17 AM, Standen Guy <guy.stan...@uk.fujitsu.com> > wrote: > > Hi Karl, > > Thanks for coming back so quickly. Unfortunately I wasn’t > using a JCIFS connection. One of the issues I was seeing was between a > crawl of an intranet site (no explicit throttling other than number of > connections) and scheduled crawl (every 5 mins) to a relational DB via > JDBC connector again no explicit throttling. To simplify things both jobs > are using a NULL output connection. Sometimes both the Web crawl and the > JDBC connection can run together but at other times 1 or both jobs will > appear to lock up with just a few active documents showing. When I get a > lock up the mcf log contains errors like: > > > > “DEBUG 2017-03-03 15:28:20,466 (Worker thread '5') - Exception Database > exception: SQLException doing query (40001): ERROR: could not serialize > access due to read/write dependencies among transactions” > > > > See the attached log extract for a little more detail. > > > > Any view why this might be happening? > > > > Best Regards, > > > > Guy > > > > > > > > > > > > > > *From:* Karl Wright [mailto:daddy...@gmail.com] > *Sent:* 03 March 2017 11:27 > *To:* user@manifoldcf.apache.org > *Subject:* Re: Advice on which PostgreSQL to use with ManifoldCF 2.6 > > > > Hi Guy, > > > > A issue with concurrent jobs is known for jobs sharing the same JCIFS > connection. Is that what you are using? This has nothing to do with the > version of Postgresql you are using; it has to do with what "bins" > documents are thought to come from. There has been a recent improvement > for this issue, which will be released in April. See > https://issues.apache.org/jira/browse/CONNECTORS-1364. > > > > The current version of MCF (2.6) supports Solr 6.x. > > > > Thanks, > > Karl > > > > > > On Fri, Mar 3, 2017 at 5:27 AM, Standen Guy <guy.stan...@uk.fujitsu.com> > wrote: > > Hi Karl, > > I am currently using MCF 2.0.1 with PostgreSQL 9.3.5 on Windows and have > had some issues with multiple jobs running concurrently. > > I am considering upgrading to MCF 2.6 and to a newer version of > PostgreSQL. Would you be able to advise which version of PostgreSQL I > should consider using with MCF 2.6 (e.g. PostgreSQL 9.3.16 or 9.6.2) > > > > I am also considering upgrading from SOLR 4.10.3 to a newer version. The > MCF compatibility matrix mentions that compatibility has been tested to > SOLR version 4.5.1. Do you have any advice about compatibility with the > newer versions of SOLR e.g. 6.4.1. > > > > Best Regards > > > > Guy > > > > > Unless otherwise stated, this email has been sent from Fujitsu Services > Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in > England No 2216100) both with registered offices at: 22 Baker Street, > London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and > Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) > both with registered offices at: Hayes Park Central, Hayes End Road, Hayes, > Middlesex, UB4 8FE. > This email is only for the use of its intended recipient. Its contents are > subject to a duty of confidence and may be privileged. Fujitsu does not > guarantee that this email has not been intercepted and amended or that it > is virus-free. > > > > > Unless otherwise stated, this email has been sent from Fujitsu Services > Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in > England No 2216100) both with registered offices at: 22 Baker Street, > London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and > Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) > both with registered offices at: Hayes Park Central, Hayes End Road, Hayes, > Middlesex, UB4 8FE. > This email is only for the use of its intended recipient. Its contents are > subject to a duty of confidence and may be privileged. Fujitsu does not > guarantee that this email has not been intercepted and amended or that it > is virus-free. > > > > Unless otherwise stated, this email has been sent from Fujitsu Services > Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in > England No 2216100) both with registered offices at: 22 Baker Street, > London W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and > Fujitsu Laboratories of Europe Limited (registered in England No. 4153469) > both with registered offices at: Hayes Park Central, Hayes End Road, Hayes, > Middlesex, UB4 8FE. > This email is only for the use of its intended recipient. Its contents are > subject to a duty of confidence and may be privileged. Fujitsu does not > guarantee that this email has not been intercepted and amended or that it > is virus-free. >
