Hi, I am also interested by the migration to PostgreSQL 10. @Steph it would be nice if you could precise the changes you did in the configuration file when you wrote : "I'm using 10.4 with no issues.
One or two of the recommended settings for MCF have changed between 9.6 and 10. Simple to resolve though. " Thanks ! Best regards, Olivier > Le 6 août 2018 à 15:52, Karl Wright <daddy...@gmail.com> a écrit : > > It is what is expected with multiple threads active at the same time. > Karl > > > On Mon, Aug 6, 2018 at 7:26 AM Standen Guy <guy.stan...@uk.fujitsu.com > <mailto:guy.stan...@uk.fujitsu.com>> wrote: > Hi Karl, > > I haven’t experienced any job aborts, so all seems OK in that respect. > > Is there anything I can do to reduce these errors in the first place, or it > is just to be expected with the nature of the multiple worker threads and the > query types issued by ManifoldCF? > > Best Regards, > > > > Guy > > > > From: Karl Wright [mailto:daddy...@gmail.com <mailto:daddy...@gmail.com>] > Sent: 06 August 2018 12:16 > To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org> > Subject: Re: PostgreSQL version to support MCF v2.10 > > > > These are exactly the same kind of issue as the first "error" reported. They > will be retried. If they did not get retried, they would abort the job > immediately. > > > > Karl > > > > > > On Mon, Aug 6, 2018 at 6:57 AM Standen Guy <guy.stan...@uk.fujitsu.com > <mailto:guy.stan...@uk.fujitsu.com>> wrote: > > Hi Karl, > > Thanks for the prompt response regarding the first error > example. Do you have a view as to second error i.e. > > “2018-08-03 15:52:42.855 BST [5272] ERROR: could not serialize access due to > concurrent update > > 2018-08-03 15:52:42.855 BST [5272] STATEMENT: SELECT id,status,checktime > FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE > > 2018-08-03 15:52:42.855 BST [7424] ERROR: could not serialize access due to > concurrent update > > 2018-08-03 15:52:42.855 BST [7424] STATEMENT: SELECT id,status,checktime > FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE > > 2018-08-03 15:52:42.855 BST [5716] ERROR: could not serialize access due to > concurrent update > > “ > > > > These errors don’t suggest a retry may sort them out - is this an issue? > > > > Many Thanks, > > > > Guy > > > > From: Karl Wright [mailto:daddy...@gmail.com <mailto:daddy...@gmail.com>] > Sent: 06 August 2018 10:52 > To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org> > Subject: Re: PostgreSQL version to support MCF v2.10 > > > > Ah, the following errors: > > >>>>>> > > 2018-08-03 15:52:25.218 BST [4140] ERROR: could not serialize access due to > read/write dependencies among transactions > > 2018-08-03 15:52:25.218 BST [4140] DETAIL: Reason code: Canceled on > identification as a pivot, during conflict in checking. > > 2018-08-03 15:52:25.218 BST [4140] HINT: The transaction might succeed if > retried. > > <<<<<< > > > > ... occur because of concurrent transactions. The transaction is indeed > retried when this occurs, so unless your job aborts, you are fine. > > > > Karl > > > > > > On Mon, Aug 6, 2018 at 5:49 AM Karl Wright <daddy...@gmail.com > <mailto:daddy...@gmail.com>> wrote: > > What errors are these? Please include them and I can let you know. > > > > Karl > > > > > > On Mon, Aug 6, 2018 at 4:50 AM Standen Guy <guy.stan...@uk.fujitsu.com > <mailto:guy.stan...@uk.fujitsu.com>> wrote: > > Thank you Karl and Steph, > > > > Steph, yes I don’t seem to have any issues with running the MCF jobs, but am > concerned about the PostgreSQL errors. Do you ( or anyone else) have a view > on the errors I have seen in the PostgreSQL logs - is this something you > have seen with 10.4 and if so was it corrected by changing some settings? > > > > Best Regards > > > > Guy > > > > From: Steph van Schalkwyk [mailto:st...@remcam.net <mailto:st...@remcam.net>] > Sent: 03 August 2018 23:21 > To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org> > Subject: Re: PostgreSQL version to support MCF v2.10 > > > > I'm using 10.4 with no issues. > > One or two of the recommended settings for MCF have changed between 9.6 and > 10. > > Simple to resolve though. > > Steph > > > > > > > > On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright <daddy...@gmail.com > <mailto:daddy...@gmail.com>> wrote: > > Hi Guy, > > > > I use Postgresql 9.6 myself and have found no issues with it. I don't know > about v 10 however. > > > > Karl > > > > > > On Fri, Aug 3, 2018 at 11:32 AM Standen Guy <guy.stan...@uk.fujitsu.com > <mailto:guy.stan...@uk.fujitsu.com>> wrote: > > Hi Karl/All, > > I am upgrading from MCF v2.6 supported by PostgreSQL v 9.3.16 > to MCF v2.10. I wonder if there is any official advice as to which > version of PostgreSQL will support MCF v2.10? The MCF v2.10 build and > deployment instructions still suggest that PostgreSQL 9.3 is the latest > tested version of PostgreSQL. Given that PostgreSQL 9.3.x is going end of > life next month ( Sept 2018), is there a preferred newer version that should > be used? > > > > As an experiment I have installed MCF 2.10 supported by PostgreSQL 10.4. > From the outside all seems to work OK, but investigation of the PostgreSQL > logs shows a lot of errors: > > > > e.g. > > “2018-08-03 15:50:00.629 BST [7920] LOG: database system was shut down at > 2018-08-03 15:47:30 BST > > 2018-08-03 15:50:00.734 BST [6344] LOG: database system is ready to accept > connections > > 2018-08-03 15:52:11.140 BST [6460] WARNING: there is already a transaction > in progress > > 2018-08-03 15:52:11.219 BST [6460] WARNING: there is no transaction in > progress > > 2018-08-03 15:52:13.844 BST [5716] WARNING: there is already a transaction > in progress > > 2018-08-03 15:52:13.879 BST [5716] WARNING: there is no transaction in > progress > > 2018-08-03 15:52:25.218 BST [4140] ERROR: could not serialize access due to > read/write dependencies among transactions > > 2018-08-03 15:52:25.218 BST [4140] DETAIL: Reason code: Canceled on > identification as a pivot, during conflict in checking. > > 2018-08-03 15:52:25.218 BST [4140] HINT: The transaction might succeed if > retried. > > 2018-08-03 15:52:25.218 BST [4140] STATEMENT: INSERT INTO jobqueue > (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status) > VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) > > 2018-08-03 15:52:25.219 BST [5800] ERROR: could not serialize access due to > read/write dependencies among transactions > > 2018-08-03 15:52:25.219 BST [5800] DETAIL: Reason code: Canceled on > identification as a pivot, during conflict in checking. > > 2018-08-03 15:52:25.219 BST [5800] HINT: The transaction might succeed if > retried. > > 2018-08-03 15:52:25.219 BST [5800] STATEMENT: INSERT INTO jobqueue > (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status) > VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) > > 2018-08-03 15:52:25.222 BST [5692] ERROR: could not serialize access due to > read/write dependencies among transactions > > 2018-08-03 15:52:25.222 BST [5692] DETAIL: Reason code: Canceled on > identification as a pivot, during conflict in checking. > > 2018-08-03 15:52:25.222 BST [5692] HINT: The transaction might succeed if > retried. > > 2018-08-03 15:52:25.222 BST [5692] STATEMENT: INSERT INTO jobqueue > (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status) > VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) > > 2018-08-03 15:52:28.149 BST [4140] ERROR: could not serialize access due to > read/write dependencies among transactions > > 2018-08-03 15:52:28.149 BST [4140] DETAIL: Reason code: Canceled on > identification as a pivot, during write. > > 2018-08-03 15:52:28.149 BST [4140] HINT: The transaction might succeed if > retried. > > 2018-08-03 15:52:28.149 BST [4140] STATEMENT: UPDATE intrinsiclink SET > processid=$1,isnew=$2 WHERE jobid=$3 AND parentidhash=$4 AND linktype=$5 AND > childidhash=$6 > > 2018-08-03 15:52:28.261 BST [5156] ERROR: could not serialize access due to > read/write dependencies among transactions > > 2018-08-03 15:52:28.261 BST [5156] DETAIL: Reason code: Canceled on > identification as a pivot, during write. > > 2018-08-03 15:52:28.261 BST [5156] HINT: The transaction might succeed if > retried.” > > > > And > > > > “2018-08-03 15:52:42.855 BST [5272] ERROR: could not serialize access due to > concurrent update > > 2018-08-03 15:52:42.855 BST [5272] STATEMENT: SELECT id,status,checktime > FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE > > 2018-08-03 15:52:42.855 BST [7424] ERROR: could not serialize access due to > concurrent update > > 2018-08-03 15:52:42.855 BST [7424] STATEMENT: SELECT id,status,checktime > FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE > > 2018-08-03 15:52:42.855 BST [5716] ERROR: could not serialize access due to > concurrent update > > 2018-08-03 15:52:42.855 BST [5716] STATEMENT: SELECT id,status,checktime > FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE > > 2018-08-03 15:52:42.856 BST [1328] ERROR: could not serialize access due to > concurrent update > > 2018-08-03 15:52:42.856 BST [1328] STATEMENT: SELECT id,status,checktime > FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE > > 2018-08-03 15:52:42.856 BST [5800] ERROR: could not serialize access due to > concurrent update > > 2018-08-03 15:52:42.856 BST [5800] STATEMENT: SELECT id,status,checktime > FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE” > > > > Do you have any advice as to whether it is sensible to use PostgreSQL v10.x > and if so can these errors be overcome? > > > > Best Regards, > > > > Guy > > > Unless otherwise stated, this email has been sent from Fujitsu Services > Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in > England No 2216100) both with registered offices at: 22 Baker Street, London > W1U 3BW > <https://maps.google.com/?q=22+Baker+Street,+London+W1U+3BW&entry=gmail&source=g>; > PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu > Laboratories of Europe Limited (registered in England No. 4153469) both with > registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, > UB4 8FE. > This email is only for the use of its intended recipient. Its contents are > subject to a duty of confidence and may be privileged. Fujitsu does not > guarantee that this email has not been intercepted and amended or that it is > virus-free. > > > > > Unless otherwise stated, this email has been sent from Fujitsu Services > Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in > England No 2216100) both with registered offices at: 22 Baker Street, London > W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu > Laboratories of Europe Limited (registered in England No. 4153469) both with > registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, > UB4 8FE. > This email is only for the use of its intended recipient. Its contents are > subject to a duty of confidence and may be privileged. Fujitsu does not > guarantee that this email has not been intercepted and amended or that it is > virus-free. > > > Unless otherwise stated, this email has been sent from Fujitsu Services > Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in > England No 2216100) both with registered offices at: 22 Baker Street, London > W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu > Laboratories of Europe Limited (registered in England No. 4153469) both with > registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, > UB4 8FE. > This email is only for the use of its intended recipient. Its contents are > subject to a duty of confidence and may be privileged. Fujitsu does not > guarantee that this email has not been intercepted and amended or that it is > virus-free. > > > Unless otherwise stated, this email has been sent from Fujitsu Services > Limited (registered in England No 96056); Fujitsu EMEA PLC (registered in > England No 2216100) both with registered offices at: 22 Baker Street, London > W1U 3BW; PFU (EMEA) Limited, (registered in England No 1578652) and Fujitsu > Laboratories of Europe Limited (registered in England No. 4153469) both with > registered offices at: Hayes Park Central, Hayes End Road, Hayes, Middlesex, > UB4 8FE. > This email is only for the use of its intended recipient. Its contents are > subject to a duty of confidence and may be privileged. Fujitsu does not > guarantee that this email has not been intercepted and amended or that it is > virus-free.
