Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

yes

On Wed, Sep 5, 2018 at 12:10 PM Steph van Schalkwyk 
wrote:

> Thank you. So I'll stop for now?
> Steph
>
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>  Skype: svanschalkwyk
> 
> 
>
> On Wed, Sep 5, 2018 at 11:05 AM, Karl Wright  wrote:
>
>> I'm already working on the Web Connector.  The UI has problems that
>> predate this change and I've alerted Kishore about them -- he'll look into
>> them later today.
>>
>> Karl
>>
>>
>> On Wed, Sep 5, 2018 at 11:55 AM Steph van Schalkwyk 
>> wrote:
>>
>>> Thank you Karl.
>>> You are of course correct in that the incremental crawl is now broken in
>>> that it does a full crawl every time.
>>> I'll jump on the Web Connector and add that functionality.
>>> Thanks for this excellent application and all the help over the years.
>>> Steph
>>>
>>>
>>>
>>>
>>> *Steph van Schalkwyk*
>>> Principal, Remcam Search Engines
>>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>>> http://remcam.net  Skype: svanschalkwyk
>>> 
>>> 
>>>
>>> On Wed, Sep 5, 2018 at 6:33 AM, Karl Wright  wrote:
>>>
 The patch I uploaded doesn't work because the entire tab is broken;
 looks like the UI refactoring broke it and it was never reported.  Fixing
 now.
 Karl


 On Wed, Sep 5, 2018 at 3:57 AM Karl Wright  wrote:

> I coded up the web connector feature I think we need.  See
> CONNECTORS-1528; I've attached a patch.  Please apply and test it out to
> see if it solves the case problem for your IIS site.
>
> For the "//" issue, can you be more specific about the mapping you
> need to do?
>
> Karl
>
>
> On Tue, Sep 4, 2018 at 4:17 PM Karl Wright  wrote:
>
>> Hi Steph,
>>
>> Right, you wouldn't want to touch the framework.
>>
>> The effect of lower-casing the documentURI parameter in the
>> addOrReplaceDocumentWithException method in an output connector would be 
>> to
>> map multiple, independently-fetched, documents that differ only by the 
>> case
>> of the URL together into one document in the index.  The ManifoldCF
>> assumption is that a document with a certain URI can be tracked in the
>> index using exactly that URI.  Mapping the URI to lower case would break
>> that assumption so the framework would make the wrong decision in many
>> cases.
>>
>> If you are picking up documents using the web connector, therefore,
>> and you are getting duplicate documents because the document URLs are
>> sloppy, it is therefore essential that INSTEAD of mapping the document 
>> URI
>> to lower case in the output connector, you map to lower case in the
>> repository connector.  Otherwise the framework will not work right.
>>
>> There is a tab in the web connector that allows you to configure URL
>> normalization, called "Canonicalization".  This would be a very 
>> appropriate
>> place to add URL mapping to lower case.  It should be as simple as adding
>> one more checkbox column in the table, and modifying the method that does
>> the URL processing to include lower-casing.
>>
>> Karl
>>
>>
>>
>> On Tue, Sep 4, 2018 at 2:46 PM Steph van Schalkwyk 
>> wrote:
>>
>>> Unless I have a massive misunderstanding somewhere...
>>>
>>>
>>>
>>>
>>> *Steph van Schalkwyk*
>>> Principal, Remcam Search Engines
>>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>>> http://remcam.net  Skype: svanschalkwyk
>>> 
>>> 
>>>
>>> On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk <
>>> st...@remcam.net> wrote:
>>>
 Hi Karl
 I'm addressing it in the ES Output Connector.
 Not touching the framework :)
 S



 *Steph van Schalkwyk*
 Principal, Remcam Search Engines
 +1.314.452. <+1+314+452+2896>2896st...@remcam.net
 http://remcam.net  Skype: svanschalkwyk
 
 

 On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright 
 wrote:

> Let's make sure we're talking about the same thing.
>
> Here is the output connector method that receives the ID (as the
> documentURI parameter):
>
>   public int addOrReplaceDocumentWithException(String documentURI,
> VersionContext pipelineDescription, RepositoryDocument document, 
> String
> authorityNameString, IOutputAddActivity activities)

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

Thank you. So I'll stop for now?
Steph




*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk



On Wed, Sep 5, 2018 at 11:05 AM, Karl Wright  wrote:

> I'm already working on the Web Connector.  The UI has problems that
> predate this change and I've alerted Kishore about them -- he'll look into
> them later today.
>
> Karl
>
>
> On Wed, Sep 5, 2018 at 11:55 AM Steph van Schalkwyk 
> wrote:
>
>> Thank you Karl.
>> You are of course correct in that the incremental crawl is now broken in
>> that it does a full crawl every time.
>> I'll jump on the Web Connector and add that functionality.
>> Thanks for this excellent application and all the help over the years.
>> Steph
>>
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>>  Skype: svanschalkwyk
>> 
>> 
>>
>> On Wed, Sep 5, 2018 at 6:33 AM, Karl Wright  wrote:
>>
>>> The patch I uploaded doesn't work because the entire tab is broken;
>>> looks like the UI refactoring broke it and it was never reported.  Fixing
>>> now.
>>> Karl
>>>
>>>
>>> On Wed, Sep 5, 2018 at 3:57 AM Karl Wright  wrote:
>>>
 I coded up the web connector feature I think we need.  See
 CONNECTORS-1528; I've attached a patch.  Please apply and test it out to
 see if it solves the case problem for your IIS site.

 For the "//" issue, can you be more specific about the mapping you need
 to do?

 Karl


 On Tue, Sep 4, 2018 at 4:17 PM Karl Wright  wrote:

> Hi Steph,
>
> Right, you wouldn't want to touch the framework.
>
> The effect of lower-casing the documentURI parameter in the
> addOrReplaceDocumentWithException method in an output connector would
> be to map multiple, independently-fetched, documents that differ only by
> the case of the URL together into one document in the index.  The
> ManifoldCF assumption is that a document with a certain URI can be tracked
> in the index using exactly that URI.  Mapping the URI to lower case would
> break that assumption so the framework would make the wrong decision in
> many cases.
>
> If you are picking up documents using the web connector, therefore,
> and you are getting duplicate documents because the document URLs are
> sloppy, it is therefore essential that INSTEAD of mapping the document URI
> to lower case in the output connector, you map to lower case in the
> repository connector.  Otherwise the framework will not work right.
>
> There is a tab in the web connector that allows you to configure URL
> normalization, called "Canonicalization".  This would be a very 
> appropriate
> place to add URL mapping to lower case.  It should be as simple as adding
> one more checkbox column in the table, and modifying the method that does
> the URL processing to include lower-casing.
>
> Karl
>
>
>
> On Tue, Sep 4, 2018 at 2:46 PM Steph van Schalkwyk 
> wrote:
>
>> Unless I have a massive misunderstanding somewhere...
>>
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>> http://remcam.net  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk > > wrote:
>>
>>> Hi Karl
>>> I'm addressing it in the ES Output Connector.
>>> Not touching the framework :)
>>> S
>>>
>>>
>>>
>>> *Steph van Schalkwyk*
>>> Principal, Remcam Search Engines
>>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>>> http://remcam.net  Skype: svanschalkwyk
>>> 
>>> 
>>>
>>> On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright 
>>> wrote:
>>>
 Let's make sure we're talking about the same thing.

 Here is the output connector method that receives the ID (as the
 documentURI parameter):

   public int addOrReplaceDocumentWithException(String documentURI,
 VersionContext pipelineDescription, RepositoryDocument document, String
 authorityNameString, IOutputAddActivity activities)
 throws ManifoldCFException, ServiceInterruption, IOException;

 ManifoldCF doesn't say anywhere that this ID is case insensitive.
 If you make it case insensitive in an output connector, this will
 potentially break a 

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

I'm already working on the Web Connector.  The UI has problems that predate
this change and I've alerted Kishore about them -- he'll look into them
later today.

Karl


On Wed, Sep 5, 2018 at 11:55 AM Steph van Schalkwyk 
wrote:

> Thank you Karl.
> You are of course correct in that the incremental crawl is now broken in
> that it does a full crawl every time.
> I'll jump on the Web Connector and add that functionality.
> Thanks for this excellent application and all the help over the years.
> Steph
>
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>  Skype: svanschalkwyk
> 
> 
>
> On Wed, Sep 5, 2018 at 6:33 AM, Karl Wright  wrote:
>
>> The patch I uploaded doesn't work because the entire tab is broken; looks
>> like the UI refactoring broke it and it was never reported.  Fixing now.
>> Karl
>>
>>
>> On Wed, Sep 5, 2018 at 3:57 AM Karl Wright  wrote:
>>
>>> I coded up the web connector feature I think we need.  See
>>> CONNECTORS-1528; I've attached a patch.  Please apply and test it out to
>>> see if it solves the case problem for your IIS site.
>>>
>>> For the "//" issue, can you be more specific about the mapping you need
>>> to do?
>>>
>>> Karl
>>>
>>>
>>> On Tue, Sep 4, 2018 at 4:17 PM Karl Wright  wrote:
>>>
 Hi Steph,

 Right, you wouldn't want to touch the framework.

 The effect of lower-casing the documentURI parameter in the
 addOrReplaceDocumentWithException method in an output connector would be to
 map multiple, independently-fetched, documents that differ only by the case
 of the URL together into one document in the index.  The ManifoldCF
 assumption is that a document with a certain URI can be tracked in the
 index using exactly that URI.  Mapping the URI to lower case would break
 that assumption so the framework would make the wrong decision in many
 cases.

 If you are picking up documents using the web connector, therefore, and
 you are getting duplicate documents because the document URLs are sloppy,
 it is therefore essential that INSTEAD of mapping the document URI to lower
 case in the output connector, you map to lower case in the repository
 connector.  Otherwise the framework will not work right.

 There is a tab in the web connector that allows you to configure URL
 normalization, called "Canonicalization".  This would be a very appropriate
 place to add URL mapping to lower case.  It should be as simple as adding
 one more checkbox column in the table, and modifying the method that does
 the URL processing to include lower-casing.

 Karl



 On Tue, Sep 4, 2018 at 2:46 PM Steph van Schalkwyk 
 wrote:

> Unless I have a massive misunderstanding somewhere...
>
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
> http://remcam.net  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk 
> wrote:
>
>> Hi Karl
>> I'm addressing it in the ES Output Connector.
>> Not touching the framework :)
>> S
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>> http://remcam.net  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright 
>> wrote:
>>
>>> Let's make sure we're talking about the same thing.
>>>
>>> Here is the output connector method that receives the ID (as the
>>> documentURI parameter):
>>>
>>>   public int addOrReplaceDocumentWithException(String documentURI,
>>> VersionContext pipelineDescription, RepositoryDocument document, String
>>> authorityNameString, IOutputAddActivity activities)
>>> throws ManifoldCFException, ServiceInterruption, IOException;
>>>
>>> ManifoldCF doesn't say anywhere that this ID is case insensitive.
>>> If you make it case insensitive in an output connector, this will
>>> potentially break a lot of things, for example incremental indexing 
>>> (which
>>> organizes the last indexed version by document ID).
>>>
>>> I therefore highly recommend that any "sloppyness" in this parameter
>>> be addressed in the Repository Connector that constructs it.  If the
>>> connector is crawling a repository that believes that URLs are case
>>> insensitive then it should map these IDs to lower case.  If not, then it
>>> shouldn't.
>>>
>>> Karl
>>>

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

Thank you Karl.
You are of course correct in that the incremental crawl is now broken in
that it does a full crawl every time.
I'll jump on the Web Connector and add that functionality.
Thanks for this excellent application and all the help over the years.
Steph




*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk



On Wed, Sep 5, 2018 at 6:33 AM, Karl Wright  wrote:

> The patch I uploaded doesn't work because the entire tab is broken; looks
> like the UI refactoring broke it and it was never reported.  Fixing now.
> Karl
>
>
> On Wed, Sep 5, 2018 at 3:57 AM Karl Wright  wrote:
>
>> I coded up the web connector feature I think we need.  See
>> CONNECTORS-1528; I've attached a patch.  Please apply and test it out to
>> see if it solves the case problem for your IIS site.
>>
>> For the "//" issue, can you be more specific about the mapping you need
>> to do?
>>
>> Karl
>>
>>
>> On Tue, Sep 4, 2018 at 4:17 PM Karl Wright  wrote:
>>
>>> Hi Steph,
>>>
>>> Right, you wouldn't want to touch the framework.
>>>
>>> The effect of lower-casing the documentURI parameter in the
>>> addOrReplaceDocumentWithException method in an output connector would
>>> be to map multiple, independently-fetched, documents that differ only by
>>> the case of the URL together into one document in the index.  The
>>> ManifoldCF assumption is that a document with a certain URI can be tracked
>>> in the index using exactly that URI.  Mapping the URI to lower case would
>>> break that assumption so the framework would make the wrong decision in
>>> many cases.
>>>
>>> If you are picking up documents using the web connector, therefore, and
>>> you are getting duplicate documents because the document URLs are sloppy,
>>> it is therefore essential that INSTEAD of mapping the document URI to lower
>>> case in the output connector, you map to lower case in the repository
>>> connector.  Otherwise the framework will not work right.
>>>
>>> There is a tab in the web connector that allows you to configure URL
>>> normalization, called "Canonicalization".  This would be a very appropriate
>>> place to add URL mapping to lower case.  It should be as simple as adding
>>> one more checkbox column in the table, and modifying the method that does
>>> the URL processing to include lower-casing.
>>>
>>> Karl
>>>
>>>
>>>
>>> On Tue, Sep 4, 2018 at 2:46 PM Steph van Schalkwyk 
>>> wrote:
>>>
 Unless I have a massive misunderstanding somewhere...




 *Steph van Schalkwyk*
 Principal, Remcam Search Engines
 +1.314.452. <+1+314+452+2896>2896st...@remcam.net
 http://remcam.net  Skype: svanschalkwyk
 
 

 On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk 
 wrote:

> Hi Karl
> I'm addressing it in the ES Output Connector.
> Not touching the framework :)
> S
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
> http://remcam.net  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright 
> wrote:
>
>> Let's make sure we're talking about the same thing.
>>
>> Here is the output connector method that receives the ID (as the
>> documentURI parameter):
>>
>>   public int addOrReplaceDocumentWithException(String documentURI,
>> VersionContext pipelineDescription, RepositoryDocument document, String
>> authorityNameString, IOutputAddActivity activities)
>> throws ManifoldCFException, ServiceInterruption, IOException;
>>
>> ManifoldCF doesn't say anywhere that this ID is case insensitive.  If
>> you make it case insensitive in an output connector, this will 
>> potentially
>> break a lot of things, for example incremental indexing (which organizes
>> the last indexed version by document ID).
>>
>> I therefore highly recommend that any "sloppyness" in this parameter
>> be addressed in the Repository Connector that constructs it.  If the
>> connector is crawling a repository that believes that URLs are case
>> insensitive then it should map these IDs to lower case.  If not, then it
>> shouldn't.
>>
>> Karl
>>
>>
>> On Tue, Sep 4, 2018 at 1:36 PM Steph van Schalkwyk 
>> wrote:
>>
>>> Hi Karl.
>>> The issue is that the ES Output Connector uses the uri to create the
>>> _id. When used with IIS which allows case variation in the URI, it 
>>> creates
>>> multiple documents. Clients on Windows IIS are rarely cognizant of that
>>> issue 

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

The patch I uploaded doesn't work because the entire tab is broken; looks
like the UI refactoring broke it and it was never reported.  Fixing now.
Karl


On Wed, Sep 5, 2018 at 3:57 AM Karl Wright  wrote:

> I coded up the web connector feature I think we need.  See
> CONNECTORS-1528; I've attached a patch.  Please apply and test it out to
> see if it solves the case problem for your IIS site.
>
> For the "//" issue, can you be more specific about the mapping you need to
> do?
>
> Karl
>
>
> On Tue, Sep 4, 2018 at 4:17 PM Karl Wright  wrote:
>
>> Hi Steph,
>>
>> Right, you wouldn't want to touch the framework.
>>
>> The effect of lower-casing the documentURI parameter in the
>> addOrReplaceDocumentWithException method in an output connector would be to
>> map multiple, independently-fetched, documents that differ only by the case
>> of the URL together into one document in the index.  The ManifoldCF
>> assumption is that a document with a certain URI can be tracked in the
>> index using exactly that URI.  Mapping the URI to lower case would break
>> that assumption so the framework would make the wrong decision in many
>> cases.
>>
>> If you are picking up documents using the web connector, therefore, and
>> you are getting duplicate documents because the document URLs are sloppy,
>> it is therefore essential that INSTEAD of mapping the document URI to lower
>> case in the output connector, you map to lower case in the repository
>> connector.  Otherwise the framework will not work right.
>>
>> There is a tab in the web connector that allows you to configure URL
>> normalization, called "Canonicalization".  This would be a very appropriate
>> place to add URL mapping to lower case.  It should be as simple as adding
>> one more checkbox column in the table, and modifying the method that does
>> the URL processing to include lower-casing.
>>
>> Karl
>>
>>
>>
>> On Tue, Sep 4, 2018 at 2:46 PM Steph van Schalkwyk 
>> wrote:
>>
>>> Unless I have a massive misunderstanding somewhere...
>>>
>>>
>>>
>>>
>>> *Steph van Schalkwyk*
>>> Principal, Remcam Search Engines
>>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>>> http://remcam.net  Skype: svanschalkwyk
>>> 
>>> 
>>>
>>> On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk 
>>> wrote:
>>>
 Hi Karl
 I'm addressing it in the ES Output Connector.
 Not touching the framework :)
 S



 *Steph van Schalkwyk*
 Principal, Remcam Search Engines
 +1.314.452. <+1+314+452+2896>2896st...@remcam.net
 http://remcam.net  Skype: svanschalkwyk
 
 

 On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright  wrote:

> Let's make sure we're talking about the same thing.
>
> Here is the output connector method that receives the ID (as the
> documentURI parameter):
>
>   public int addOrReplaceDocumentWithException(String documentURI,
> VersionContext pipelineDescription, RepositoryDocument document, String
> authorityNameString, IOutputAddActivity activities)
> throws ManifoldCFException, ServiceInterruption, IOException;
>
> ManifoldCF doesn't say anywhere that this ID is case insensitive.  If
> you make it case insensitive in an output connector, this will potentially
> break a lot of things, for example incremental indexing (which organizes
> the last indexed version by document ID).
>
> I therefore highly recommend that any "sloppyness" in this parameter
> be addressed in the Repository Connector that constructs it.  If the
> connector is crawling a repository that believes that URLs are case
> insensitive then it should map these IDs to lower case.  If not, then it
> shouldn't.
>
> Karl
>
>
> On Tue, Sep 4, 2018 at 1:36 PM Steph van Schalkwyk 
> wrote:
>
>> Hi Karl.
>> The issue is that the ES Output Connector uses the uri to create the
>> _id. When used with IIS which allows case variation in the URI, it 
>> creates
>> multiple documents. Clients on Windows IIS are rarely cognizant of that
>> issue as IIS is so lax in policing that OTB.
>> Currently, every case variation in URI results in a new doc in the
>> index. This is only in the ES output connector.
>> I can add an optional checkbox to do determien that particular action
>> if that would help?
>> Regards,
>> Steph
>>
>>
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>> http://remcam.net  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 12:22 PM, Karl Wright 
>> wrote:

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

I coded up the web connector feature I think we need.  See CONNECTORS-1528;
I've attached a patch.  Please apply and test it out to see if it solves
the case problem for your IIS site.

For the "//" issue, can you be more specific about the mapping you need to
do?

Karl


On Tue, Sep 4, 2018 at 4:17 PM Karl Wright  wrote:

> Hi Steph,
>
> Right, you wouldn't want to touch the framework.
>
> The effect of lower-casing the documentURI parameter in the
> addOrReplaceDocumentWithException method in an output connector would be to
> map multiple, independently-fetched, documents that differ only by the case
> of the URL together into one document in the index.  The ManifoldCF
> assumption is that a document with a certain URI can be tracked in the
> index using exactly that URI.  Mapping the URI to lower case would break
> that assumption so the framework would make the wrong decision in many
> cases.
>
> If you are picking up documents using the web connector, therefore, and
> you are getting duplicate documents because the document URLs are sloppy,
> it is therefore essential that INSTEAD of mapping the document URI to lower
> case in the output connector, you map to lower case in the repository
> connector.  Otherwise the framework will not work right.
>
> There is a tab in the web connector that allows you to configure URL
> normalization, called "Canonicalization".  This would be a very appropriate
> place to add URL mapping to lower case.  It should be as simple as adding
> one more checkbox column in the table, and modifying the method that does
> the URL processing to include lower-casing.
>
> Karl
>
>
>
> On Tue, Sep 4, 2018 at 2:46 PM Steph van Schalkwyk 
> wrote:
>
>> Unless I have a massive misunderstanding somewhere...
>>
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>>  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk 
>> wrote:
>>
>>> Hi Karl
>>> I'm addressing it in the ES Output Connector.
>>> Not touching the framework :)
>>> S
>>>
>>>
>>>
>>> *Steph van Schalkwyk*
>>> Principal, Remcam Search Engines
>>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>>> http://remcam.net  Skype: svanschalkwyk
>>> 
>>> 
>>>
>>> On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright  wrote:
>>>
 Let's make sure we're talking about the same thing.

 Here is the output connector method that receives the ID (as the
 documentURI parameter):

   public int addOrReplaceDocumentWithException(String documentURI,
 VersionContext pipelineDescription, RepositoryDocument document, String
 authorityNameString, IOutputAddActivity activities)
 throws ManifoldCFException, ServiceInterruption, IOException;

 ManifoldCF doesn't say anywhere that this ID is case insensitive.  If
 you make it case insensitive in an output connector, this will potentially
 break a lot of things, for example incremental indexing (which organizes
 the last indexed version by document ID).

 I therefore highly recommend that any "sloppyness" in this parameter be
 addressed in the Repository Connector that constructs it.  If the connector
 is crawling a repository that believes that URLs are case insensitive then
 it should map these IDs to lower case.  If not, then it shouldn't.

 Karl


 On Tue, Sep 4, 2018 at 1:36 PM Steph van Schalkwyk 
 wrote:

> Hi Karl.
> The issue is that the ES Output Connector uses the uri to create the
> _id. When used with IIS which allows case variation in the URI, it creates
> multiple documents. Clients on Windows IIS are rarely cognizant of that
> issue as IIS is so lax in policing that OTB.
> Currently, every case variation in URI results in a new doc in the
> index. This is only in the ES output connector.
> I can add an optional checkbox to do determien that particular action
> if that would help?
> Regards,
> Steph
>
>
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
> http://remcam.net  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 12:22 PM, Karl Wright 
> wrote:
>
>> THanks for the update.
>> Lower-casing the ID would be fine except there are some connectors
>> that care about case.  The web connector is one such because it's up to 
>> the
>> web service to decide if case matters, so the web connector does not view
>> urls with case differences as being the same.  Other connectors 

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

Hi Steph,

Right, you wouldn't want to touch the framework.

The effect of lower-casing the documentURI parameter in the
addOrReplaceDocumentWithException method in an output connector would be to
map multiple, independently-fetched, documents that differ only by the case
of the URL together into one document in the index.  The ManifoldCF
assumption is that a document with a certain URI can be tracked in the
index using exactly that URI.  Mapping the URI to lower case would break
that assumption so the framework would make the wrong decision in many
cases.

If you are picking up documents using the web connector, therefore, and you
are getting duplicate documents because the document URLs are sloppy, it is
therefore essential that INSTEAD of mapping the document URI to lower case
in the output connector, you map to lower case in the repository
connector.  Otherwise the framework will not work right.

There is a tab in the web connector that allows you to configure URL
normalization, called "Canonicalization".  This would be a very appropriate
place to add URL mapping to lower case.  It should be as simple as adding
one more checkbox column in the table, and modifying the method that does
the URL processing to include lower-casing.

Karl



On Tue, Sep 4, 2018 at 2:46 PM Steph van Schalkwyk  wrote:

> Unless I have a massive misunderstanding somewhere...
>
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk 
> wrote:
>
>> Hi Karl
>> I'm addressing it in the ES Output Connector.
>> Not touching the framework :)
>> S
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>>  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright  wrote:
>>
>>> Let's make sure we're talking about the same thing.
>>>
>>> Here is the output connector method that receives the ID (as the
>>> documentURI parameter):
>>>
>>>   public int addOrReplaceDocumentWithException(String documentURI,
>>> VersionContext pipelineDescription, RepositoryDocument document, String
>>> authorityNameString, IOutputAddActivity activities)
>>> throws ManifoldCFException, ServiceInterruption, IOException;
>>>
>>> ManifoldCF doesn't say anywhere that this ID is case insensitive.  If
>>> you make it case insensitive in an output connector, this will potentially
>>> break a lot of things, for example incremental indexing (which organizes
>>> the last indexed version by document ID).
>>>
>>> I therefore highly recommend that any "sloppyness" in this parameter be
>>> addressed in the Repository Connector that constructs it.  If the connector
>>> is crawling a repository that believes that URLs are case insensitive then
>>> it should map these IDs to lower case.  If not, then it shouldn't.
>>>
>>> Karl
>>>
>>>
>>> On Tue, Sep 4, 2018 at 1:36 PM Steph van Schalkwyk 
>>> wrote:
>>>
 Hi Karl.
 The issue is that the ES Output Connector uses the uri to create the
 _id. When used with IIS which allows case variation in the URI, it creates
 multiple documents. Clients on Windows IIS are rarely cognizant of that
 issue as IIS is so lax in policing that OTB.
 Currently, every case variation in URI results in a new doc in the
 index. This is only in the ES output connector.
 I can add an optional checkbox to do determien that particular action
 if that would help?
 Regards,
 Steph





 *Steph van Schalkwyk*
 Principal, Remcam Search Engines
 +1.314.452. <+1+314+452+2896>2896st...@remcam.net
 http://remcam.net  Skype: svanschalkwyk
 
 

 On Tue, Sep 4, 2018 at 12:22 PM, Karl Wright 
 wrote:

> THanks for the update.
> Lower-casing the ID would be fine except there are some connectors
> that care about case.  The web connector is one such because it's up to 
> the
> web service to decide if case matters, so the web connector does not view
> urls with case differences as being the same.  Other connectors also will
> likely care as well. So I don't think lower-casing the document id is a
> smart thing to do.
>
> You could add this bit of configuration to the web connector, if
> that's what you are using, or to whatever other connector constructs the 
> ID.
>
> Karl
>
>
>
> On Tue, Sep 4, 2018 at 12:04 PM Steph van Schalkwyk 
> wrote:
>
>> Thanks Karl.
>>
>> I'll look into that.
>>
>> Another 

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

Unless I have a massive misunderstanding somewhere...




*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk



On Tue, Sep 4, 2018 at 1:42 PM, Steph van Schalkwyk 
wrote:

> Hi Karl
> I'm addressing it in the ES Output Connector.
> Not touching the framework :)
> S
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright  wrote:
>
>> Let's make sure we're talking about the same thing.
>>
>> Here is the output connector method that receives the ID (as the
>> documentURI parameter):
>>
>>   public int addOrReplaceDocumentWithException(String documentURI,
>> VersionContext pipelineDescription, RepositoryDocument document, String
>> authorityNameString, IOutputAddActivity activities)
>> throws ManifoldCFException, ServiceInterruption, IOException;
>>
>> ManifoldCF doesn't say anywhere that this ID is case insensitive.  If you
>> make it case insensitive in an output connector, this will potentially
>> break a lot of things, for example incremental indexing (which organizes
>> the last indexed version by document ID).
>>
>> I therefore highly recommend that any "sloppyness" in this parameter be
>> addressed in the Repository Connector that constructs it.  If the connector
>> is crawling a repository that believes that URLs are case insensitive then
>> it should map these IDs to lower case.  If not, then it shouldn't.
>>
>> Karl
>>
>>
>> On Tue, Sep 4, 2018 at 1:36 PM Steph van Schalkwyk 
>> wrote:
>>
>>> Hi Karl.
>>> The issue is that the ES Output Connector uses the uri to create the
>>> _id. When used with IIS which allows case variation in the URI, it creates
>>> multiple documents. Clients on Windows IIS are rarely cognizant of that
>>> issue as IIS is so lax in policing that OTB.
>>> Currently, every case variation in URI results in a new doc in the
>>> index. This is only in the ES output connector.
>>> I can add an optional checkbox to do determien that particular action if
>>> that would help?
>>> Regards,
>>> Steph
>>>
>>>
>>>
>>>
>>>
>>> *Steph van Schalkwyk*
>>> Principal, Remcam Search Engines
>>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>>> http://remcam.net  Skype: svanschalkwyk
>>> 
>>> 
>>>
>>> On Tue, Sep 4, 2018 at 12:22 PM, Karl Wright  wrote:
>>>
 THanks for the update.
 Lower-casing the ID would be fine except there are some connectors that
 care about case.  The web connector is one such because it's up to the web
 service to decide if case matters, so the web connector does not view urls
 with case differences as being the same.  Other connectors also will likely
 care as well. So I don't think lower-casing the document id is a smart
 thing to do.

 You could add this bit of configuration to the web connector, if that's
 what you are using, or to whatever other connector constructs the ID.

 Karl



 On Tue, Sep 4, 2018 at 12:04 PM Steph van Schalkwyk 
 wrote:

> Thanks Karl.
>
> I'll look into that.
>
> Another note:
> Regarding the ES connector - I have made two additions to it and
> should probably diff them for inclusion after approval:
> 1. lowercased _id (the doc URI).
> 2. Removed dual "/" , e.g. "//" in the _id (I have sloppy sources,
> particularly IIS...)
> 3. Added a "url" metadata field to the ES connector (as ES 6.x does
> not allow accedd to _id in the schema anymore, so no copy_field etc. from
> _id). Hence "url".
>
> Regards,
> Steph
>
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
> http://remcam.net  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 10:50 AM, Karl Wright 
> wrote:
>
>> Hi Steph, I suspect that Jetty is leaking some resource, and we may
>> need to upgrade it.
>>
>> Karl
>>
>>
>> On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk 
>> wrote:
>>
>>> Olivier
>>> By all means.
>>> The only issue I have seen (totally unrelated) is with Jetty, which
>>> has to be restarted about once a week. Still trying to find the issue.
>>> I may be overly sensitive, but I suspect MCF 2.10 with Postgres10
>>> may be a bit slower. I have no empiric evidence at the moment as I'm 
>>> still

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

Hi Karl
I'm addressing it in the ES Output Connector.
Not touching the framework :)
S



*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk



On Tue, Sep 4, 2018 at 1:33 PM, Karl Wright  wrote:

> Let's make sure we're talking about the same thing.
>
> Here is the output connector method that receives the ID (as the
> documentURI parameter):
>
>   public int addOrReplaceDocumentWithException(String documentURI,
> VersionContext pipelineDescription, RepositoryDocument document, String
> authorityNameString, IOutputAddActivity activities)
> throws ManifoldCFException, ServiceInterruption, IOException;
>
> ManifoldCF doesn't say anywhere that this ID is case insensitive.  If you
> make it case insensitive in an output connector, this will potentially
> break a lot of things, for example incremental indexing (which organizes
> the last indexed version by document ID).
>
> I therefore highly recommend that any "sloppyness" in this parameter be
> addressed in the Repository Connector that constructs it.  If the connector
> is crawling a repository that believes that URLs are case insensitive then
> it should map these IDs to lower case.  If not, then it shouldn't.
>
> Karl
>
>
> On Tue, Sep 4, 2018 at 1:36 PM Steph van Schalkwyk 
> wrote:
>
>> Hi Karl.
>> The issue is that the ES Output Connector uses the uri to create the _id.
>> When used with IIS which allows case variation in the URI, it creates
>> multiple documents. Clients on Windows IIS are rarely cognizant of that
>> issue as IIS is so lax in policing that OTB.
>> Currently, every case variation in URI results in a new doc in the index.
>> This is only in the ES output connector.
>> I can add an optional checkbox to do determien that particular action if
>> that would help?
>> Regards,
>> Steph
>>
>>
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>>  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 12:22 PM, Karl Wright  wrote:
>>
>>> THanks for the update.
>>> Lower-casing the ID would be fine except there are some connectors that
>>> care about case.  The web connector is one such because it's up to the web
>>> service to decide if case matters, so the web connector does not view urls
>>> with case differences as being the same.  Other connectors also will likely
>>> care as well. So I don't think lower-casing the document id is a smart
>>> thing to do.
>>>
>>> You could add this bit of configuration to the web connector, if that's
>>> what you are using, or to whatever other connector constructs the ID.
>>>
>>> Karl
>>>
>>>
>>>
>>> On Tue, Sep 4, 2018 at 12:04 PM Steph van Schalkwyk 
>>> wrote:
>>>
 Thanks Karl.

 I'll look into that.

 Another note:
 Regarding the ES connector - I have made two additions to it and should
 probably diff them for inclusion after approval:
 1. lowercased _id (the doc URI).
 2. Removed dual "/" , e.g. "//" in the _id (I have sloppy sources,
 particularly IIS...)
 3. Added a "url" metadata field to the ES connector (as ES 6.x does not
 allow accedd to _id in the schema anymore, so no copy_field etc. from _id).
 Hence "url".

 Regards,
 Steph




 *Steph van Schalkwyk*
 Principal, Remcam Search Engines
 +1.314.452. <+1+314+452+2896>2896st...@remcam.net
 http://remcam.net  Skype: svanschalkwyk
 
 

 On Tue, Sep 4, 2018 at 10:50 AM, Karl Wright 
 wrote:

> Hi Steph, I suspect that Jetty is leaking some resource, and we may
> need to upgrade it.
>
> Karl
>
>
> On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk 
> wrote:
>
>> Olivier
>> By all means.
>> The only issue I have seen (totally unrelated) is with Jetty, which
>> has to be restarted about once a week. Still trying to find the issue.
>> I may be overly sensitive, but I suspect MCF 2.10 with Postgres10 may
>> be a bit slower. I have no empiric evidence at the moment as I'm still
>> delivering the project to UAT. Will keep you posted.
>> Regards,
>> Steph
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>> http://remcam.net  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tavard <
>> olivier.tav...@francelabs.com> wrote:
>>

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

Let's make sure we're talking about the same thing.

Here is the output connector method that receives the ID (as the
documentURI parameter):

  public int addOrReplaceDocumentWithException(String documentURI,
VersionContext pipelineDescription, RepositoryDocument document, String
authorityNameString, IOutputAddActivity activities)
throws ManifoldCFException, ServiceInterruption, IOException;

ManifoldCF doesn't say anywhere that this ID is case insensitive.  If you
make it case insensitive in an output connector, this will potentially
break a lot of things, for example incremental indexing (which organizes
the last indexed version by document ID).

I therefore highly recommend that any "sloppyness" in this parameter be
addressed in the Repository Connector that constructs it.  If the connector
is crawling a repository that believes that URLs are case insensitive then
it should map these IDs to lower case.  If not, then it shouldn't.

Karl


On Tue, Sep 4, 2018 at 1:36 PM Steph van Schalkwyk  wrote:

> Hi Karl.
> The issue is that the ES Output Connector uses the uri to create the _id.
> When used with IIS which allows case variation in the URI, it creates
> multiple documents. Clients on Windows IIS are rarely cognizant of that
> issue as IIS is so lax in policing that OTB.
> Currently, every case variation in URI results in a new doc in the index.
> This is only in the ES output connector.
> I can add an optional checkbox to do determien that particular action if
> that would help?
> Regards,
> Steph
>
>
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 12:22 PM, Karl Wright  wrote:
>
>> THanks for the update.
>> Lower-casing the ID would be fine except there are some connectors that
>> care about case.  The web connector is one such because it's up to the web
>> service to decide if case matters, so the web connector does not view urls
>> with case differences as being the same.  Other connectors also will likely
>> care as well. So I don't think lower-casing the document id is a smart
>> thing to do.
>>
>> You could add this bit of configuration to the web connector, if that's
>> what you are using, or to whatever other connector constructs the ID.
>>
>> Karl
>>
>>
>>
>> On Tue, Sep 4, 2018 at 12:04 PM Steph van Schalkwyk 
>> wrote:
>>
>>> Thanks Karl.
>>>
>>> I'll look into that.
>>>
>>> Another note:
>>> Regarding the ES connector - I have made two additions to it and should
>>> probably diff them for inclusion after approval:
>>> 1. lowercased _id (the doc URI).
>>> 2. Removed dual "/" , e.g. "//" in the _id (I have sloppy sources,
>>> particularly IIS...)
>>> 3. Added a "url" metadata field to the ES connector (as ES 6.x does not
>>> allow accedd to _id in the schema anymore, so no copy_field etc. from _id).
>>> Hence "url".
>>>
>>> Regards,
>>> Steph
>>>
>>>
>>>
>>>
>>> *Steph van Schalkwyk*
>>> Principal, Remcam Search Engines
>>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
>>> http://remcam.net  Skype: svanschalkwyk
>>> 
>>> 
>>>
>>> On Tue, Sep 4, 2018 at 10:50 AM, Karl Wright  wrote:
>>>
 Hi Steph, I suspect that Jetty is leaking some resource, and we may
 need to upgrade it.

 Karl


 On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk 
 wrote:

> Olivier
> By all means.
> The only issue I have seen (totally unrelated) is with Jetty, which
> has to be restarted about once a week. Still trying to find the issue.
> I may be overly sensitive, but I suspect MCF 2.10 with Postgres10 may
> be a bit slower. I have no empiric evidence at the moment as I'm still
> delivering the project to UAT. Will keep you posted.
> Regards,
> Steph
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net
> http://remcam.net  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tavard <
> olivier.tav...@francelabs.com> wrote:
>
>> Hello,
>>
>> Thanks a lot for sharing your PostgreSQL configuration (sorry for the
>> late answer). I will test it soon.
>>
>> Best regards,
>>
>>
>> Olivier TAVARD
>>
>>
>> Le 23 août 2018 à 19:20, Steph van Schalkwyk  a
>> écrit :
>>
>>
>>
>> These are the rpm installs:
>> -
>> file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm
>> - file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm
>> -
>> 

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

Hi Karl.
The issue is that the ES Output Connector uses the uri to create the _id.
When used with IIS which allows case variation in the URI, it creates
multiple documents. Clients on Windows IIS are rarely cognizant of that
issue as IIS is so lax in policing that OTB.
Currently, every case variation in URI results in a new doc in the index.
This is only in the ES output connector.
I can add an optional checkbox to do determien that particular action if
that would help?
Regards,
Steph





*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk



On Tue, Sep 4, 2018 at 12:22 PM, Karl Wright  wrote:

> THanks for the update.
> Lower-casing the ID would be fine except there are some connectors that
> care about case.  The web connector is one such because it's up to the web
> service to decide if case matters, so the web connector does not view urls
> with case differences as being the same.  Other connectors also will likely
> care as well. So I don't think lower-casing the document id is a smart
> thing to do.
>
> You could add this bit of configuration to the web connector, if that's
> what you are using, or to whatever other connector constructs the ID.
>
> Karl
>
>
>
> On Tue, Sep 4, 2018 at 12:04 PM Steph van Schalkwyk 
> wrote:
>
>> Thanks Karl.
>>
>> I'll look into that.
>>
>> Another note:
>> Regarding the ES connector - I have made two additions to it and should
>> probably diff them for inclusion after approval:
>> 1. lowercased _id (the doc URI).
>> 2. Removed dual "/" , e.g. "//" in the _id (I have sloppy sources,
>> particularly IIS...)
>> 3. Added a "url" metadata field to the ES connector (as ES 6.x does not
>> allow accedd to _id in the schema anymore, so no copy_field etc. from _id).
>> Hence "url".
>>
>> Regards,
>> Steph
>>
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>>  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 10:50 AM, Karl Wright  wrote:
>>
>>> Hi Steph, I suspect that Jetty is leaking some resource, and we may need
>>> to upgrade it.
>>>
>>> Karl
>>>
>>>
>>> On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk 
>>> wrote:
>>>
 Olivier
 By all means.
 The only issue I have seen (totally unrelated) is with Jetty, which has
 to be restarted about once a week. Still trying to find the issue.
 I may be overly sensitive, but I suspect MCF 2.10 with Postgres10 may
 be a bit slower. I have no empiric evidence at the moment as I'm still
 delivering the project to UAT. Will keep you posted.
 Regards,
 Steph



 *Steph van Schalkwyk*
 Principal, Remcam Search Engines
 +1.314.452. <+1+314+452+2896>2896st...@remcam.net
 http://remcam.net  Skype: svanschalkwyk
 
 

 On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tavard <
 olivier.tav...@francelabs.com> wrote:

> Hello,
>
> Thanks a lot for sharing your PostgreSQL configuration (sorry for the
> late answer). I will test it soon.
>
> Best regards,
>
>
> Olivier TAVARD
>
>
> Le 23 août 2018 à 19:20, Steph van Schalkwyk  a
> écrit :
>
>
>
> These are the rpm installs:
> - file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-contrib-10.4-
> 1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-devel-10.4-1PGDG.
> rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-server-10.4-
> 1PGDG.rhel7.x86_64.rpm
>
> postgresql_version: 10
> postgresql_data_dir: /var/lib/pgsql/10/data
> postgresql_bin_path: /usr/pgsql-10/bin
> postgresql_config_path: /var/lib/pgsql/10/data
> postgresql_daemon: postgresql-10.service
> postgresql_packages:
> - postgresql10-libs
> - postgresql10
> - postgresql10-server
> - postgresql10-contrib
> # - postgresql10-devel
>
> postgresql_hba_entries:
> - { type: local, database: all, user: postgres, auth_method: peer }
> - { type: local, database: all, user: all, auth_method: peer }
> - { type: host, database: all, user: all, address: '127.0.0.1/32',
> auth_method: md5 }
> - { type: host, database: all, user: all, address: '::1/128',
> auth_method: md5 }
> - { type: host, database: all, user: all, address: '0.0.0.0/0',
> auth_method: md5 }
> - { type: host, database: all, user: all, address: '::0/0',
> auth_method: md5 }
>

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

Thanks Karl.

I'll look into that.

Another note:
Regarding the ES connector - I have made two additions to it and should
probably diff them for inclusion after approval:
1. lowercased _id (the doc URI).
2. Removed dual "/" , e.g. "//" in the _id (I have sloppy sources,
particularly IIS...)
3. Added a "url" metadata field to the ES connector (as ES 6.x does not
allow accedd to _id in the schema anymore, so no copy_field etc. from _id).
Hence "url".

Regards,
Steph




*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk



On Tue, Sep 4, 2018 at 10:50 AM, Karl Wright  wrote:

> Hi Steph, I suspect that Jetty is leaking some resource, and we may need
> to upgrade it.
>
> Karl
>
>
> On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk 
> wrote:
>
>> Olivier
>> By all means.
>> The only issue I have seen (totally unrelated) is with Jetty, which has
>> to be restarted about once a week. Still trying to find the issue.
>> I may be overly sensitive, but I suspect MCF 2.10 with Postgres10 may be
>> a bit slower. I have no empiric evidence at the moment as I'm still
>> delivering the project to UAT. Will keep you posted.
>> Regards,
>> Steph
>>
>>
>>
>> *Steph van Schalkwyk*
>> Principal, Remcam Search Engines
>> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>>  Skype: svanschalkwyk
>> 
>> 
>>
>> On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tavard <
>> olivier.tav...@francelabs.com> wrote:
>>
>>> Hello,
>>>
>>> Thanks a lot for sharing your PostgreSQL configuration (sorry for the
>>> late answer). I will test it soon.
>>>
>>> Best regards,
>>>
>>>
>>> Olivier TAVARD
>>>
>>>
>>> Le 23 août 2018 à 19:20, Steph van Schalkwyk  a écrit
>>> :
>>>
>>>
>>>
>>> These are the rpm installs:
>>> - file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm
>>> - file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm
>>> - file:///tmp/postgres10/postgresql10-contrib-10.4-
>>> 1PGDG.rhel7.x86_64.rpm
>>> - file:///tmp/postgres10/postgresql10-devel-10.4-1PGDG.rhel7.x86_64.rpm
>>> - file:///tmp/postgres10/postgresql10-server-10.4-1PGDG.rhel7.x86_64.rpm
>>>
>>> postgresql_version: 10
>>> postgresql_data_dir: /var/lib/pgsql/10/data
>>> postgresql_bin_path: /usr/pgsql-10/bin
>>> postgresql_config_path: /var/lib/pgsql/10/data
>>> postgresql_daemon: postgresql-10.service
>>> postgresql_packages:
>>> - postgresql10-libs
>>> - postgresql10
>>> - postgresql10-server
>>> - postgresql10-contrib
>>> # - postgresql10-devel
>>>
>>> postgresql_hba_entries:
>>> - { type: local, database: all, user: postgres, auth_method: peer }
>>> - { type: local, database: all, user: all, auth_method: peer }
>>> - { type: host, database: all, user: all, address: '127.0.0.1/32',
>>> auth_method: md5 }
>>> - { type: host, database: all, user: all, address: '::1/128',
>>> auth_method: md5 }
>>> - { type: host, database: all, user: all, address: '0.0.0.0/0',
>>> auth_method: md5 }
>>> - { type: host, database: all, user: all, address: '::0/0', auth_method:
>>> md5 }
>>>
>>> postgresql_global_config_options:
>>> - option: unix_socket_directories
>>> value: '{{ postgresql_unix_socket_directories | join(",") }}'
>>>
>>> - option: standard_conforming_strings
>>> value: 'on'
>>>
>>> - option: shared_buffers
>>> value: '1024MB'
>>>
>>> # max_wal_size = (3 * checkpoint_segments) * 16MB
>>> # checkpoint_segments=300
>>> - option: max_wal_size
>>> value: '14400MB'
>>>
>>> - option: min_wal_size
>>> value: '80MB'
>>>
>>> - option: maintenance_work_mem
>>> value: '2MB'
>>>
>>> - option: listen_addresses
>>> value: '*'
>>>
>>> - option: max_connections
>>> value: '400'
>>>
>>> - option: checkpoint_timeout
>>> value: '900'
>>>
>>> - option: datestyle
>>> value: "iso, mdy"
>>>
>>> - option: autovacuum
>>> value: 'off'
>>>
>>> # vacuum all databases every night (full vacuum on Sunday night, lazy
>>> vacuum every night)
>>> - name: add postgresql cron lazy vacuum
>>> cron:
>>> name: lazy_vacuum
>>> hour: 8
>>> minute: 0
>>> job: "su - postgres -c 'vacuumdb --all --analyze --quiet'"
>>> - name: add postgresql cron full vacuum
>>> cron:
>>> name: full_vacuum
>>> weekday: 0
>>> hour: 10
>>> minute: 0
>>> job: "su - postgres -c 'vacuumdb --all --full --analyze --quiet'"
>>> # re-index all databases once a week
>>> - name: add postgresql cron reindex
>>> cron:
>>> name: reindex
>>> weekday: 0
>>> hour: 12
>>> minute: 0
>>> job: "su - postgres -c 'psql -t -c \"select datname from pg_database
>>> order by datname;\" | xargs -n 1 -I\"{}\" -- psql -U postgres {} -c
>>> \"reindex database {};\"' "
>>>
>>>
>>> This is how I run 2.10.
>>> Been running fine for some weeks without user intervention.
>>> @Karl: Any comments please?
>>> Steph
>>>
>>>
>>>
>>>
>>

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

Hi Steph, I suspect that Jetty is leaking some resource, and we may need to
upgrade it.

Karl


On Tue, Sep 4, 2018 at 11:26 AM Steph van Schalkwyk 
wrote:

> Olivier
> By all means.
> The only issue I have seen (totally unrelated) is with Jetty, which has to
> be restarted about once a week. Still trying to find the issue.
> I may be overly sensitive, but I suspect MCF 2.10 with Postgres10 may be a
> bit slower. I have no empiric evidence at the moment as I'm still
> delivering the project to UAT. Will keep you posted.
> Regards,
> Steph
>
>
>
> *Steph van Schalkwyk*
> Principal, Remcam Search Engines
> +1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
>  Skype: svanschalkwyk
> 
> 
>
> On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tavard <
> olivier.tav...@francelabs.com> wrote:
>
>> Hello,
>>
>> Thanks a lot for sharing your PostgreSQL configuration (sorry for the
>> late answer). I will test it soon.
>>
>> Best regards,
>>
>>
>> Olivier TAVARD
>>
>>
>> Le 23 août 2018 à 19:20, Steph van Schalkwyk  a écrit :
>>
>>
>>
>> These are the rpm installs:
>> - file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm
>> - file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm
>> - file:///tmp/postgres10/postgresql10-contrib-10.4-1PGDG.rhel7.x86_64.rpm
>> - file:///tmp/postgres10/postgresql10-devel-10.4-1PGDG.rhel7.x86_64.rpm
>> - file:///tmp/postgres10/postgresql10-server-10.4-1PGDG.rhel7.x86_64.rpm
>>
>> postgresql_version: 10
>> postgresql_data_dir: /var/lib/pgsql/10/data
>> postgresql_bin_path: /usr/pgsql-10/bin
>> postgresql_config_path: /var/lib/pgsql/10/data
>> postgresql_daemon: postgresql-10.service
>> postgresql_packages:
>> - postgresql10-libs
>> - postgresql10
>> - postgresql10-server
>> - postgresql10-contrib
>> # - postgresql10-devel
>>
>> postgresql_hba_entries:
>> - { type: local, database: all, user: postgres, auth_method: peer }
>> - { type: local, database: all, user: all, auth_method: peer }
>> - { type: host, database: all, user: all, address: '127.0.0.1/32',
>> auth_method: md5 }
>> - { type: host, database: all, user: all, address: '::1/128', auth_method:
>> md5 }
>> - { type: host, database: all, user: all, address: '0.0.0.0/0',
>> auth_method: md5 }
>> - { type: host, database: all, user: all, address: '::0/0', auth_method:
>> md5 }
>>
>> postgresql_global_config_options:
>> - option: unix_socket_directories
>> value: '{{ postgresql_unix_socket_directories | join(",") }}'
>>
>> - option: standard_conforming_strings
>> value: 'on'
>>
>> - option: shared_buffers
>> value: '1024MB'
>>
>> # max_wal_size = (3 * checkpoint_segments) * 16MB
>> # checkpoint_segments=300
>> - option: max_wal_size
>> value: '14400MB'
>>
>> - option: min_wal_size
>> value: '80MB'
>>
>> - option: maintenance_work_mem
>> value: '2MB'
>>
>> - option: listen_addresses
>> value: '*'
>>
>> - option: max_connections
>> value: '400'
>>
>> - option: checkpoint_timeout
>> value: '900'
>>
>> - option: datestyle
>> value: "iso, mdy"
>>
>> - option: autovacuum
>> value: 'off'
>>
>> # vacuum all databases every night (full vacuum on Sunday night, lazy
>> vacuum every night)
>> - name: add postgresql cron lazy vacuum
>> cron:
>> name: lazy_vacuum
>> hour: 8
>> minute: 0
>> job: "su - postgres -c 'vacuumdb --all --analyze --quiet'"
>> - name: add postgresql cron full vacuum
>> cron:
>> name: full_vacuum
>> weekday: 0
>> hour: 10
>> minute: 0
>> job: "su - postgres -c 'vacuumdb --all --full --analyze --quiet'"
>> # re-index all databases once a week
>> - name: add postgresql cron reindex
>> cron:
>> name: reindex
>> weekday: 0
>> hour: 12
>> minute: 0
>> job: "su - postgres -c 'psql -t -c \"select datname from pg_database
>> order by datname;\" | xargs -n 1 -I\"{}\" -- psql -U postgres {} -c
>> \"reindex database {};\"' "
>>
>>
>> This is how I run 2.10.
>> Been running fine for some weeks without user intervention.
>> @Karl: Any comments please?
>> Steph
>>
>>
>>
>>
>

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

Olivier
By all means.
The only issue I have seen (totally unrelated) is with Jetty, which has to
be restarted about once a week. Still trying to find the issue.
I may be overly sensitive, but I suspect MCF 2.10 with Postgres10 may be a
bit slower. I have no empiric evidence at the moment as I'm still
delivering the project to UAT. Will keep you posted.
Regards,
Steph



*Steph van Schalkwyk*
Principal, Remcam Search Engines
+1.314.452. <+1+314+452+2896>2896st...@remcam.net   http://remcam.net
 Skype: svanschalkwyk



On Tue, Sep 4, 2018 at 9:59 AM, Olivier Tavard <
olivier.tav...@francelabs.com> wrote:

> Hello,
>
> Thanks a lot for sharing your PostgreSQL configuration (sorry for the late
> answer). I will test it soon.
>
> Best regards,
>
>
> Olivier TAVARD
>
>
> Le 23 août 2018 à 19:20, Steph van Schalkwyk  a écrit :
>
>
>
> These are the rpm installs:
> - file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-contrib-10.4-1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-devel-10.4-1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-server-10.4-1PGDG.rhel7.x86_64.rpm
>
> postgresql_version: 10
> postgresql_data_dir: /var/lib/pgsql/10/data
> postgresql_bin_path: /usr/pgsql-10/bin
> postgresql_config_path: /var/lib/pgsql/10/data
> postgresql_daemon: postgresql-10.service
> postgresql_packages:
> - postgresql10-libs
> - postgresql10
> - postgresql10-server
> - postgresql10-contrib
> # - postgresql10-devel
>
> postgresql_hba_entries:
> - { type: local, database: all, user: postgres, auth_method: peer }
> - { type: local, database: all, user: all, auth_method: peer }
> - { type: host, database: all, user: all, address: '127.0.0.1/32',
> auth_method: md5 }
> - { type: host, database: all, user: all, address: '::1/128', auth_method:
> md5 }
> - { type: host, database: all, user: all, address: '0.0.0.0/0',
> auth_method: md5 }
> - { type: host, database: all, user: all, address: '::0/0', auth_method:
> md5 }
>
> postgresql_global_config_options:
> - option: unix_socket_directories
> value: '{{ postgresql_unix_socket_directories | join(",") }}'
>
> - option: standard_conforming_strings
> value: 'on'
>
> - option: shared_buffers
> value: '1024MB'
>
> # max_wal_size = (3 * checkpoint_segments) * 16MB
> # checkpoint_segments=300
> - option: max_wal_size
> value: '14400MB'
>
> - option: min_wal_size
> value: '80MB'
>
> - option: maintenance_work_mem
> value: '2MB'
>
> - option: listen_addresses
> value: '*'
>
> - option: max_connections
> value: '400'
>
> - option: checkpoint_timeout
> value: '900'
>
> - option: datestyle
> value: "iso, mdy"
>
> - option: autovacuum
> value: 'off'
>
> # vacuum all databases every night (full vacuum on Sunday night, lazy
> vacuum every night)
> - name: add postgresql cron lazy vacuum
> cron:
> name: lazy_vacuum
> hour: 8
> minute: 0
> job: "su - postgres -c 'vacuumdb --all --analyze --quiet'"
> - name: add postgresql cron full vacuum
> cron:
> name: full_vacuum
> weekday: 0
> hour: 10
> minute: 0
> job: "su - postgres -c 'vacuumdb --all --full --analyze --quiet'"
> # re-index all databases once a week
> - name: add postgresql cron reindex
> cron:
> name: reindex
> weekday: 0
> hour: 12
> minute: 0
> job: "su - postgres -c 'psql -t -c \"select datname from pg_database
> order by datname;\" | xargs -n 1 -I\"{}\" -- psql -U postgres {} -c
> \"reindex database {};\"' "
>
>
> This is how I run 2.10.
> Been running fine for some weeks without user intervention.
> @Karl: Any comments please?
> Steph
>
>
>
>

Re: PostgreSQL version to support MCF v2.10

[Olivier Tavard]

Hello,

Thanks a lot for sharing your PostgreSQL configuration (sorry for the late 
answer). I will test it soon.

Best regards,


Olivier TAVARD


> Le 23 août 2018 à 19:20, Steph van Schalkwyk  a écrit :
> 
> 
> 
> These are the rpm installs:
> - file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm
> - file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm
> - 
> file:///tmp/postgres10/postgresql10-contrib-10.4-1PGDG.rhel7.x86_64.rpm
> - 
> file:///tmp/postgres10/postgresql10-devel-10.4-1PGDG.rhel7.x86_64.rpm
> - 
> file:///tmp/postgres10/postgresql10-server-10.4-1PGDG.rhel7.x86_64.rpm
> 
>   postgresql_version: 10
>   postgresql_data_dir: /var/lib/pgsql/10/data
>   postgresql_bin_path: /usr/pgsql-10/bin
>   postgresql_config_path: /var/lib/pgsql/10/data
>   postgresql_daemon: postgresql-10.service
>   postgresql_packages:
> - postgresql10-libs
> - postgresql10
> - postgresql10-server
> - postgresql10-contrib
> #- postgresql10-devel
> 
>   postgresql_hba_entries:
> - { type: local, database: all, user: postgres, auth_method: peer }
> - { type: local, database: all, user: all, auth_method: peer }
> - { type: host, database: all, user: all, address: '127.0.0.1/32 
> ', auth_method: md5 }
> - { type: host, database: all, user: all, address: '::1/128', 
> auth_method: md5 } 
> - { type: host, database: all, user: all, address: '0.0.0.0/0 
> ', auth_method: md5 }
> - { type: host, database: all, user: all, address: '::0/0', 
> auth_method: md5 }
> 
>   postgresql_global_config_options:
> - option: unix_socket_directories
>   value: '{{ postgresql_unix_socket_directories | join(",") }}'
> 
> - option: standard_conforming_strings
>   value: 'on'
> 
> - option: shared_buffers
>   value: '1024MB'
> 
> # max_wal_size = (3 * checkpoint_segments) * 16MB
> # checkpoint_segments=300
> - option: max_wal_size
>   value: '14400MB'
> 
> - option: min_wal_size
>   value: '80MB'
> 
> - option: maintenance_work_mem
>   value: '2MB'
> 
> - option: listen_addresses
>   value: '*'
> 
> - option: max_connections
>   value: '400'
> 
> - option: checkpoint_timeout
>   value: '900'
> 
> - option: datestyle
>   value: "iso, mdy"
> 
> - option: autovacuum
>   value: 'off'
> 
> # vacuum all databases every night (full vacuum on Sunday night, lazy 
> vacuum every night)
> - name: add postgresql cron lazy vacuum
>   cron:
> name: lazy_vacuum
> hour: 8
> minute: 0
> job: "su - postgres -c 'vacuumdb --all --analyze --quiet'"
> - name: add postgresql cron full vacuum
>   cron:
> name: full_vacuum
> weekday: 0
> hour: 10
> minute: 0
> job: "su - postgres -c 'vacuumdb --all --full --analyze --quiet'"
> # re-index all databases once a week
> - name: add postgresql cron reindex
>   cron:
> name: reindex
> weekday: 0
> hour: 12
> minute: 0
> job: "su - postgres -c 'psql -t -c \"select datname from pg_database 
> order by datname;\" | xargs -n 1 -I\"{}\" -- psql -U postgres {} -c \"reindex 
> database {};\"' "
> 
> 
> This is how I run 2.10.
> Been running fine for some weeks without user intervention.
> @Karl: Any comments please?
> Steph
> 
> 

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

These are the rpm installs:
- file:///tmp/postgres10/postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm
- file:///tmp/postgres10/postgresql10-10.4-1PGDG.rhel7.x86_64.rpm
- file:///tmp/postgres10/postgresql10-contrib-10.4-1PGDG.rhel7.x86_64.rpm
- file:///tmp/postgres10/postgresql10-devel-10.4-1PGDG.rhel7.x86_64.rpm
- file:///tmp/postgres10/postgresql10-server-10.4-1PGDG.rhel7.x86_64.rpm

postgresql_version: 10
postgresql_data_dir: /var/lib/pgsql/10/data
postgresql_bin_path: /usr/pgsql-10/bin
postgresql_config_path: /var/lib/pgsql/10/data
postgresql_daemon: postgresql-10.service
postgresql_packages:
- postgresql10-libs
- postgresql10
- postgresql10-server
- postgresql10-contrib
# - postgresql10-devel

postgresql_hba_entries:
- { type: local, database: all, user: postgres, auth_method: peer }
- { type: local, database: all, user: all, auth_method: peer }
- { type: host, database: all, user: all, address: '127.0.0.1/32',
auth_method: md5 }
- { type: host, database: all, user: all, address: '::1/128', auth_method:
md5 }
- { type: host, database: all, user: all, address: '0.0.0.0/0', auth_method:
md5 }
- { type: host, database: all, user: all, address: '::0/0', auth_method: md5
}

postgresql_global_config_options:
- option: unix_socket_directories
value: '{{ postgresql_unix_socket_directories | join(",") }}'

- option: standard_conforming_strings
value: 'on'

- option: shared_buffers
value: '1024MB'

# max_wal_size = (3 * checkpoint_segments) * 16MB
# checkpoint_segments=300
- option: max_wal_size
value: '14400MB'

- option: min_wal_size
value: '80MB'

- option: maintenance_work_mem
value: '2MB'

- option: listen_addresses
value: '*'

- option: max_connections
value: '400'

- option: checkpoint_timeout
value: '900'

- option: datestyle
value: "iso, mdy"

- option: autovacuum
value: 'off'

# vacuum all databases every night (full vacuum on Sunday night, lazy
vacuum every night)
- name: add postgresql cron lazy vacuum
cron:
name: lazy_vacuum
hour: 8
minute: 0
job: "su - postgres -c 'vacuumdb --all --analyze --quiet'"
- name: add postgresql cron full vacuum
cron:
name: full_vacuum
weekday: 0
hour: 10
minute: 0
job: "su - postgres -c 'vacuumdb --all --full --analyze --quiet'"
# re-index all databases once a week
- name: add postgresql cron reindex
cron:
name: reindex
weekday: 0
hour: 12
minute: 0
job: "su - postgres -c 'psql -t -c \"select datname from pg_database order
by datname;\" | xargs -n 1 -I\"{}\" -- psql -U postgres {} -c \"reindex
database {};\"' "


This is how I run 2.10.
Been running fine for some weeks without user intervention.
@Karl: Any comments please?
Steph

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

I'll publish them in a bit.

Re: PostgreSQL version to support MCF v2.10

[Olivier Tavard]

Hi,

I am also interested by the migration to PostgreSQL 10.
@Steph it would be nice if you could precise the changes you did in the 
configuration file when you wrote :
"I'm using 10.4 with no issues. 

One or two of the recommended settings for MCF have changed between 9.6 and 10. 

Simple to resolve though.
"

Thanks !
Best regards,

Olivier


> Le 6 août 2018 à 15:52, Karl Wright  a écrit :
> 
> It is what is expected with multiple threads active at the same time.
> Karl
> 
> 
> On Mon, Aug 6, 2018 at 7:26 AM Standen Guy  <mailto:guy.stan...@uk.fujitsu.com>> wrote:
> Hi Karl,
> 
> I haven’t experienced any job aborts, so all seems OK in that respect.
> 
> Is there anything I can do to reduce these errors in the first place, or it 
> is just to be expected with the nature of the multiple worker threads and the 
> query types issued by ManifoldCF?
> 
> Best Regards,
> 
>  
> 
> Guy
> 
>  
> 
> From: Karl Wright [mailto:daddy...@gmail.com <mailto:daddy...@gmail.com>] 
> Sent: 06 August 2018 12:16
> To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org>
> Subject: Re: PostgreSQL version to support MCF v2.10
> 
>  
> 
> These are exactly the same kind of issue as the first "error" reported.  They 
> will be retried.  If they did not get retried, they would abort the job 
> immediately.
> 
>  
> 
> Karl
> 
>  
> 
>  
> 
> On Mon, Aug 6, 2018 at 6:57 AM Standen Guy  <mailto:guy.stan...@uk.fujitsu.com>> wrote:
> 
> Hi Karl,
> 
>Thanks for the prompt response regarding the first  error 
> example.   Do you have a view as to second error  i.e.
> 
> “2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access due to 
> concurrent update
> 
> 2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime 
> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due to 
> concurrent update
> 
> 2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime 
> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
> 
> 2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due to 
> concurrent update
> 
> “
> 
>  
> 
> These errors don’t suggest a retry may sort them out  - is this an issue?
> 
>  
> 
> Many Thanks,
> 
>  
> 
> Guy
> 
>  
> 
> From: Karl Wright [mailto:daddy...@gmail.com <mailto:daddy...@gmail.com>] 
> Sent: 06 August 2018 10:52
> To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org>
> Subject: Re: PostgreSQL version to support MCF v2.10
> 
>  
> 
> Ah, the following errors:
> 
> >>>>>>
> 
> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due to 
> read/write dependencies among transactions
> 
> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on 
> identification as a pivot, during conflict in checking.
> 
> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if 
> retried.
> 
> <<<<<< 
> 
>  
> 
> ... occur because of concurrent transactions.  The transaction is indeed 
> retried when this occurs, so unless your job aborts, you are fine.
> 
>  
> 
> Karl
> 
>  
> 
>  
> 
> On Mon, Aug 6, 2018 at 5:49 AM Karl Wright  <mailto:daddy...@gmail.com>> wrote:
> 
> What errors are these?  Please include them and I can let you know.
> 
>  
> 
> Karl
> 
>  
> 
>  
> 
> On Mon, Aug 6, 2018 at 4:50 AM Standen Guy  <mailto:guy.stan...@uk.fujitsu.com>> wrote:
> 
> Thank you Karl and Steph,
> 
>  
> 
> Steph, yes I don’t seem to have any issues with running the MCF jobs, but am 
> concerned about the PostgreSQL errors. Do you ( or anyone else)  have a view 
> on the errors I have seen in the PostgreSQL logs  - is this something you 
> have seen with 10.4  and if so was it corrected by changing some settings? 
> 
>  
> 
> Best Regards
> 
>  
> 
> Guy
> 
>  
> 
> From: Steph van Schalkwyk [mailto:st...@remcam.net <mailto:st...@remcam.net>] 
> Sent: 03 August 2018 23:21
> To: user@manifoldcf.apache.org <mailto:user@manifoldcf.apache.org>
> Subject: Re: PostgreSQL version to support MCF v2.10
> 
>  
> 
> I'm using 10.4 with no issues. 
> 
> One or two of the recommended settings for MCF have changed between 9.6 and 
> 10. 
> 
> Simple to resolve though.
> 
> Steph
> 
>  
> 
> 
> 
>  
> 
> On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright  <mailto:daddy...@gmail.com&g

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

It is what is expected with multiple threads active at the same time.
Karl


On Mon, Aug 6, 2018 at 7:26 AM Standen Guy 
wrote:

> Hi Karl,
>
> I haven’t experienced any job aborts, so all seems OK in that respect.
>
> Is there anything I can do to reduce these errors in the first place, or
> it is just to be expected with the nature of the multiple worker threads
> and the query types issued by ManifoldCF?
>
> Best Regards,
>
>
>
> Guy
>
>
>
> *From:* Karl Wright [mailto:daddy...@gmail.com]
> *Sent:* 06 August 2018 12:16
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: PostgreSQL version to support MCF v2.10
>
>
>
> These are exactly the same kind of issue as the first "error" reported.
> They will be retried.  If they did not get retried, they would abort the
> job immediately.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 6, 2018 at 6:57 AM Standen Guy 
> wrote:
>
> Hi Karl,
>
>Thanks for the prompt response regarding the first  error
> example.   Do you have a view as to second error  i.e.
>
> “2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access
> due to concurrent update
>
> 2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime
> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
>
> 2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due
> to concurrent update
>
> 2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime
> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
>
> 2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due
> to concurrent update
>
> “
>
>
>
> These errors don’t suggest a retry may sort them out  - is this an issue?
>
>
>
> Many Thanks,
>
>
>
> Guy
>
>
>
> *From:* Karl Wright [mailto:daddy...@gmail.com]
> *Sent:* 06 August 2018 10:52
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: PostgreSQL version to support MCF v2.10
>
>
>
> Ah, the following errors:
>
> >>>>>>
>
> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due
> to read/write dependencies among transactions
>
> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on
> identification as a pivot, during conflict in checking.
>
> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if
> retried.
>
> <<<<<<
>
>
>
> ... occur because of concurrent transactions.  The transaction is indeed
> retried when this occurs, so unless your job aborts, you are fine.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 6, 2018 at 5:49 AM Karl Wright  wrote:
>
> What errors are these?  Please include them and I can let you know.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 6, 2018 at 4:50 AM Standen Guy 
> wrote:
>
> Thank you Karl and Steph,
>
>
>
> Steph, yes I don’t seem to have any issues with running the MCF jobs, but
> am concerned about the PostgreSQL errors. Do you ( or anyone else)  have a
> view on the errors I have seen in the PostgreSQL logs  - is this something
> you have seen with 10.4  and if so was it corrected by changing some
> settings?
>
>
>
> Best Regards
>
>
>
> Guy
>
>
>
> *From:* Steph van Schalkwyk [mailto:st...@remcam.net]
> *Sent:* 03 August 2018 23:21
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: PostgreSQL version to support MCF v2.10
>
>
>
> I'm using 10.4 with no issues.
>
> One or two of the recommended settings for MCF have changed between 9.6
> and 10.
>
> Simple to resolve though.
>
> Steph
>
>
>
>
>
>
> On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright  wrote:
>
> Hi Guy,
>
>
>
> I use Postgresql 9.6 myself and have found no issues with it.  I don't
> know about v 10 however.
>
>
>
> Karl
>
>
>
>
>
> On Fri, Aug 3, 2018 at 11:32 AM Standen Guy 
> wrote:
>
> Hi Karl/All,
>
>I am upgrading from MCF v2.6  supported by PostgreSQL v
> 9.3.16   to  MCF v2.10.  I wonder if there is any official advice as to
> which version of PostgreSQL  will support  MCF v2.10? The  MCF v2.10 build
> and deployment instructions still suggest that PostgreSQL 9.3 is the latest
> tested version of PostgreSQL.  Given that PostgreSQL 9.3.x  is going end of
> life next month ( Sept 2018), is there a preferred newer version that
> should be used?
>
>
>
> As an experiment I have installed MCF 2.10  supported by PostgreSQL 10.4.
> From the outside all seems to work OK, but investigation of the PostgreSQL
> logs shows a lot of errors:
>

RE: PostgreSQL version to support MCF v2.10

[Standen Guy]

Hi Karl,
I haven’t experienced any job aborts, so all seems OK in that respect.
Is there anything I can do to reduce these errors in the first place, or it is 
just to be expected with the nature of the multiple worker threads and the 
query types issued by ManifoldCF?
Best Regards,

Guy

From: Karl Wright [mailto:daddy...@gmail.com]
Sent: 06 August 2018 12:16
To: user@manifoldcf.apache.org
Subject: Re: PostgreSQL version to support MCF v2.10

These are exactly the same kind of issue as the first "error" reported.  They 
will be retried.  If they did not get retried, they would abort the job 
immediately.

Karl


On Mon, Aug 6, 2018 at 6:57 AM Standen Guy 
mailto:guy.stan...@uk.fujitsu.com>> wrote:
Hi Karl,
   Thanks for the prompt response regarding the first  error 
example.   Do you have a view as to second error  i.e.
“2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access due to 
concurrent update
2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime FROM 
jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due to 
concurrent update
2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime FROM 
jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due to 
concurrent update
“

These errors don’t suggest a retry may sort them out  - is this an issue?

Many Thanks,

Guy

From: Karl Wright [mailto:daddy...@gmail.com<mailto:daddy...@gmail.com>]
Sent: 06 August 2018 10:52
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: PostgreSQL version to support MCF v2.10

Ah, the following errors:

>>>>>>
2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due to 
read/write dependencies among transactions
2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on 
identification as a pivot, during conflict in checking.
2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if 
retried.
<<<<<<

... occur because of concurrent transactions.  The transaction is indeed 
retried when this occurs, so unless your job aborts, you are fine.

Karl


On Mon, Aug 6, 2018 at 5:49 AM Karl Wright 
mailto:daddy...@gmail.com>> wrote:
What errors are these?  Please include them and I can let you know.

Karl


On Mon, Aug 6, 2018 at 4:50 AM Standen Guy 
mailto:guy.stan...@uk.fujitsu.com>> wrote:
Thank you Karl and Steph,

Steph, yes I don’t seem to have any issues with running the MCF jobs, but am 
concerned about the PostgreSQL errors. Do you ( or anyone else)  have a view on 
the errors I have seen in the PostgreSQL logs  - is this something you have 
seen with 10.4  and if so was it corrected by changing some settings?

Best Regards

Guy

From: Steph van Schalkwyk [mailto:st...@remcam.net<mailto:st...@remcam.net>]
Sent: 03 August 2018 23:21
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: PostgreSQL version to support MCF v2.10

I'm using 10.4 with no issues.
One or two of the recommended settings for MCF have changed between 9.6 and 10.
Simple to resolve though.
Steph



On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright 
mailto:daddy...@gmail.com>> wrote:
Hi Guy,

I use Postgresql 9.6 myself and have found no issues with it.  I don't know 
about v 10 however.

Karl


On Fri, Aug 3, 2018 at 11:32 AM Standen Guy 
mailto:guy.stan...@uk.fujitsu.com>> wrote:
Hi Karl/All,
   I am upgrading from MCF v2.6  supported by PostgreSQL v 9.3.16   
to  MCF v2.10.  I wonder if there is any official advice as to which version of 
PostgreSQL  will support  MCF v2.10? The  MCF v2.10 build and deployment 
instructions still suggest that PostgreSQL 9.3 is the latest tested version of 
PostgreSQL.  Given that PostgreSQL 9.3.x  is going end of life next month ( 
Sept 2018), is there a preferred newer version that should be used?

As an experiment I have installed MCF 2.10  supported by PostgreSQL 10.4.  From 
the outside all seems to work OK, but investigation of the PostgreSQL  logs 
shows a lot of errors:

e.g.
“2018-08-03 15:50:00.629 BST [7920] LOG:  database system was shut down at 
2018-08-03 15:47:30 BST
2018-08-03 15:50:00.734 BST [6344] LOG:  database system is ready to accept 
connections
2018-08-03 15:52:11.140 BST [6460] WARNING:  there is already a transaction in 
progress
2018-08-03 15:52:11.219 BST [6460] WARNING:  there is no transaction in progress
2018-08-03 15:52:13.844 BST [5716] WARNING:  there is already a transaction in 
progress
2018-08-03 15:52:13.879 BST [5716] WARNING:  there is no transaction in progress
2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due to 
read/write dependencies among transactions
2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on 
identification as a pivot, 

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

These are exactly the same kind of issue as the first "error" reported.
They will be retried.  If they did not get retried, they would abort the
job immediately.

Karl


On Mon, Aug 6, 2018 at 6:57 AM Standen Guy 
wrote:

> Hi Karl,
>
>Thanks for the prompt response regarding the first  error
> example.   Do you have a view as to second error  i.e.
>
> “2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access
> due to concurrent update
>
> 2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime
> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
>
> 2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due
> to concurrent update
>
> 2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime
> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
>
> 2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due
> to concurrent update
>
> “
>
>
>
> These errors don’t suggest a retry may sort them out  - is this an issue?
>
>
>
> Many Thanks,
>
>
>
> Guy
>
>
>
> *From:* Karl Wright [mailto:daddy...@gmail.com]
> *Sent:* 06 August 2018 10:52
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: PostgreSQL version to support MCF v2.10
>
>
>
> Ah, the following errors:
>
> >>>>>>
>
> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due
> to read/write dependencies among transactions
>
> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on
> identification as a pivot, during conflict in checking.
>
> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if
> retried.
>
> <<<<<<
>
>
>
> ... occur because of concurrent transactions.  The transaction is indeed
> retried when this occurs, so unless your job aborts, you are fine.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 6, 2018 at 5:49 AM Karl Wright  wrote:
>
> What errors are these?  Please include them and I can let you know.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 6, 2018 at 4:50 AM Standen Guy 
> wrote:
>
> Thank you Karl and Steph,
>
>
>
> Steph, yes I don’t seem to have any issues with running the MCF jobs, but
> am concerned about the PostgreSQL errors. Do you ( or anyone else)  have a
> view on the errors I have seen in the PostgreSQL logs  - is this something
> you have seen with 10.4  and if so was it corrected by changing some
> settings?
>
>
>
> Best Regards
>
>
>
> Guy
>
>
>
> *From:* Steph van Schalkwyk [mailto:st...@remcam.net]
> *Sent:* 03 August 2018 23:21
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: PostgreSQL version to support MCF v2.10
>
>
>
> I'm using 10.4 with no issues.
>
> One or two of the recommended settings for MCF have changed between 9.6
> and 10.
>
> Simple to resolve though.
>
> Steph
>
>
>
>
>
>
> On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright  wrote:
>
> Hi Guy,
>
>
>
> I use Postgresql 9.6 myself and have found no issues with it.  I don't
> know about v 10 however.
>
>
>
> Karl
>
>
>
>
>
> On Fri, Aug 3, 2018 at 11:32 AM Standen Guy 
> wrote:
>
> Hi Karl/All,
>
>I am upgrading from MCF v2.6  supported by PostgreSQL v
> 9.3.16   to  MCF v2.10.  I wonder if there is any official advice as to
> which version of PostgreSQL  will support  MCF v2.10? The  MCF v2.10 build
> and deployment instructions still suggest that PostgreSQL 9.3 is the latest
> tested version of PostgreSQL.  Given that PostgreSQL 9.3.x  is going end of
> life next month ( Sept 2018), is there a preferred newer version that
> should be used?
>
>
>
> As an experiment I have installed MCF 2.10  supported by PostgreSQL 10.4.
> From the outside all seems to work OK, but investigation of the PostgreSQL
> logs shows a lot of errors:
>
>
>
> e.g.
>
> “2018-08-03 15:50:00.629 BST [7920] LOG:  database system was shut down at
> 2018-08-03 15:47:30 BST
>
> 2018-08-03 15:50:00.734 BST [6344] LOG:  database system is ready to
> accept connections
>
> 2018-08-03 15:52:11.140 BST [6460] WARNING:  there is already a
> transaction in progress
>
> 2018-08-03 15:52:11.219 BST [6460] WARNING:  there is no transaction in
> progress
>
> 2018-08-03 15:52:13.844 BST [5716] WARNING:  there is already a
> transaction in progress
>
> 2018-08-03 15:52:13.879 BST [5716] WARNING:  there is no transaction in
> progress
>
> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due
> to read/write dependencies among tra

RE: PostgreSQL version to support MCF v2.10

[Standen Guy]

Hi Karl,
   Thanks for the prompt response regarding the first  error 
example.   Do you have a view as to second error  i.e.
“2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access due to 
concurrent update
2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime FROM 
jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due to 
concurrent update
2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime FROM 
jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due to 
concurrent update
“

These errors don’t suggest a retry may sort them out  - is this an issue?

Many Thanks,

Guy

From: Karl Wright [mailto:daddy...@gmail.com]
Sent: 06 August 2018 10:52
To: user@manifoldcf.apache.org
Subject: Re: PostgreSQL version to support MCF v2.10

Ah, the following errors:

>>>>>>
2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due to 
read/write dependencies among transactions
2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on 
identification as a pivot, during conflict in checking.
2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if 
retried.
<<<<<<

... occur because of concurrent transactions.  The transaction is indeed 
retried when this occurs, so unless your job aborts, you are fine.

Karl


On Mon, Aug 6, 2018 at 5:49 AM Karl Wright 
mailto:daddy...@gmail.com>> wrote:
What errors are these?  Please include them and I can let you know.

Karl


On Mon, Aug 6, 2018 at 4:50 AM Standen Guy 
mailto:guy.stan...@uk.fujitsu.com>> wrote:
Thank you Karl and Steph,

Steph, yes I don’t seem to have any issues with running the MCF jobs, but am 
concerned about the PostgreSQL errors. Do you ( or anyone else)  have a view on 
the errors I have seen in the PostgreSQL logs  - is this something you have 
seen with 10.4  and if so was it corrected by changing some settings?

Best Regards

Guy

From: Steph van Schalkwyk [mailto:st...@remcam.net<mailto:st...@remcam.net>]
Sent: 03 August 2018 23:21
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: PostgreSQL version to support MCF v2.10

I'm using 10.4 with no issues.
One or two of the recommended settings for MCF have changed between 9.6 and 10.
Simple to resolve though.
Steph



On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright 
mailto:daddy...@gmail.com>> wrote:
Hi Guy,

I use Postgresql 9.6 myself and have found no issues with it.  I don't know 
about v 10 however.

Karl


On Fri, Aug 3, 2018 at 11:32 AM Standen Guy 
mailto:guy.stan...@uk.fujitsu.com>> wrote:
Hi Karl/All,
   I am upgrading from MCF v2.6  supported by PostgreSQL v 9.3.16   
to  MCF v2.10.  I wonder if there is any official advice as to which version of 
PostgreSQL  will support  MCF v2.10? The  MCF v2.10 build and deployment 
instructions still suggest that PostgreSQL 9.3 is the latest tested version of 
PostgreSQL.  Given that PostgreSQL 9.3.x  is going end of life next month ( 
Sept 2018), is there a preferred newer version that should be used?

As an experiment I have installed MCF 2.10  supported by PostgreSQL 10.4.  From 
the outside all seems to work OK, but investigation of the PostgreSQL  logs 
shows a lot of errors:

e.g.
“2018-08-03 15:50:00.629 BST [7920] LOG:  database system was shut down at 
2018-08-03 15:47:30 BST
2018-08-03 15:50:00.734 BST [6344] LOG:  database system is ready to accept 
connections
2018-08-03 15:52:11.140 BST [6460] WARNING:  there is already a transaction in 
progress
2018-08-03 15:52:11.219 BST [6460] WARNING:  there is no transaction in progress
2018-08-03 15:52:13.844 BST [5716] WARNING:  there is already a transaction in 
progress
2018-08-03 15:52:13.879 BST [5716] WARNING:  there is no transaction in progress
2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due to 
read/write dependencies among transactions
2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on 
identification as a pivot, during conflict in checking.
2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if 
retried.
2018-08-03 15:52:25.218 BST [4140] STATEMENT:  INSERT INTO jobqueue 
(jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status) 
VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
2018-08-03 15:52:25.219 BST [5800] ERROR:  could not serialize access due to 
read/write dependencies among transactions
2018-08-03 15:52:25.219 BST [5800] DETAIL:  Reason code: Canceled on 
identification as a pivot, during conflict in checking.
2018-08-03 15:52:25.219 BST [5800] HINT:  The transaction might succeed if 
retried.
2018-08-03 15:52:25.219 BST [5800] STATEMENT:  INSERT INTO jobqueue 
(jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status) 
VALUES ($1,$

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

Ah, the following errors:

>>>>>>

2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due
to read/write dependencies among transactions

2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on
identification as a pivot, during conflict in checking.

2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if
retried.

<<<<<<


... occur because of concurrent transactions.  The transaction is indeed
retried when this occurs, so unless your job aborts, you are fine.


Karl



On Mon, Aug 6, 2018 at 5:49 AM Karl Wright  wrote:

> What errors are these?  Please include them and I can let you know.
>
> Karl
>
>
> On Mon, Aug 6, 2018 at 4:50 AM Standen Guy 
> wrote:
>
>> Thank you Karl and Steph,
>>
>>
>>
>> Steph, yes I don’t seem to have any issues with running the MCF jobs, but
>> am concerned about the PostgreSQL errors. Do you ( or anyone else)  have a
>> view on the errors I have seen in the PostgreSQL logs  - is this something
>> you have seen with 10.4  and if so was it corrected by changing some
>> settings?
>>
>>
>>
>> Best Regards
>>
>>
>>
>> Guy
>>
>>
>>
>> *From:* Steph van Schalkwyk [mailto:st...@remcam.net]
>> *Sent:* 03 August 2018 23:21
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: PostgreSQL version to support MCF v2.10
>>
>>
>>
>> I'm using 10.4 with no issues.
>>
>> One or two of the recommended settings for MCF have changed between 9.6
>> and 10.
>>
>> Simple to resolve though.
>>
>> Steph
>>
>>
>>
>>
>>
>>
>> On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright  wrote:
>>
>> Hi Guy,
>>
>>
>>
>> I use Postgresql 9.6 myself and have found no issues with it.  I don't
>> know about v 10 however.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Fri, Aug 3, 2018 at 11:32 AM Standen Guy 
>> wrote:
>>
>> Hi Karl/All,
>>
>>I am upgrading from MCF v2.6  supported by PostgreSQL v
>> 9.3.16   to  MCF v2.10.  I wonder if there is any official advice as to
>> which version of PostgreSQL  will support  MCF v2.10? The  MCF v2.10 build
>> and deployment instructions still suggest that PostgreSQL 9.3 is the latest
>> tested version of PostgreSQL.  Given that PostgreSQL 9.3.x  is going end of
>> life next month ( Sept 2018), is there a preferred newer version that
>> should be used?
>>
>>
>>
>> As an experiment I have installed MCF 2.10  supported by PostgreSQL
>> 10.4.  From the outside all seems to work OK, but investigation of the
>> PostgreSQL  logs shows a lot of errors:
>>
>>
>>
>> e.g.
>>
>> “2018-08-03 15:50:00.629 BST [7920] LOG:  database system was shut down
>> at 2018-08-03 15:47:30 BST
>>
>> 2018-08-03 15:50:00.734 BST [6344] LOG:  database system is ready to
>> accept connections
>>
>> 2018-08-03 15:52:11.140 BST [6460] WARNING:  there is already a
>> transaction in progress
>>
>> 2018-08-03 15:52:11.219 BST [6460] WARNING:  there is no transaction in
>> progress
>>
>> 2018-08-03 15:52:13.844 BST [5716] WARNING:  there is already a
>> transaction in progress
>>
>> 2018-08-03 15:52:13.879 BST [5716] WARNING:  there is no transaction in
>> progress
>>
>> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due
>> to read/write dependencies among transactions
>>
>> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on
>> identification as a pivot, during conflict in checking.
>>
>> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed
>> if retried.
>>
>> 2018-08-03 15:52:25.218 BST [4140] STATEMENT:  INSERT INTO jobqueue
>> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
>> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>>
>> 2018-08-03 15:52:25.219 BST [5800] ERROR:  could not serialize access due
>> to read/write dependencies among transactions
>>
>> 2018-08-03 15:52:25.219 BST [5800] DETAIL:  Reason code: Canceled on
>> identification as a pivot, during conflict in checking.
>>
>> 2018-08-03 15:52:25.219 BST [5800] HINT:  The transaction might succeed
>> if retried.
>>
>> 2018-08-03 15:52:25.219 BST [5800] STATEMENT:  INSERT INTO jobqueue
>> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
>> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>>
>> 2018-08-03

Re: PostgreSQL version to support MCF v2.10

[Karl Wright]

What errors are these?  Please include them and I can let you know.

Karl


On Mon, Aug 6, 2018 at 4:50 AM Standen Guy 
wrote:

> Thank you Karl and Steph,
>
>
>
> Steph, yes I don’t seem to have any issues with running the MCF jobs, but
> am concerned about the PostgreSQL errors. Do you ( or anyone else)  have a
> view on the errors I have seen in the PostgreSQL logs  - is this something
> you have seen with 10.4  and if so was it corrected by changing some
> settings?
>
>
>
> Best Regards
>
>
>
> Guy
>
>
>
> *From:* Steph van Schalkwyk [mailto:st...@remcam.net]
> *Sent:* 03 August 2018 23:21
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: PostgreSQL version to support MCF v2.10
>
>
>
> I'm using 10.4 with no issues.
>
> One or two of the recommended settings for MCF have changed between 9.6
> and 10.
>
> Simple to resolve though.
>
> Steph
>
>
>
>
>
>
> On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright  wrote:
>
> Hi Guy,
>
>
>
> I use Postgresql 9.6 myself and have found no issues with it.  I don't
> know about v 10 however.
>
>
>
> Karl
>
>
>
>
>
> On Fri, Aug 3, 2018 at 11:32 AM Standen Guy 
> wrote:
>
> Hi Karl/All,
>
>I am upgrading from MCF v2.6  supported by PostgreSQL v
> 9.3.16   to  MCF v2.10.  I wonder if there is any official advice as to
> which version of PostgreSQL  will support  MCF v2.10? The  MCF v2.10 build
> and deployment instructions still suggest that PostgreSQL 9.3 is the latest
> tested version of PostgreSQL.  Given that PostgreSQL 9.3.x  is going end of
> life next month ( Sept 2018), is there a preferred newer version that
> should be used?
>
>
>
> As an experiment I have installed MCF 2.10  supported by PostgreSQL 10.4.
> From the outside all seems to work OK, but investigation of the PostgreSQL
> logs shows a lot of errors:
>
>
>
> e.g.
>
> “2018-08-03 15:50:00.629 BST [7920] LOG:  database system was shut down at
> 2018-08-03 15:47:30 BST
>
> 2018-08-03 15:50:00.734 BST [6344] LOG:  database system is ready to
> accept connections
>
> 2018-08-03 15:52:11.140 BST [6460] WARNING:  there is already a
> transaction in progress
>
> 2018-08-03 15:52:11.219 BST [6460] WARNING:  there is no transaction in
> progress
>
> 2018-08-03 15:52:13.844 BST [5716] WARNING:  there is already a
> transaction in progress
>
> 2018-08-03 15:52:13.879 BST [5716] WARNING:  there is no transaction in
> progress
>
> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due
> to read/write dependencies among transactions
>
> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on
> identification as a pivot, during conflict in checking.
>
> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed if
> retried.
>
> 2018-08-03 15:52:25.218 BST [4140] STATEMENT:  INSERT INTO jobqueue
> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>
> 2018-08-03 15:52:25.219 BST [5800] ERROR:  could not serialize access due
> to read/write dependencies among transactions
>
> 2018-08-03 15:52:25.219 BST [5800] DETAIL:  Reason code: Canceled on
> identification as a pivot, during conflict in checking.
>
> 2018-08-03 15:52:25.219 BST [5800] HINT:  The transaction might succeed if
> retried.
>
> 2018-08-03 15:52:25.219 BST [5800] STATEMENT:  INSERT INTO jobqueue
> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>
> 2018-08-03 15:52:25.222 BST [5692] ERROR:  could not serialize access due
> to read/write dependencies among transactions
>
> 2018-08-03 15:52:25.222 BST [5692] DETAIL:  Reason code: Canceled on
> identification as a pivot, during conflict in checking.
>
> 2018-08-03 15:52:25.222 BST [5692] HINT:  The transaction might succeed if
> retried.
>
> 2018-08-03 15:52:25.222 BST [5692] STATEMENT:  INSERT INTO jobqueue
> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>
> 2018-08-03 15:52:28.149 BST [4140] ERROR:  could not serialize access due
> to read/write dependencies among transactions
>
> 2018-08-03 15:52:28.149 BST [4140] DETAIL:  Reason code: Canceled on
> identification as a pivot, during write.
>
> 2018-08-03 15:52:28.149 BST [4140] HINT:  The transaction might succeed if
> retried.
>
> 2018-08-03 15:52:28.149 BST [4140] STATEMENT:  UPDATE intrinsiclink SET
> processid=$1,isnew=$2 WHERE jobid=$3 AND parentidhash=$4 AND linktype=$5
> AND childidhash=$6
>
> 2018-08-03 15:52:28.261 BST [5156] ERRO

Re: PostgreSQL version to support MCF v2.10

[Steph van Schalkwyk]

I'm using 10.4 with no issues.
One or two of the recommended settings for MCF have changed between 9.6 and
10.
Simple to resolve though.
Steph



On Fri, Aug 3, 2018 at 1:29 PM, Karl Wright  wrote:

> Hi Guy,
>
> I use Postgresql 9.6 myself and have found no issues with it.  I don't
> know about v 10 however.
>
> Karl
>
>
> On Fri, Aug 3, 2018 at 11:32 AM Standen Guy 
> wrote:
>
>> Hi Karl/All,
>>
>>I am upgrading from MCF v2.6  supported by PostgreSQL v
>> 9.3.16   to  MCF v2.10.  I wonder if there is any official advice as to
>> which version of PostgreSQL  will support  MCF v2.10? The  MCF v2.10 build
>> and deployment instructions still suggest that PostgreSQL 9.3 is the latest
>> tested version of PostgreSQL.  Given that PostgreSQL 9.3.x  is going end of
>> life next month ( Sept 2018), is there a preferred newer version that
>> should be used?
>>
>>
>>
>> As an experiment I have installed MCF 2.10  supported by PostgreSQL
>> 10.4.  From the outside all seems to work OK, but investigation of the
>> PostgreSQL  logs shows a lot of errors:
>>
>>
>>
>> e.g.
>>
>> “2018-08-03 15:50:00.629 BST [7920] LOG:  database system was shut down
>> at 2018-08-03 15:47:30 BST
>>
>> 2018-08-03 15:50:00.734 BST [6344] LOG:  database system is ready to
>> accept connections
>>
>> 2018-08-03 15:52:11.140 BST [6460] WARNING:  there is already a
>> transaction in progress
>>
>> 2018-08-03 15:52:11.219 BST [6460] WARNING:  there is no transaction in
>> progress
>>
>> 2018-08-03 15:52:13.844 BST [5716] WARNING:  there is already a
>> transaction in progress
>>
>> 2018-08-03 15:52:13.879 BST [5716] WARNING:  there is no transaction in
>> progress
>>
>> 2018-08-03 15:52:25.218 BST [4140] ERROR:  could not serialize access due
>> to read/write dependencies among transactions
>>
>> 2018-08-03 15:52:25.218 BST [4140] DETAIL:  Reason code: Canceled on
>> identification as a pivot, during conflict in checking.
>>
>> 2018-08-03 15:52:25.218 BST [4140] HINT:  The transaction might succeed
>> if retried.
>>
>> 2018-08-03 15:52:25.218 BST [4140] STATEMENT:  INSERT INTO jobqueue
>> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
>> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>>
>> 2018-08-03 15:52:25.219 BST [5800] ERROR:  could not serialize access due
>> to read/write dependencies among transactions
>>
>> 2018-08-03 15:52:25.219 BST [5800] DETAIL:  Reason code: Canceled on
>> identification as a pivot, during conflict in checking.
>>
>> 2018-08-03 15:52:25.219 BST [5800] HINT:  The transaction might succeed
>> if retried.
>>
>> 2018-08-03 15:52:25.219 BST [5800] STATEMENT:  INSERT INTO jobqueue
>> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
>> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>>
>> 2018-08-03 15:52:25.222 BST [5692] ERROR:  could not serialize access due
>> to read/write dependencies among transactions
>>
>> 2018-08-03 15:52:25.222 BST [5692] DETAIL:  Reason code: Canceled on
>> identification as a pivot, during conflict in checking.
>>
>> 2018-08-03 15:52:25.222 BST [5692] HINT:  The transaction might succeed
>> if retried.
>>
>> 2018-08-03 15:52:25.222 BST [5692] STATEMENT:  INSERT INTO jobqueue
>> (jobid,docpriority,checktime,docid,needpriority,dochash,id,checkaction,status)
>> VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9)
>>
>> 2018-08-03 15:52:28.149 BST [4140] ERROR:  could not serialize access due
>> to read/write dependencies among transactions
>>
>> 2018-08-03 15:52:28.149 BST [4140] DETAIL:  Reason code: Canceled on
>> identification as a pivot, during write.
>>
>> 2018-08-03 15:52:28.149 BST [4140] HINT:  The transaction might succeed
>> if retried.
>>
>> 2018-08-03 15:52:28.149 BST [4140] STATEMENT:  UPDATE intrinsiclink SET
>> processid=$1,isnew=$2 WHERE jobid=$3 AND parentidhash=$4 AND linktype=$5
>> AND childidhash=$6
>>
>> 2018-08-03 15:52:28.261 BST [5156] ERROR:  could not serialize access due
>> to read/write dependencies among transactions
>>
>> 2018-08-03 15:52:28.261 BST [5156] DETAIL:  Reason code: Canceled on
>> identification as a pivot, during write.
>>
>> 2018-08-03 15:52:28.261 BST [5156] HINT:  The transaction might succeed
>> if retried.”
>>
>>
>>
>> And
>>
>>
>>
>> “2018-08-03 15:52:42.855 BST [5272] ERROR:  could not serialize access
>> due to concurrent update
>>
>> 2018-08-03 15:52:42.855 BST [5272] STATEMENT:  SELECT id,status,checktime
>> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
>>
>> 2018-08-03 15:52:42.855 BST [7424] ERROR:  could not serialize access due
>> to concurrent update
>>
>> 2018-08-03 15:52:42.855 BST [7424] STATEMENT:  SELECT id,status,checktime
>> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
>>
>> 2018-08-03 15:52:42.855 BST [5716] ERROR:  could not serialize access due
>> to concurrent update
>>
>> 2018-08-03 15:52:42.855 BST [5716] STATEMENT:  SELECT id,status,checktime
>> FROM jobqueue WHERE dochash=$1 AND jobid=$2 FOR UPDATE
>>
>> 2018-08-03 15:52:42.856 BST [1328] ERROR:  could not