Re: Query to get the number of documents processed from PostgreSQL

2018-10-09 Thread Romaric Pighetti
Thanks Karl for the quick answer. I guess to get only the documents completed while the job is running i will have to fiddle around with the status, for which potential values are expressed in the JobQueue class. I noticed that sometimes (mainly when pausing and restarting a job), selecting

Re: Option to skip documents

2018-10-09 Thread Karl Wright
Hi Romaric, If the error is transient, then the right thing to do is *not* to skip the file, but to retry later. What currently happens? Karl On Tue, Oct 9, 2018 at 10:05 AM Romaric Pighetti < romaric.pighe...@francelabs.com> wrote: > Hi Karl, > Along the lines of this ticket >

Re: Option to skip documents

2018-10-09 Thread Romaric Pighetti
Hi Karl, You're right it might be better to reschedule the file for later in this case. In my case, I was able to crawl the files the first time I tried. When launching another crawl a few days later, the same files were locked. I tried to crawl them several times during the day but never

Option to skip documents

2018-10-09 Thread Romaric Pighetti
Hi Karl, Along the lines of this ticket https://issues.apache.org/jira/projects/CONNECTORS/issues/CONNECTORS-1455?filter=allissues submitted by Julien, I recently stumbled across another smb exception thrown when dealing with some kind of locked files. The error was SmbException tossed

Re: Option to skip documents

2018-10-09 Thread Karl Wright
I can make it retry then skip if it doesn't succeed in a while. Karl On Tue, Oct 9, 2018 at 11:38 AM Romaric Pighetti < romaric.pighe...@francelabs.com> wrote: > Hi Karl, > > You're right it might be better to reschedule the file for later in this > case. > > In my case, I was able to crawl

Re: Option to skip documents

2018-10-09 Thread Karl Wright
r1843343 adds this condition to the list of caught conditions. In the future it would be better to create a ticket. Karl On Tue, Oct 9, 2018 at 3:06 PM Karl Wright wrote: > I can make it retry then skip if it doesn't succeed in a while. > > Karl > > > On Tue, Oct 9, 2018 at 11:38 AM Romaric