[jira] [Commented] (CONNECTORS-1535) Documentum Connector cannot find dfc.properties
[
https://issues.apache.org/jira/browse/CONNECTORS-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745820#comment-16745820
]
James Thomas commented on CONNECTORS-1535:
--
One other thing: I don't believe I have evidence that it's necessary, but it is
now my practice to edit the CLASSPATH initialisation in the
documentum-registry/run.sh script in the same way as documentum-server/run.sh,
again to parallel the Windows run.bat scripts.
In the experiment I just described, I reverted to CLASSPATH="" in both scripts
initially, but only changed documentum-server/run.sh before restarting MCF.
> Documentum Connector cannot find dfc.properties
> ---
>
> Key: CONNECTORS-1535
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1535
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10, ManifoldCF 2.11
> Environment: Manifold 2.11
> CentOS Linux release 7.5.1804 (Core)
> OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)
>
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.13
>
>
> I have found that when installing a clean MCF instance I cannot get
> Documentum repository connectors to connect to Documentum until I have added
> this line to the processes/documentum-server/run.sh script before the call to
> Java:
>
> {code:java}
> CLASSPATH="$CLASSPATH""$PATHSEP""$DOCUMENTUM"{code}
> Until I do this, attempts to save the connector will result in this output to
> the console:
>
> {noformat}
> 4 [RMI TCP Connection(2)-127.0.0.1] ERROR
> com.documentum.fc.common.impl.preferences.PreferencesManager -
> [DFC_PREFERENCE_LOAD_FAILED] Failed to load persistent preferences from null
> java.io.FileNotFoundException: dfc.properties
> at
> com.documentum.fc.common.impl.preferences.PreferencesManager.locateMainPersistentStore(PreferencesManager.java:378)
> at
> com.documentum.fc.common.impl.preferences.PreferencesManager.readPersistentProperties(PreferencesManager.java:329)
> at
> com.documentum.fc.common.impl.preferences.PreferencesManager.(PreferencesManager.java:37)
> at
> com.documentum.fc.common.DfPreferences.initialize(DfPreferences.java:64)
> ..{noformat}
> and this message in the MCF UI:
>
> {noformat}
> Connection failed: Documentum error: No DocBrokers are configured{noformat}
>
>
> I mentioned this in #1512 for MCF 2.10 but it got lost in the other work done
> in that ticket. While setting up 2.11 from scratch I encountered it again.
>
> Once I have edited the run.sh script I get this in the console, showing that
> (for whatever reason) the change is significant:
>
> {noformat}
> Reading DFC configuration from
> "file:/opt/manifold/apache-manifoldcf-2.11/processes/documentum-server/dfc.properties"
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1535) Documentum Connector cannot find dfc.properties
[
https://issues.apache.org/jira/browse/CONNECTORS-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745336#comment-16745336
]
James Thomas commented on CONNECTORS-1535:
--
I have found an alternative fix.
If you are prepared to call run.sh by cd-ing into the directory in which it
exists, you can modify in a less intrusive way than I initially suggested.
change
{code:java}
CLASSPATH=""{code}
to
{code:java}
CLASSPATH ="."{code}
This makes the script more like the run.bat equivalent for Windows, which
already has the working directory on the CLASSPATH.
> Documentum Connector cannot find dfc.properties
> ---
>
> Key: CONNECTORS-1535
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1535
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10, ManifoldCF 2.11
> Environment: Manifold 2.11
> CentOS Linux release 7.5.1804 (Core)
> OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)
>
>Reporter: James Thomas
>Priority: Major
>
> I have found that when installing a clean MCF instance I cannot get
> Documentum repository connectors to connect to Documentum until I have added
> this line to the processes/documentum-server/run.sh script before the call to
> Java:
>
> {code:java}
> CLASSPATH="$CLASSPATH""$PATHSEP""$DOCUMENTUM"{code}
> Until I do this, attempts to save the connector will result in this output to
> the console:
>
> {noformat}
> 4 [RMI TCP Connection(2)-127.0.0.1] ERROR
> com.documentum.fc.common.impl.preferences.PreferencesManager -
> [DFC_PREFERENCE_LOAD_FAILED] Failed to load persistent preferences from null
> java.io.FileNotFoundException: dfc.properties
> at
> com.documentum.fc.common.impl.preferences.PreferencesManager.locateMainPersistentStore(PreferencesManager.java:378)
> at
> com.documentum.fc.common.impl.preferences.PreferencesManager.readPersistentProperties(PreferencesManager.java:329)
> at
> com.documentum.fc.common.impl.preferences.PreferencesManager.(PreferencesManager.java:37)
> at
> com.documentum.fc.common.DfPreferences.initialize(DfPreferences.java:64)
> ..{noformat}
> and this message in the MCF UI:
>
> {noformat}
> Connection failed: Documentum error: No DocBrokers are configured{noformat}
>
>
> I mentioned this in #1512 for MCF 2.10 but it got lost in the other work done
> in that ticket. While setting up 2.11 from scratch I encountered it again.
>
> Once I have edited the run.sh script I get this in the console, showing that
> (for whatever reason) the change is significant:
>
> {noformat}
> Reading DFC configuration from
> "file:/opt/manifold/apache-manifoldcf-2.11/processes/documentum-server/dfc.properties"
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722781#comment-16722781
]
James Thomas commented on CONNECTORS-1521:
--
{quote}[~jamesthomas] Any update on this ticket? I'm moving it to 2.13...
{quote}
I submitted a ticket but didn't get a useful answer, I'm afraid.
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.13
>
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT=09018000e515
> drl?versionLabel=CURRENT=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
> r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
> {noformat}
> But if I set the time inside DST, the time is an hour before:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Sat Oct 27 00:00:06 CEST 2018
> r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
> {noformat}
> This is perhaps a Java issue rather than a logic issue in the connector? See
> e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1524) Documentum Connector fails with NullPointerException
[
https://issues.apache.org/jira/browse/CONNECTORS-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635195#comment-16635195
]
James Thomas commented on CONNECTORS-1524:
--
I have found another issue that manifests across Java versions.
If I start MCF 2.11 and Documentum processes using Java 9 or later I find that
attempting to view or edit a Documentum connector will result in this error
into the console:
{noformat}
3 [RMI TCP Connection(2)-127.0.0.1] ERROR
com.documentum.fc.common.DfPreferences - [DFC_PREFERENCE_BAD_VALUE] Bad value
for preference "dfc.date_format", value="M/d/, h:mm:ss a"
com.documentum.fc.common.DfException: Illegal syntax found in the date format
'M/d/, h:mm:ss a'. The default localized short date format will be used.
{noformat}
I can work around this by adding this line to my dfc.properties:
{noformat}
dfc.date_format=.MM.dd HH:mm:s{noformat}
I don't see this issue with Java(TM) SE Runtime Environment (build
1.8.0_181-b13)
> Documentum Connector fails with NullPointerException
>
>
> Key: CONNECTORS-1524
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1524
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: 500_server_error.txt
>
>
> I am running ManifoldCF 2.10 patched for issues 1512 and 1517.
> I recently upgraded Java to 10.0.2 (for another server on the same machine)
> and find that when I try to create jobs using the Documentum connector,
> clicking on (e.g.) the Paths tab will give this error dialog:
> {code:java}
> HTTP ERROR 500
> Problem accessing /mcf-crawler-ui/execute.jsp. Reason:
> Server Error
> Caused by:
> org.apache.jasper.JasperException: An exception occurred processing JSP page
> /execute.jsp at line 1542
> 1539: {
> 1540: threadContext.save("JobObject",job);
> 1541: %>
> 1542:
> 1543: <%
> 1544: }
> 1545: else if (op.equals("Save")){code}
> There's a lot of stack trace (which I'll attach) but it includes some
> ManifoldCF calls:
>
> {code:java}
> Caused by: java.lang.RuntimeException: Unexpected exception type:
> java.lang.NullPointerException: null
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$GetSessionThread.finishUp(DCTM.java:117)
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM.getSession(DCTM.java:173)
> ...
> Caused by: java.lang.NullPointerException
> at
> com.documentum.fc.client.impl.docbase.DocbaseDateFormat.getSeparator(DocbaseDateFormat.java:119)
> at ...
> {code}
> If I revert to OpenJDK 1.8.0_171 (the version I upgraded from) the issue is
> not seen.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CONNECTORS-1535) Documentum Connector cannot find dfc.properties
James Thomas created CONNECTORS-1535:
Summary: Documentum Connector cannot find dfc.properties
Key: CONNECTORS-1535
URL: https://issues.apache.org/jira/browse/CONNECTORS-1535
Project: ManifoldCF
Issue Type: Bug
Components: Documentum connector
Affects Versions: ManifoldCF 2.10, ManifoldCF 2.11
Environment: Manifold 2.11
CentOS Linux release 7.5.1804 (Core)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)
Reporter: James Thomas
I have found that when installing a clean MCF instance I cannot get Documentum
repository connectors to connect to Documentum until I have added this line to
the processes/documentum-server/run.sh script before the call to Java:
{code:java}
CLASSPATH="$CLASSPATH""$PATHSEP""$DOCUMENTUM"{code}
Until I do this, attempts to save the connector will result in this output to
the console:
{noformat}
4 [RMI TCP Connection(2)-127.0.0.1] ERROR
com.documentum.fc.common.impl.preferences.PreferencesManager -
[DFC_PREFERENCE_LOAD_FAILED] Failed to load persistent preferences from null
java.io.FileNotFoundException: dfc.properties
at
com.documentum.fc.common.impl.preferences.PreferencesManager.locateMainPersistentStore(PreferencesManager.java:378)
at
com.documentum.fc.common.impl.preferences.PreferencesManager.readPersistentProperties(PreferencesManager.java:329)
at
com.documentum.fc.common.impl.preferences.PreferencesManager.(PreferencesManager.java:37)
at
com.documentum.fc.common.DfPreferences.initialize(DfPreferences.java:64)
..{noformat}
and this message in the MCF UI:
{noformat}
Connection failed: Documentum error: No DocBrokers are configured{noformat}
I mentioned this in #1512 for MCF 2.10 but it got lost in the other work done
in that ticket. While setting up 2.11 from scratch I encountered it again.
Once I have edited the run.sh script I get this in the console, showing that
(for whatever reason) the change is significant:
{noformat}
Reading DFC configuration from
"file:/opt/manifold/apache-manifoldcf-2.11/processes/documentum-server/dfc.properties"
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633624#comment-16633624
]
James Thomas commented on CONNECTORS-1532:
--
Good news.
The repro now produces these two results of 'ls -l', and corresponding log
messages:
{noformat}
-rw-r--r--. 1 root root 27 Oct 1 07:31
drl?versionLabel=CURRENT=09018000fc6e
-rw-r--r--. 1 root root 0 Oct 1 07:33
drl?versionLabel=CURRENT=09018000fc6e
INFO 2018-10-01T05:31:28.584Z (Startup thread) - Preparing non-continuous
non-partial, either MODEL_ALL or fromBeginningOfTime, 1538299958323 for run:
prepareFullScan
INFO 2018-10-01T05:32:58.717Z (Startup thread) - Preparing incremental scan for
1538299958323: prepareIncrementalScan{noformat}
I have also sanity-checked that deletion behaves as it did before these changes.
[~kwri...@metacarta.com] in comment-1665 you mentioned that you might need
to add an additional check. Does that need to be done? If so, in this ticket?
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, CONNECTORS-1532.patch, DCTM.java,
> logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633371#comment-16633371
]
James Thomas commented on CONNECTORS-1532:
--
I applied DCTM.java, rebuilt, patched my installation, restarted all processes,
and re-did the repro.
Again, the file sizes remain unchanged across the two runs:
{noformat}
INFO 2018-09-30T13:14:35.380Z (Startup thread) - Preparing non-continuous
non-partial, either MODEL_ALL or fromBeginningOfTime, 1538299958323 for run:
prepareFullScan
INFO 2018-09-30T13:17:29.020Z (Startup thread) - Preparing incremental scan for
1538299958323: prepareIncrementalScan
-rw-r--r--. 1 root root 27 Sep 30 15:14
drl?versionLabel=CURRENT=09018000fc6d
-rw-r--r--. 1 root root 27 Sep 30 15:17
drl?versionLabel=CURRENT=09018000fc6d{noformat}
There is stdout from the first run:
{noformat}
All mime types allowed
Found '/Administrator/45523' in specified set; match!{noformat}
and the second:
{noformat}
All mime types allowed
Did not find any matching paths{noformat}
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, CONNECTORS-1532.patch, DCTM.java,
> logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
>
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1665#comment-1665
]
James Thomas commented on CONNECTORS-1532:
--
{quote}I would also like to know if it is possible to change the mime type of a
document after it has been created; if so, a check also needs to be added for
that situation.
{quote}
[~kwri...@metacarta.com] I believe that it is possible to do this, assuming
that the MIME type is what is shown in the Format field for a file when viewing
it in DM's Documentum Administrator.
To change that:
* right-click on the file and choose Properties
* Check Show all Properties
* Search for "format" in the page
* Edit the field associated with the format
* Click OK
The format field value has now changed
If there is a specific query I can run for you to check that this corresponds
to whatever the DM connector is looking for, please say.
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, CONNECTORS-1532.patch,
> logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1663#comment-1663
]
James Thomas commented on CONNECTORS-1532:
--
{quote}I have attached a patch for you to try. Please let me know if it
addresses the folder issue
{quote}
[~kwri...@metacarta.com] I don't see any significant change in behaviour using
the same repro steps as comment-16623888.
The file is still shown on the file system at full size after the second run of
the job. Here are the log and file system details of this run:
{code:java}
INFO 2018-09-30T09:34:38.276Z (Startup thread) - Preparing non-continuous
non-partial, either MODEL_ALL or fromBeginningOfTime, 1538299958323 for run:
prepareFullScan
INFO 2018-09-30T09:35:58.444Z (Startup thread) - Preparing incremental scan for
1538299958323: prepareIncrementalScan
-rw-r--r--. 1 root root 27 Sep 30 11:34
drl?versionLabel=CURRENT=09018000fc6b
-rw-r--r--. 1 root root 27 Sep 30 11:36
drl?versionLabel=CURRENT=09018000fc6b{code}
I went on and repeated the repro in the same state just to see what might
happen (essentially the same), then I reset seeding on the job and ran it
again. Here's the file system and logs for that:
{code:java}
INFO 2018-09-30T09:48:51.303Z (Startup thread) - Preparing incremental scan
for 1538299958323: prepareIncrementalScan
INFO 2018-09-30T09:50:06.483Z (Startup thread) - Preparing incremental scan
for 1538299958323: prepareIncrementalScan
INFO 2018-09-30T09:51:06.800Z (Startup thread) - Preparing non-continuous
non-partial, either MODEL_ALL or fromBeginningOfTime, 1538299958323 for run:
prepareFullScan
$ ### after adding another file
-rw-r--r--. 1 root root 27 Sep 30 11:36
drl?versionLabel=CURRENT=09018000fc6b
-rw-r--r--. 1 root root 27 Sep 30 11:49
drl?versionLabel=CURRENT=09018000fc6c
$ ### after running the job again
-rw-r--r--. 1 root root 27 Sep 30 11:36
drl?versionLabel=CURRENT=09018000fc6b
-rw-r--r--. 1 root root 27 Sep 30 11:50
drl?versionLabel=CURRENT=09018000fc6c
$ ## after resetting seeding and running the job
-rw-r--r--. 1 root root 0 Sep 30 11:51
drl?versionLabel=CURRENT=09018000fc6b
-rw-r--r--. 1 root root 0 Sep 30 11:51
drl?versionLabel=CURRENT=09018000fc6c
{code}
So it appears that reseeding can give the desired outcome.
FYI, I applied this patch on top of the logging patch from this ticket, which
is itself on top of 2.10 patched for #1512, #1517:
{code:java}
$ wget
https://issues.apache.org/jira/secure/attachment/12940883/CONNECTORS-1532.patch
$ dos2unix CONNECTORS-1532.patch
$ dos2unix
connectors/documentum/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/DCTM/DCTM.java
$ patch -p0 -i CONNECTORS-1532.patch
$ ant build
$ find dist -type f -exec ls -l {} \; > /tmp/diff{code}
Here's the set of files that changed after I applied your patch:
{code:java}
$ grep Sep\ 30 /tmp/diff | grep jar
-rw-rw-r-- 1 james staff 12544 Sep 30 10:01
dist/connector-lib/mcf-documentum-connector-rmistub.jar
-rw-rw-r-- 1 james staff 100864 Sep 30 10:01
dist/connector-lib/mcf-documentum-connector.jar
-rw-rw-r-- 1 james staff 6292 Sep 30 10:01
dist/connector-lib/mcf-filenet-connector-rmistub.jar
-rw-rw-r-- 1 james staff 3916082 Sep 30 10:02
dist/connector-lib/mcf-meridio-connector.jar
-rw-rw-r-- 1 james staff 838582 Sep 30 10:02
dist/connector-lib/mcf-sharepoint-connector.jar
-rw-rw-r-- 1 james staff 8567 Sep 30 10:01
dist/processes/documentum-server/lib/mcf-documentum-connector-rmiskel.jar
-rw-rw-r-- 1 james staff 4494 Sep 30 10:01
dist/processes/filenet-server/lib/mcf-filenet-connector-rmiskel.jar
{code}
I stopped my MFC instance and the DM processes, applied the changed files, and
restarted DM processes and MFC server. Then attempted the repro I described
above.
It's possible that I haven't applied the patch correctly. Is there something I
can do to check?
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, CONNECTORS-1532.patch,
> logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file
[jira] [Commented] (CONNECTORS-1534) File System connector "deletes" files by making them zero bytes
[
https://issues.apache.org/jira/browse/CONNECTORS-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624497#comment-16624497
]
James Thomas commented on CONNECTORS-1534:
--
{quote}For backwards compatibility reasons, you'd therefore want to add
specification information to the connector describing how you want deletions to
be handled, which is basically a significant enhancement to the connector, so I
would ask if this is really something you need or just something you don't like.
{quote}
I find it (very) unexpected but it's not breaking anything for me.
Given what I know, I think that from the consumer of the FS connector's output
it's (at best) ambiguous whether the file originated at zero size or existed
and was then deleted. This may make a difference in some applications.
At the moment – as you know :) – I'm primiarly working with the Documentum
repository connector and it won't fetch zero size files
(https://issues.apache.org/jira/browse/CONNECTORS-1526) so I don't have the
ambiguity. I'm also only really using the FS connector as a tool for testing so
far.
> File System connector "deletes" files by making them zero bytes
> ---
>
> Key: CONNECTORS-1534
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1534
> Project: ManifoldCF
> Issue Type: Bug
> Components: File system connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF next
>
>
> In https://issues.apache.org/jira/browse/CONNECTORS-1532 I observed that
> deleting a file from a repository does not (reliably?) delete it from a File
> System output. Instead the output file remains but is given zero size and a
> new modification time.
> I would expect that a file deletion in the repository results in file
> deletion in the output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624487#comment-16624487
]
James Thomas commented on CONNECTORS-1532:
--
OK. Thanks for continuing to pursue this.
I have made a new ticket for the File System connector apparently zeroing files
instead of deleting them, https://issues.apache.org/jira/browse/CONNECTORS-1534
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:25
> drl?versionLabel=CURRENT=09018000f7c4{code}
> In DM, I move a file from another folder to F and it shows up with the
> timestamp of the move (07:28):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root
[jira] [Created] (CONNECTORS-1534) File System connector "deletes" files by making them zero bytes
James Thomas created CONNECTORS-1534: Summary: File System connector "deletes" files by making them zero bytes Key: CONNECTORS-1534 URL: https://issues.apache.org/jira/browse/CONNECTORS-1534 Project: ManifoldCF Issue Type: Bug Components: File system connector Affects Versions: ManifoldCF 2.10 Reporter: James Thomas In https://issues.apache.org/jira/browse/CONNECTORS-1532 I observed that deleting a file from a repository does not (reliably?) delete it from a File System output. Instead the output file remains but is given zero size and a new modification time. I would expect that a file deletion in the repository results in file deletion in the output. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624105#comment-16624105
]
James Thomas commented on CONNECTORS-1532:
--
The data is for two runs of the job, as in the repro steps: there are two runs
of 'ls -l' and two log messages. (Or perhaps I'm misunderstanding your question)
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:25
> drl?versionLabel=CURRENT=09018000f7c4{code}
> In DM, I move a file from another folder to F and it shows up with the
> timestamp of the move (07:28):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
>
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623888#comment-16623888
]
James Thomas commented on CONNECTORS-1532:
--
[~kwri...@metacarta.com] I had some difficulty getting your patch to show
anything in the log file until I'd added an entry to the properties.xml file.
Here's the config, the manifoldcf.log lines and the file system timestamps:
{noformat}
{noformat}
{noformat}
INFO 2018-09-21T16:51:08.645Z (Startup thread) - Preparing non-continuous
non-partial, either MODEL_ALL or fromBeginningOfTime, 1537332478051 for run:
prepareFullScan
INFO 2018-09-21T16:55:48.781Z (Startup thread) - Preparing incremental scan
for 1537332478051: prepareIncrementalScan
{noformat}
{noformat}
-rw-r--r--. 1 root i2e 27 Sep 21 18:51
drl?versionLabel=CURRENT=09018000fc6a
-rw-r--r--. 1 root i2e 27 Sep 21 18:55
drl?versionLabel=CURRENT=09018000fc6a
{noformat}
My repro was as earlier:
* In DM I created a folder F containing a single file X
* In MF I configured a job to look at F and output to File System
* In MF I ran the job
* On the File System I ran 'ls -l' and found F
* In DM I moved X to a folder outside of F (Right click, add to clipboard, go
to new folder, Edit> Move here).
* In MF I ran the job again
* On the File System I ran 'ls' again
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
>
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622188#comment-16622188
]
James Thomas commented on CONNECTORS-1532:
--
[~kwri...@metacarta.com] yep, I've patched and it's building right now.
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png, logging_patch.diff
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:25
> drl?versionLabel=CURRENT=09018000f7c4{code}
> In DM, I move a file from another folder to F and it shows up with the
> timestamp of the move (07:28):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
>
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621894#comment-16621894
]
James Thomas commented on CONNECTORS-1532:
--
DQL experiment:
* In DM I created a folder F containing a single file X
* In DM's DQL too (From Document Administrator> Tools> Dql Editor), I ran the
query you gave
* An object Id was returned
* I confirmed that was the ObjectId of X (via right-click> Properties on X)
* In DM I moved X to a folder outside of F (By right click, add to clipboard,
go to new folder, Edit> Move here).
* In DM's DQL tool, I ran the query you gave
* No object Id was returned
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
> Attachments: 2018-09-19_1758.png
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:25
> drl?versionLabel=CURRENT=09018000f7c4{code}
> In DM, I move a file from another folder to F and it shows up with the
>
[jira] [Commented] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621700#comment-16621700
]
James Thomas commented on CONNECTORS-1521:
--
{quote}I didn't have the necessary credentials to open support tickets with
OpenText, but I've organised that now, so I'll go ahead and do it.
{quote}
Getting the appropriate credentials is taking much longer than we initially
thought it would.
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.12
>
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT=09018000e515
> drl?versionLabel=CURRENT=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
> r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
> {noformat}
> But if I set the time inside DST, the time is an hour before:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Sat Oct 27 00:00:06 CEST 2018
> r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
> {noformat}
> This is perhaps a Java issue rather than a logic issue in the connector? See
> e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621531#comment-16621531
]
James Thomas commented on CONNECTORS-1532:
--
{quote}I may have found the reason you see this behavior, though. If the folder
affinity is versioned information, and I believe it is, then the seeding query
will pick up the last version of the document that was in the right folder.
That's because the seeding query uses the chronicle_id, which is really a
specific document version:
{quote}
Thanks, is this consistent with my observation that re-seeding explicitly (from
the Job's View page) forces the change to be recognised?
Is there some logging I can provide or other experiment I can do that will help
you to narrow this down further or confirm your thoughts?
Also, this may be an unrelated question, and I can take it to a different
ticket if you like, but why does the File System connector zero these files
rather than deleting them from disk?
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Attachments: 2018-09-19_1758.png
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620854#comment-16620854
]
James Thomas commented on CONNECTORS-1532:
--
It's a "scan every document once" job, and I'm running by clicking Start.
!2018-09-19_1758.png!
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Attachments: 2018-09-19_1758.png
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:25
> drl?versionLabel=CURRENT=09018000f7c4{code}
> In DM, I move a file from another folder to F and it shows up with the
> timestamp of the move (07:28):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
>
[jira] [Updated] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Thomas updated CONNECTORS-1532:
-
Attachment: 2018-09-19_1758.png
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Attachments: 2018-09-19_1758.png
>
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:25
> drl?versionLabel=CURRENT=09018000f7c4{code}
> In DM, I move a file from another folder to F and it shows up with the
> timestamp of the move (07:28):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620830#comment-16620830
]
James Thomas commented on CONNECTORS-1532:
--
Having not touched anything since I reported the cut-down experiment, I re-ran
the job (I am not sure this is what you mean by a complete job?)
The File System view did not change, and the Simple History Report has a new
entry:
{code:java}
-rw-r--r--. 1 root i2e 26 Sep 19 17:46
drl?versionLabel=CURRENT=09018000f99a{code}
||Start Time||Activity||Identifier||Result Code||Bytes||Time||Result
Description||
|09-19-2018 16:30:47.179|fetch|09018000f99a|OK|26|26| |
I then reset seeding on the job and re-ran it.
There is no additional History, but this time the file is zeroed:
{code:java}
-rw-r--r--. 1 root i2e 0 Sep 19 18:36
drl?versionLabel=CURRENT=09018000f99a{code}
So, from my empirical data, I think that a move in DM requires reseeding before
the output is synchronised with the repository? Does that fit with what you
were saying?
You mentioned that continuous jobs would not delete - do you mean that they
won't delete until the job ends, or until the next reseed, if a reseed interval
is specified?
This may be an unrelated question, and I can take it to a different ticket if
you like, but why does the File System connector zero files rather than
deleting them from disk?
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19
[jira] [Commented] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620779#comment-16620779
]
James Thomas commented on CONNECTORS-1532:
--
[~kwri...@metacarta.com], here's a simple experiment cut down from the original
which I hope is what you're after.
* In DM I created a folder F containing a single file X
* In MF I configured a job to look at F and output to File System
* In MF I ran the job
* On the File System I ran 'ls -l' and found F
* In DM I moved X to a folder outside of F (Right click, add to clipboard, go
to new folder, Edit> Move here).
* In MF I ran the job again
* On the File System I ran 'ls' again
* In MF I ran Simple History Report
(In the data below, note that the time stamps are two hours different because
of the time zone setup I have currently to work around #1521.)
Two calls to 'ls':
{code:java}
-rw-r--r--. 1 root i2e 26 Sep 19 17:44
drl?versionLabel=CURRENT=09018000f99a
-rw-r--r--. 1 root i2e 26 Sep 19 17:46
drl?versionLabel=CURRENT=09018000f99a
{code}
Simple History Report for object 09018000f99a:
||Start Time||Activity||Identifier||Result Code||Bytes||Time||Result
Description||
|09-19-2018 15:46:04.940|fetch|09018000f99a|OK|26|284| |
|09-19-2018 15:44:30.228|fetch|09018000f99a|OK|26|34| |
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep
[jira] [Updated] (CONNECTORS-1532) Moving a file outside of the job's Paths is not the same as deleting it
[
https://issues.apache.org/jira/browse/CONNECTORS-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Thomas updated CONNECTORS-1532:
-
Summary: Moving a file outside of the job's Paths is not the same as
deleting it (was: Moving a file outside of the jobs Paths is not the same as
deleting it)
> Moving a file outside of the job's Paths is not the same as deleting it
> ---
>
> Key: CONNECTORS-1532
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
> Environment: Manifold 2.10 patched for #1512, #1517
>Reporter: James Thomas
>Priority: Major
>
> If I have a MF job which is connecting a specific folder, F, in Documentum to
> a File System output then:
> 1. deleting files in Documentum shows them as zero size in the file system
> 2. moving files out of F does not remove them or zero them in the file system
> Note that moving a file from another folder (which the job is not looking at)
> to F has the same effect as adding it to F by e.g. importing it in DM or
> POSTing it to DM via the REST interface.
> Intuitively, I expect that moving a file out of the "view" of the Documentum
> connector would have the same effect on the File System as deleting it. (My
> model here is of MF synchronising content between the Paths (DM) and the
> Output Path (File System) that I have specified in the job.)
> Starting point, I have run the MF job to fetch a bunch of files from a folder
> - call it F - in DM (i.e. I have configured Paths in the job to be F). This
> is what 'ls -l' on the file system looks like:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 85772 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf{code}
> In DM, I delete one of the files in F and it shows as zero size, and the
> modification date has changed:
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c4
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7{code}
> In DM, I move a file from F to another folder. (Right click, add to
> clipboard, go to new folder, Edit> Move here).
> The file shows as modified (07:25), but is still apparently in F (i.e. in the
> Path my MF job is looking at):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7c1
> -rw-r--r--. 1 root i2e 26112 Sep 19 07:22
> drl?versionLabel=CURRENT=09018000f7bf
> -rw-r--r--. 1 root i2e 0 Sep 19 07:23
> drl?versionLabel=CURRENT=09018000f7c7
> -rw-r--r--. 1 root i2e 32783 Sep 19 07:25
> drl?versionLabel=CURRENT=09018000f7c4{code}
> In DM, I move a file from another folder to F and it shows up with the
> timestamp of the move (07:28):
> {code:java}
> -rw-r--r--. 1 root i2e 12541 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c0
> -rw-r--r--. 1 root i2e 26 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7be
> -rw-r--r--. 1 root i2e 8790 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c2
> -rw-r--r--. 1 root i2e 101888 Sep 19 07:21
> drl?versionLabel=CURRENT=09018000f7c3
> -rw-r--r--. 1 root i2e 23040 Sep 19 07:22
>
[jira] [Created] (CONNECTORS-1532) Moving a file outside of the jobs Paths is not the same as deleting it
James Thomas created CONNECTORS-1532:
Summary: Moving a file outside of the jobs Paths is not the same
as deleting it
Key: CONNECTORS-1532
URL: https://issues.apache.org/jira/browse/CONNECTORS-1532
Project: ManifoldCF
Issue Type: Bug
Components: Documentum connector
Affects Versions: ManifoldCF 2.10
Environment: Manifold 2.10 patched for #1512, #1517
Reporter: James Thomas
If I have a MF job which is connecting a specific folder, F, in Documentum to a
File System output then:
1. deleting files in Documentum shows them as zero size in the file system
2. moving files out of F does not remove them or zero them in the file system
Note that moving a file from another folder (which the job is not looking at)
to F has the same effect as adding it to F by e.g. importing it in DM or
POSTing it to DM via the REST interface.
Intuitively, I expect that moving a file out of the "view" of the Documentum
connector would have the same effect on the File System as deleting it. (My
model here is of MF synchronising content between the Paths (DM) and the Output
Path (File System) that I have specified in the job.)
Starting point, I have run the MF job to fetch a bunch of files from a folder -
call it F - in DM (i.e. I have configured Paths in the job to be F). This is
what 'ls -l' on the file system looks like:
{code:java}
-rw-r--r--. 1 root i2e 12541 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c0
-rw-r--r--. 1 root i2e 26 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7be
-rw-r--r--. 1 root i2e 85772 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c7
-rw-r--r--. 1 root i2e 8790 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c2
-rw-r--r--. 1 root i2e 101888 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c3
-rw-r--r--. 1 root i2e 32783 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c4
-rw-r--r--. 1 root i2e 23040 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7c1
-rw-r--r--. 1 root i2e 26112 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7bf{code}
In DM, I delete one of the files in F and it shows as zero size, and the
modification date has changed:
{code:java}
-rw-r--r--. 1 root i2e 12541 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c0
-rw-r--r--. 1 root i2e 26 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7be
-rw-r--r--. 1 root i2e 8790 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c2
-rw-r--r--. 1 root i2e 101888 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c3
-rw-r--r--. 1 root i2e 32783 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c4
-rw-r--r--. 1 root i2e 23040 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7c1
-rw-r--r--. 1 root i2e 26112 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7bf
-rw-r--r--. 1 root i2e 0 Sep 19 07:23
drl?versionLabel=CURRENT=09018000f7c7{code}
In DM, I move a file from F to another folder. (Right click, add to clipboard,
go to new folder, Edit> Move here).
The file shows as modified (07:25), but is still apparently in F (i.e. in the
Path my MF job is looking at):
{code:java}
-rw-r--r--. 1 root i2e 12541 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c0
-rw-r--r--. 1 root i2e 26 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7be
-rw-r--r--. 1 root i2e 8790 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c2
-rw-r--r--. 1 root i2e 101888 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c3
-rw-r--r--. 1 root i2e 23040 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7c1
-rw-r--r--. 1 root i2e 26112 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7bf
-rw-r--r--. 1 root i2e 0 Sep 19 07:23
drl?versionLabel=CURRENT=09018000f7c7
-rw-r--r--. 1 root i2e 32783 Sep 19 07:25
drl?versionLabel=CURRENT=09018000f7c4{code}
In DM, I move a file from another folder to F and it shows up with the
timestamp of the move (07:28):
{code:java}
-rw-r--r--. 1 root i2e 12541 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c0
-rw-r--r--. 1 root i2e 26 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7be
-rw-r--r--. 1 root i2e 8790 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c2
-rw-r--r--. 1 root i2e 101888 Sep 19 07:21
drl?versionLabel=CURRENT=09018000f7c3
-rw-r--r--. 1 root i2e 23040 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7c1
-rw-r--r--. 1 root i2e 26112 Sep 19 07:22
drl?versionLabel=CURRENT=09018000f7bf
-rw-r--r--. 1 root i2e 0 Sep 19 07:23
drl?versionLabel=CURRENT=09018000f7c7
-rw-r--r--. 1 root i2e 32783 Sep 19 07:25
drl?versionLabel=CURRENT=09018000f7c4
-rw-r--r--. 1 root i2e 191513 Sep 19 07:28
drl?versionLabel=CURRENT=0901800045b9{code}
But if I immediately move it out in DM then, again, the timestamp (07:30)
alters but the file apparently remains:
{code:java}
-rw-r--r--. 1 root i2e 12541 Sep
[jira] [Commented] (CONNECTORS-1517) Documentum Connector uses different "unconstrained" a_content_type filters depending on whether the Content Types tab has been edited
[
https://issues.apache.org/jira/browse/CONNECTORS-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578610#comment-16578610
]
James Thomas commented on CONNECTORS-1517:
--
{quote}Can you be more explicit? There are many conditionally enabled options
in many connectors under MCF.
{quote}
Yes, I was thinking specifically of the kinds of things I mentioned earlier:
* I wonder whether "No content type restriction" checked should disable all of
the others?
* It is surprising to be able to submit with no checkboxes checked
* Generalising the above, I think I'd prefer to see some more restriction on
what combinations of box can be checked at the same time.
> Documentum Connector uses different "unconstrained" a_content_type filters
> depending on whether the Content Types tab has been edited
> -
>
> Key: CONNECTORS-1517
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1517
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1517-2.patch, CONNECTORS-1517.patch, Notes.txt
>
>
> I am using Manifold 2.10 patched for issue
> https://issues.apache.org/jira/browse/CONNECTORS-1512
> I find that the "unconstrained" query submitted to Documentum differs
> depending on whether the Content Types in the job have been edited or not.
> This can dramatically affect which files are fetched. After editing, there
> are likely to be fewer.
> For example, having simply created a job connecting to DM and setting only
> the Paths value to Administrator/james the following request is generated.
> (Taken from manifoldcf.log).
> Note that there are no a_content_type constraints (and my line break for
> readibility):
> {code:java}
> DEBUG 2018-07-26T05:52:56,422 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:52:56','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> Once the Content Types tab has been edited (e.g. to remove the 123w type) it
> looks like this, i.e. the search constrains to only the selected types (my
> ellipsis for readibility):
> {code:java}
> DEBUG 2018-07-26T05:58:36,755 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:58:36','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('acad', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> If the 123w type is now reselected in the Content Types tab, the search adds
> it to the list of a_content_type entries, but doesn't return to the
> unconstrained initial search:
> {code:java}
> DEBUG 2018-07-26T05:59:16,863 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:59:16','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('123w', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> This means that running what appears to be an equivalent job several times
> may not fetch the same set of documents from Documentum.
> I expect that the same configuration in the UI produces the same search to
> Documentum, regardless of how the configuration was arrived at.
> If the selected items in the Content Types list is treated as the only set of
> files to fetch (i,.e. the initial unconstrained search is considered
> incorrect here) then I guess I might also like to have flexibility to fetch
> file types not on the checklist in the Content Types tab.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1517) Documentum Connector uses different "unconstrained" a_content_type filters depending on whether the Content Types tab has been edited
[
https://issues.apache.org/jira/browse/CONNECTORS-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578576#comment-16578576
]
James Thomas commented on CONNECTORS-1517:
--
Great to hear that backwards compatibility has been covered.
I agree that there's a functional correctness case here: the behaviour _is_
what has been configured. I guess I'm interested (with a user head on) in it
being hard to configure something that isn't useful.
Forgive me if this is a stupid question, but is there much scope for
conditional enabling/disabling of options or client-side validation in
ManifoldCF's connectors as things stand?
> Documentum Connector uses different "unconstrained" a_content_type filters
> depending on whether the Content Types tab has been edited
> -
>
> Key: CONNECTORS-1517
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1517
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1517-2.patch, CONNECTORS-1517.patch, Notes.txt
>
>
> I am using Manifold 2.10 patched for issue
> https://issues.apache.org/jira/browse/CONNECTORS-1512
> I find that the "unconstrained" query submitted to Documentum differs
> depending on whether the Content Types in the job have been edited or not.
> This can dramatically affect which files are fetched. After editing, there
> are likely to be fewer.
> For example, having simply created a job connecting to DM and setting only
> the Paths value to Administrator/james the following request is generated.
> (Taken from manifoldcf.log).
> Note that there are no a_content_type constraints (and my line break for
> readibility):
> {code:java}
> DEBUG 2018-07-26T05:52:56,422 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:52:56','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> Once the Content Types tab has been edited (e.g. to remove the 123w type) it
> looks like this, i.e. the search constrains to only the selected types (my
> ellipsis for readibility):
> {code:java}
> DEBUG 2018-07-26T05:58:36,755 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:58:36','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('acad', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> If the 123w type is now reselected in the Content Types tab, the search adds
> it to the list of a_content_type entries, but doesn't return to the
> unconstrained initial search:
> {code:java}
> DEBUG 2018-07-26T05:59:16,863 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:59:16','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('123w', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> This means that running what appears to be an equivalent job several times
> may not fetch the same set of documents from Documentum.
> I expect that the same configuration in the UI produces the same search to
> Documentum, regardless of how the configuration was arrived at.
> If the selected items in the Content Types list is treated as the only set of
> files to fetch (i,.e. the initial unconstrained search is considered
> incorrect here) then I guess I might also like to have flexibility to fetch
> file types not on the checklist in the Content Types tab.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1517) Documentum Connector uses different "unconstrained" a_content_type filters depending on whether the Content Types tab has been edited
[
https://issues.apache.org/jira/browse/CONNECTORS-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578357#comment-16578357
]
James Thomas commented on CONNECTORS-1517:
--
Hi Karl, I have now had a chance to try out the patches. I'll attach a
transcript which shows the queries executed (from manifoldcf.log) when I ran a
job with particular configuration in the Content Types tab of the Documentum
Connector.
My observations and thoughts:
* The core bug that I reported - that editing the Content Types tab and then
resetting it results in different semantics at search time appears fixed.
* The default search is still unconstrained.
* It is surprising to be able to have both "No content type restriction" and
any other checkbox checked at the same time. I wonder whether "No content type
restriction" checked should disable all of the others?
* It is surprising to be able to submit with no checkboxes checked and have a
query constrained with "1<0" as it looks like this can never succeed
* Generalising the above, I think I'd prefer to see some more restriction on
what combinations of box can be checked at the same time.
* It would be convenient as a user to have a control for check all/uncheck all
* I haven't been using ManifoldCF/Documentum long enough to know whether there
are likely to be backwards compatibility issues in changing the UI this way
[^Notes.txt]
> Documentum Connector uses different "unconstrained" a_content_type filters
> depending on whether the Content Types tab has been edited
> -
>
> Key: CONNECTORS-1517
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1517
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1517-2.patch, CONNECTORS-1517.patch, Notes.txt
>
>
> I am using Manifold 2.10 patched for issue
> https://issues.apache.org/jira/browse/CONNECTORS-1512
> I find that the "unconstrained" query submitted to Documentum differs
> depending on whether the Content Types in the job have been edited or not.
> This can dramatically affect which files are fetched. After editing, there
> are likely to be fewer.
> For example, having simply created a job connecting to DM and setting only
> the Paths value to Administrator/james the following request is generated.
> (Taken from manifoldcf.log).
> Note that there are no a_content_type constraints (and my line break for
> readibility):
> {code:java}
> DEBUG 2018-07-26T05:52:56,422 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:52:56','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> Once the Content Types tab has been edited (e.g. to remove the 123w type) it
> looks like this, i.e. the search constrains to only the selected types (my
> ellipsis for readibility):
> {code:java}
> DEBUG 2018-07-26T05:58:36,755 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:58:36','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('acad', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> If the 123w type is now reselected in the Content Types tab, the search adds
> it to the list of a_content_type entries, but doesn't return to the
> unconstrained initial search:
> {code:java}
> DEBUG 2018-07-26T05:59:16,863 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:59:16','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('123w', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> This means that running what appears to be an equivalent job several times
> may not fetch the same set of documents from Documentum.
> I expect that the same configuration in the UI produces the same search to
> Documentum, regardless of how the configuration
[jira] [Updated] (CONNECTORS-1517) Documentum Connector uses different "unconstrained" a_content_type filters depending on whether the Content Types tab has been edited
[
https://issues.apache.org/jira/browse/CONNECTORS-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Thomas updated CONNECTORS-1517:
-
Attachment: Notes.txt
> Documentum Connector uses different "unconstrained" a_content_type filters
> depending on whether the Content Types tab has been edited
> -
>
> Key: CONNECTORS-1517
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1517
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1517-2.patch, CONNECTORS-1517.patch, Notes.txt
>
>
> I am using Manifold 2.10 patched for issue
> https://issues.apache.org/jira/browse/CONNECTORS-1512
> I find that the "unconstrained" query submitted to Documentum differs
> depending on whether the Content Types in the job have been edited or not.
> This can dramatically affect which files are fetched. After editing, there
> are likely to be fewer.
> For example, having simply created a job connecting to DM and setting only
> the Paths value to Administrator/james the following request is generated.
> (Taken from manifoldcf.log).
> Note that there are no a_content_type constraints (and my line break for
> readibility):
> {code:java}
> DEBUG 2018-07-26T05:52:56,422 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:52:56','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> Once the Content Types tab has been edited (e.g. to remove the 123w type) it
> looks like this, i.e. the search constrains to only the selected types (my
> ellipsis for readibility):
> {code:java}
> DEBUG 2018-07-26T05:58:36,755 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:58:36','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('acad', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> If the 123w type is now reselected in the Content Types tab, the search adds
> it to the list of a_content_type entries, but doesn't return to the
> unconstrained initial search:
> {code:java}
> DEBUG 2018-07-26T05:59:16,863 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:59:16','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('123w', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> This means that running what appears to be an equivalent job several times
> may not fetch the same set of documents from Documentum.
> I expect that the same configuration in the UI produces the same search to
> Documentum, regardless of how the configuration was arrived at.
> If the selected items in the Content Types list is treated as the only set of
> files to fetch (i,.e. the initial unconstrained search is considered
> incorrect here) then I guess I might also like to have flexibility to fetch
> file types not on the checklist in the Content Types tab.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1524) Documentum Connector fails with NullPointerException
[
https://issues.apache.org/jira/browse/CONNECTORS-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577054#comment-16577054
]
James Thomas commented on CONNECTORS-1524:
--
I'm prepared to believe that I might have a configuration error. However, I can
show this issue and remove it by switching between Java versions. (After more
experimentation, I can see that the issue manifests in at least Oracle Java 9
too.)
I'm not going to pursue this further in this ManifoldCF ticket at this time.
Instead I'll arrange for my different applications to use different Java
variants to work around it.
I can't find supported Java version information in OpenText's online material
so I've opened a support ticket there to ask the question.
> Documentum Connector fails with NullPointerException
>
>
> Key: CONNECTORS-1524
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1524
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: 500_server_error.txt
>
>
> I am running ManifoldCF 2.10 patched for issues 1512 and 1517.
> I recently upgraded Java to 10.0.2 (for another server on the same machine)
> and find that when I try to create jobs using the Documentum connector,
> clicking on (e.g.) the Paths tab will give this error dialog:
> {code:java}
> HTTP ERROR 500
> Problem accessing /mcf-crawler-ui/execute.jsp. Reason:
> Server Error
> Caused by:
> org.apache.jasper.JasperException: An exception occurred processing JSP page
> /execute.jsp at line 1542
> 1539: {
> 1540: threadContext.save("JobObject",job);
> 1541: %>
> 1542:
> 1543: <%
> 1544: }
> 1545: else if (op.equals("Save")){code}
> There's a lot of stack trace (which I'll attach) but it includes some
> ManifoldCF calls:
>
> {code:java}
> Caused by: java.lang.RuntimeException: Unexpected exception type:
> java.lang.NullPointerException: null
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$GetSessionThread.finishUp(DCTM.java:117)
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM.getSession(DCTM.java:173)
> ...
> Caused by: java.lang.NullPointerException
> at
> com.documentum.fc.client.impl.docbase.DocbaseDateFormat.getSeparator(DocbaseDateFormat.java:119)
> at ...
> {code}
> If I revert to OpenJDK 1.8.0_171 (the version I upgraded from) the issue is
> not seen.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1524) Documentum Connector fails with NullPointerException under OpenJDK 10
[
https://issues.apache.org/jira/browse/CONNECTORS-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576550#comment-16576550
]
James Thomas commented on CONNECTORS-1524:
--
Thanks, Karl.
For the record, it does manifest for me under Oracle Java 10.0.2, Java(TM) SE
Runtime Environment 18.3 (build 10.0.2+13).
There is no difference in the stack trace produced:
{code:java}
$ diff 500_server_error.txt 500_server_error_oracle_java.txt
$
{code}
I'll contact OpenText support.
> Documentum Connector fails with NullPointerException under OpenJDK 10
> -
>
> Key: CONNECTORS-1524
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1524
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: 500_server_error.txt
>
>
> I am running ManifoldCF 2.10 patched for issues 1512 and 1517.
> I recently upgraded Java to 10.0.2 (for another server on the same machine)
> and find that when I try to create jobs using the Documentum connector,
> clicking on (e.g.) the Paths tab will give this error dialog:
> {code:java}
> HTTP ERROR 500
> Problem accessing /mcf-crawler-ui/execute.jsp. Reason:
> Server Error
> Caused by:
> org.apache.jasper.JasperException: An exception occurred processing JSP page
> /execute.jsp at line 1542
> 1539: {
> 1540: threadContext.save("JobObject",job);
> 1541: %>
> 1542:
> 1543: <%
> 1544: }
> 1545: else if (op.equals("Save")){code}
> There's a lot of stack trace (which I'll attach) but it includes some
> ManifoldCF calls:
>
> {code:java}
> Caused by: java.lang.RuntimeException: Unexpected exception type:
> java.lang.NullPointerException: null
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$GetSessionThread.finishUp(DCTM.java:117)
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM.getSession(DCTM.java:173)
> ...
> Caused by: java.lang.NullPointerException
> at
> com.documentum.fc.client.impl.docbase.DocbaseDateFormat.getSeparator(DocbaseDateFormat.java:119)
> at ...
> {code}
> If I revert to OpenJDK 1.8.0_171 (the version I upgraded from) the issue is
> not seen.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CONNECTORS-1524) Documentum Connector fails with NullPointerException under OpenJDK 10
James Thomas created CONNECTORS-1524:
Summary: Documentum Connector fails with NullPointerException
under OpenJDK 10
Key: CONNECTORS-1524
URL: https://issues.apache.org/jira/browse/CONNECTORS-1524
Project: ManifoldCF
Issue Type: Bug
Components: Documentum connector
Affects Versions: ManifoldCF 2.10
Reporter: James Thomas
Attachments: 500_server_error.txt
I am running ManifoldCF 2.10 patched for issues 1512 and 1517.
I recently upgraded Java to 10.0.2 (for another server on the same machine) and
find that when I try to create jobs using the Documentum connector, clicking on
(e.g.) the Paths tab will give this error dialog:
{code:java}
HTTP ERROR 500
Problem accessing /mcf-crawler-ui/execute.jsp. Reason:
Server Error
Caused by:
org.apache.jasper.JasperException: An exception occurred processing JSP page
/execute.jsp at line 1542
1539: {
1540: threadContext.save("JobObject",job);
1541: %>
1542:
1543: <%
1544: }
1545: else if (op.equals("Save")){code}
There's a lot of stack trace (which I'll attach) but it includes some
ManifoldCF calls:
{code:java}
Caused by: java.lang.RuntimeException: Unexpected exception type:
java.lang.NullPointerException: null
at
org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$GetSessionThread.finishUp(DCTM.java:117)
at
org.apache.manifoldcf.crawler.connectors.DCTM.DCTM.getSession(DCTM.java:173)
...
Caused by: java.lang.NullPointerException
at
com.documentum.fc.client.impl.docbase.DocbaseDateFormat.getSeparator(DocbaseDateFormat.java:119)
at ...
{code}
If I revert to OpenJDK 1.8.0_171 (the version I upgraded from) the issue is not
seen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575081#comment-16575081
]
James Thomas commented on CONNECTORS-1521:
--
I didn't have the necessary credentials to open support tickets with OpenText,
but I've organised that now, so I'll go ahead and do it.
I don't think we should do a horrible hack :)
And, yes, the consistent timezone approach is the one I'd taken internally
after discovering this issue.
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT=09018000e515
> drl?versionLabel=CURRENT=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
> r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
> {noformat}
> But if I set the time inside DST, the time is an hour before:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Sat Oct 27 00:00:06 CEST 2018
> r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
> {noformat}
> This is perhaps a Java issue rather than a logic issue in the connector? See
> e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572701#comment-16572701
]
James Thomas commented on CONNECTORS-1521:
--
Another potential approach: is it possible to request the time of the DM server
(or perhaps parse it out of responses to other requests, as in HTTP headers?)
and use that in the DQL instead of local time?
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT=09018000e515
> drl?versionLabel=CURRENT=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
> r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
> {noformat}
> But if I set the time inside DST, the time is an hour before:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Sat Oct 27 00:00:06 CEST 2018
> r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
> {noformat}
> This is perhaps a Java issue rather than a logic issue in the connector? See
> e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571832#comment-16571832
]
James Thomas commented on CONNECTORS-1521:
--
No, there's no patterns with "Z"
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT=09018000e515
> drl?versionLabel=CURRENT=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
> r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
> {noformat}
> But if I set the time inside DST, the time is an hour before:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Sat Oct 27 00:00:06 CEST 2018
> r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
> {noformat}
> This is perhaps a Java issue rather than a logic issue in the connector? See
> e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571557#comment-16571557
]
James Thomas commented on CONNECTORS-1521:
--
(I am not a developer but ...) I have explored the Javadoc for
com.documentum.fc.common, Interface IDfTime in DFC 16.4
There are ~50 patterns, here's a couple of examples:
{noformat}
DF_TIME_PATTERN8 Represents time pattern string "month dd, ".
DF_TIME_PATTERN34 Represents time pattern string "mon- hh:mi:ss".
{noformat}
As far as I can see none of the patterns gives anything related to time zone or
locale, or mentions UTC, DST etc.
This is consistent with the blog post that you found.
Googling around a little, I came across these:
[https://stackoverflow.com/a/8315831] which looks related to your comment about
asString().
[https://msroth.wordpress.com/2011/05/15/the-content-server-and-time-zones|https://msroth.wordpress.com/2011/05/15/the-content-server-and-time-zones,m]
and [https://blog.documentum.pro/2015/02/04/time-in-documentum] which are in
this general area.
> Documentum Connector users ManifoldCF's local time in queries constraints
> against the Documentum server without reference to time zones
> ---
>
> Key: CONNECTORS-1521
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
>
> I find that the time/date constraints in queries to the Documentum server are
> based on the "raw" local time of the ManifoldCF server but appear to take no
> account of the time zones of the two servers.
> This can lead to recently modified files not being transferred to the output
> repository when you would naturally expect them to be. I'd like the times to
> be aligned, perhaps by including time zone in the query. In particular, is
> there a way to use UTC perhaps?
> Here's an example ...
> * create a folder in Documentum
> * set up a job to point at the folder and output to the file system
> * put two documents into a folder in Documentum
> * Select them, right click and export as CSV (to show the timestamps):
> {noformat}
> 1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
> 2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
> Check the local time on the ManifoldCF server machine. Observe that it's
> reporting consistent time with the DM server:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 09:07:25 BST 2018{noformat}
> Start the job and look for the query to Documentum in the manifoldcf.log file
> (line break added for readability):
> {noformat}
> DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
> AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
> ^C{noformat}
> Notice that the latest date asked for is *before* the modification date of
> the files added to DM. (And is an hour out, see footnote.)
>
> See whether anything has been output by the File System connector. It hasn't:
> {noformat}
> [james@manifold]$ ls /bigdisc/source/PDFs/timezones/
> [james@manifold]$
> {noformat}
> Now:
> * change the timezone on the ManifoldCF server machine
> * restart the ManifoldCF server and the Documentum processes
> * reseed the job
> Check the local time on the ManifoldCF server machine; it has changed:
> {noformat}
> [james@manifold]$ date
> Tue Aug 7 10:10:29 CEST 2018{noformat}
> Start the job again and notice that the query has changed by an hour, plus
> the few minutes it took to change the date etc (and is still an hour out, see
> footnote):
> {noformat}
> r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
> {noformat}
> Observe that the range of dates now covers the timestamps on the DM data, and
> also that some data has now been transferred by the File System connector:
> {noformat}
> [james@manifold]$ ls
> /bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
> drl?versionLabel=CURRENT=09018000e515
> drl?versionLabel=CURRENT=09018000e516
> {noformat}
>
>
> [Footnote] It appears that something is trying to take account of Daylight
> Saving Time too.
> If I set the server date to a time outside of DST, the query is aligned with
> the current time:
> {noformat}
> [i2e@i2ehost manifold]$ date
> Mon Oct 29 00:01:13 CET 2018
[jira] [Updated] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
[
https://issues.apache.org/jira/browse/CONNECTORS-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Thomas updated CONNECTORS-1521:
-
Description:
I find that the time/date constraints in queries to the Documentum server are
based on the "raw" local time of the ManifoldCF server but appear to take no
account of the time zones of the two servers.
This can lead to recently modified files not being transferred to the output
repository when you would naturally expect them to be. I'd like the times to be
aligned, perhaps by including time zone in the query. In particular, is there a
way to use UTC perhaps?
Here's an example ...
* create a folder in Documentum
* set up a job to point at the folder and output to the file system
* put two documents into a folder in Documentum
* Select them, right click and export as CSV (to show the timestamps):
{noformat}
1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
Check the local time on the ManifoldCF server machine. Observe that it's
reporting consistent time with the DM server:
{noformat}
[james@manifold]$ date
Tue Aug 7 09:07:25 BST 2018{noformat}
Start the job and look for the query to Documentum in the manifoldcf.log file
(line break added for readability):
{noformat}
DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute query=
(select for READ distinct i_chronicle_id from dm_document where r_modify_date
>= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
^C{noformat}
Notice that the latest date asked for is *before* the modification date of the
files added to DM. (And is an hour out, see footnote.)
See whether anything has been output by the File System connector. It hasn't:
{noformat}
[james@manifold]$ ls /bigdisc/source/PDFs/timezones/
[james@manifold]$
{noformat}
Now:
* change the timezone on the ManifoldCF server machine
* restart the ManifoldCF server and the Documentum processes
* reseed the job
Check the local time on the ManifoldCF server machine; it has changed:
{noformat}
[james@manifold]$ date
Tue Aug 7 10:10:29 CEST 2018{noformat}
Start the job again and notice that the query has changed by an hour, plus the
few minutes it took to change the date etc (and is still an hour out, see
footnote):
{noformat}
r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
{noformat}
Observe that the range of dates now covers the timestamps on the DM data, and
also that some data has now been transferred by the File System connector:
{noformat}
[james@manifold]$ ls
/bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
drl?versionLabel=CURRENT=09018000e515
drl?versionLabel=CURRENT=09018000e516
{noformat}
[Footnote] It appears that something is trying to take account of Daylight
Saving Time too.
If I set the server date to a time outside of DST, the query is aligned with
the current time:
{noformat}
[i2e@i2ehost manifold]$ date
Mon Oct 29 00:01:13 CET 2018
r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
{noformat}
But if I set the time inside DST, the time is an hour before:
{noformat}
[i2e@i2ehost manifold]$ date
Sat Oct 27 00:00:06 CEST 2018
r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
{noformat}
This is perhaps a Java issue rather than a logic issue in the connector? See
e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
was:
I find that the time/date constraints in queries to the Documentum server are
based on the "raw" local time of the ManifoldCF server but appear to take no
account of the time zones of the two servers.
This can lead to recently modified files not being transferred to the output
repository when you would naturally expect them to be. I'd like the times to be
aligned, perhaps by including time zone in the query. In particular, is there a
way to use UTC perhaps?
Here's an example ...
* create a folder in Documentum
* set up a job to point at the folder and output to the file system
* put two documents into a folder in Documentum
* Select them, right click and export as CSV (to show the timestamps):
{noformat}
1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
Check the local time on the ManifoldCF server machine. Observe that it's
reporting consistent time with the DM server:
{noformat}
[james@manifold]$ date
Tue Aug 7 09:07:25 BST 2018{noformat}
Start the job and look for the query to Documentum in the manifoldcf.log file
(line break added for readability):
{noformat}
DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute
[jira] [Created] (CONNECTORS-1521) Documentum Connector users ManifoldCF's local time in queries constraints against the Documentum server without reference to time zones
James Thomas created CONNECTORS-1521:
Summary: Documentum Connector users ManifoldCF's local time in
queries constraints against the Documentum server without reference to time
zones
Key: CONNECTORS-1521
URL: https://issues.apache.org/jira/browse/CONNECTORS-1521
Project: ManifoldCF
Issue Type: Bug
Components: Documentum connector
Affects Versions: ManifoldCF 2.10
Reporter: James Thomas
I find that the time/date constraints in queries to the Documentum server are
based on the "raw" local time of the ManifoldCF server but appear to take no
account of the time zones of the two servers.
This can lead to recently modified files not being transferred to the output
repository when you would naturally expect them to be. I'd like the times to be
aligned, perhaps by including time zone in the query. In particular, is there a
way to use UTC perhaps?
Here's an example ...
* create a folder in Documentum
* set up a job to point at the folder and output to the file system
* put two documents into a folder in Documentum
* Select them, right click and export as CSV (to show the timestamps):
{noformat}
1.png,48489.0,Portable Network Graphics,8/7/2018 9:04 AM,
2.png,28620.0,Portable Network Graphics,8/7/2018 9:04 AM,,{noformat}
Check the local time on the ManifoldCF server machine. Observe that it's
reporting consistent time with the DM server:
{noformat}
[james@manifold]$ date
Tue Aug 7 09:07:25 BST 2018{noformat}
Start the job and look for the query to Documentum in the manifoldcf.log file
(line break added for readability):
{noformat}
DEBUG 2018-08-07T08:07:47.297Z (Startup thread) - DCTM: About to execute query=
(select for READ distinct i_chronicle_id from dm_document where r_modify_date
>= date('01/01/1970 00:00:00','mm/dd/ hh:mi:ss') and
r_modify_date<=date('08/07/2018 08:07:34','mm/dd/ hh:mi:ss')
AND (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
r_content_size>0)) AND ( Folder('/Administrator/james', DESCEND) ))
^C{noformat}
Notice that the latest date asked for is *before* the modification date of the
files added to DM. (And is an hour out, see footnote.)
See whether anything has been output by the File System connector. It hasn't:
{noformat}
[james@manifold]$ ls /bigdisc/source/PDFs/timezones/
[james@manifold]$
{noformat}
Now:
* change the timezone on the ManifoldCF server machine
* restart the ManifoldCF server and the Documentum processes
* reseed the job
Check the local time on the ManifoldCF server machine; it has changed:
{noformat}
[james@manifold]$ date
Tue Aug 7 10:10:29 CEST 2018{noformat}
Start the job again and notice that the query has changed by an hour, plus the
few minutes it took to change the date etc (and is still an hour out, see
footnote):
{noformat}
r_modify_date<=date('08/07/2018 09:11:02','mm/dd/ hh:mi:ss')
{noformat}
Observe that the range of dates now covers the timestamps on the DM data, and
also that some data has now been transferred by the File System connector:
{noformat}
[james@manifold]$ ls
/bigdisc/source/PDFs/timezones/http/mfserver\:8080/da/component/
drl?versionLabel=CURRENT=09018000e515
drl?versionLabel=CURRENT=09018000e516
{noformat}
[Footnote] It appears that something is trying to take account of Daylight
Saving Time too.
If I set the server date to a time outside of DST, the query is aligned with
the current time:
{noformat}
[i2e@i2ehost manifold]$ date
Mon Oct 29 00:01:13 CET 2018
r_modify_date<=date('10/29/2018 00:01:39','mm/dd/ hh:mi:ss')
{noformat}
But if I set the time inside DST, the time is an hour before:
{noformat}
[i2e@i2ehost manifold]$ date
Sat Oct 27 00:00:06 CEST 2018
r_modify_date<=date('10/26/2018 23:00:26','mm/dd/ hh:mi:ss')
{noformat}
This is perhaps a Java issue rather than a logic issue in the connector? See
e.g. [https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1517) Documentum Connector uses different "unconstrained" a_content_type filters depending on whether the Content Types tab has been edited
[
https://issues.apache.org/jira/browse/CONNECTORS-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569399#comment-16569399
]
James Thomas commented on CONNECTORS-1517:
--
I don't know DQL, I'm afraid.
Another alternative would be a "no whitelist" option, i.e. the equivalent of
the unconstrained search that is currently happening in the first step I
described in this ticket.
> Documentum Connector uses different "unconstrained" a_content_type filters
> depending on whether the Content Types tab has been edited
> -
>
> Key: CONNECTORS-1517
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1517
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.10
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
>
> I am using Manifold 2.10 patched for issue
> https://issues.apache.org/jira/browse/CONNECTORS-1512
> I find that the "unconstrained" query submitted to Documentum differs
> depending on whether the Content Types in the job have been edited or not.
> This can dramatically affect which files are fetched. After editing, there
> are likely to be fewer.
> For example, having simply created a job connecting to DM and setting only
> the Paths value to Administrator/james the following request is generated.
> (Taken from manifoldcf.log).
> Note that there are no a_content_type constraints (and my line break for
> readibility):
> {code:java}
> DEBUG 2018-07-26T05:52:56,422 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:52:56','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> Once the Content Types tab has been edited (e.g. to remove the 123w type) it
> looks like this, i.e. the search constrains to only the selected types (my
> ellipsis for readibility):
> {code:java}
> DEBUG 2018-07-26T05:58:36,755 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:58:36','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('acad', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> If the 123w type is now reselected in the Content Types tab, the search adds
> it to the list of a_content_type entries, but doesn't return to the
> unconstrained initial search:
> {code:java}
> DEBUG 2018-07-26T05:59:16,863 (Startup thread) - DCTM: About to execute
> query= (select for READ distinct i_chronicle_id from dm_document where
> r_modify_date >= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
> r_modify_date<=date('07/26/2018 05:59:16','mm/dd/ hh:mi:ss') AND
> (i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
> r_content_size>0
> AND a_content_type IN ('123w', ... 'zip_pub_html')))
> AND ( Folder('/Administrator/james', DESCEND) ))
> {code}
> This means that running what appears to be an equivalent job several times
> may not fetch the same set of documents from Documentum.
> I expect that the same configuration in the UI produces the same search to
> Documentum, regardless of how the configuration was arrived at.
> If the selected items in the Content Types list is treated as the only set of
> files to fetch (i,.e. the initial unconstrained search is considered
> incorrect here) then I guess I might also like to have flexibility to fetch
> file types not on the checklist in the Content Types tab.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CONNECTORS-1517) Documentum Connector uses different "unconstrained" a_content_type filters depending on whether the Content Types tab has been edited
James Thomas created CONNECTORS-1517:
Summary: Documentum Connector uses different "unconstrained"
a_content_type filters depending on whether the Content Types tab has been
edited
Key: CONNECTORS-1517
URL: https://issues.apache.org/jira/browse/CONNECTORS-1517
Project: ManifoldCF
Issue Type: Bug
Components: Documentum connector
Affects Versions: ManifoldCF 2.10
Reporter: James Thomas
I am using Manifold 2.10 patched for issue
https://issues.apache.org/jira/browse/CONNECTORS-1512
I find that the "unconstrained" query submitted to Documentum differs depending
on whether the Content Types in the job have been edited or not. This can
dramatically affect which files are fetched. After editing, there are likely to
be fewer.
For example, having simply created a job connecting to DM and setting only the
Paths value to Administrator/james the following request is generated. (Taken
from manifoldcf.log).
Note that there are no a_content_type constraints (and my line break for
readibility):
{code:java}
DEBUG 2018-07-26T05:52:56,422 (Startup thread) - DCTM: About to execute query=
(select for READ distinct i_chronicle_id from dm_document where r_modify_date
>= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
r_modify_date<=date('07/26/2018 05:52:56','mm/dd/ hh:mi:ss') AND
(i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
r_content_size>0))
AND ( Folder('/Administrator/james', DESCEND) ))
{code}
Once the Content Types tab has been edited (e.g. to remove the 123w type) it
looks like this, i.e. the search constrains to only the selected types (my
ellipsis for readibility):
{code:java}
DEBUG 2018-07-26T05:58:36,755 (Startup thread) - DCTM: About to execute query=
(select for READ distinct i_chronicle_id from dm_document where r_modify_date
>= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
r_modify_date<=date('07/26/2018 05:58:36','mm/dd/ hh:mi:ss') AND
(i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
r_content_size>0
AND a_content_type IN ('acad', ... 'zip_pub_html')))
AND ( Folder('/Administrator/james', DESCEND) ))
{code}
If the 123w type is now reselected in the Content Types tab, the search adds it
to the list of a_content_type entries, but doesn't return to the unconstrained
initial search:
{code:java}
DEBUG 2018-07-26T05:59:16,863 (Startup thread) - DCTM: About to execute query=
(select for READ distinct i_chronicle_id from dm_document where r_modify_date
>= date('01/01/1970 01:00:00','mm/dd/ hh:mi:ss') and
r_modify_date<=date('07/26/2018 05:59:16','mm/dd/ hh:mi:ss') AND
(i_is_deleted=TRUE Or (i_is_deleted=FALSE AND a_full_text=TRUE AND
r_content_size>0
AND a_content_type IN ('123w', ... 'zip_pub_html')))
AND ( Folder('/Administrator/james', DESCEND) ))
{code}
This means that running what appears to be an equivalent job several times may
not fetch the same set of documents from Documentum.
I expect that the same configuration in the UI produces the same search to
Documentum, regardless of how the configuration was arrived at.
If the selected items in the Content Types list is treated as the only set of
files to fetch (i,.e. the initial unconstrained search is considered incorrect
here) then I guess I might also like to have flexibility to fetch file types
not on the checklist in the Content Types tab.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1512) Documentum Connector throws NPE when a job is run after a document is deleted from the repository
[
https://issues.apache.org/jira/browse/CONNECTORS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530838#comment-16530838
]
James Thomas commented on CONNECTORS-1512:
--
Thanks, yes, I highlighted the fact that it was the same lines because it was
unexpected.
I believe that I have now got the patch deployed and the DM and MF servers
running without NPE through my repro.
In case it's useful data, I think the build and deploy of the patch were
actually OK. The things that got in my way were configuration:
* setting the MCF_HOME environment variable
* setting the DOCUMENTUM environment variable
* editing my CLASSPATH workaround (see earlier in this thread) for the
Documentum scripts
My inexperience with your software is a barrier for me here. Thanks again for
your help.
> Documentum Connector throws NPE when a job is run after a document is deleted
> from the repository
> -
>
> Key: CONNECTORS-1512
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1512
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.8.1, ManifoldCF 2.10
> Environment: Documentum 16.4
> Manifold 2.10, using dfc jars 16.4
> Manifold server machine details (although also seen on a Windows-based MF
> 2.8.1):
> {code:java}
> $ hostnamectl
> ...
> Chassis: vm
> Virtualization: kvm
> Operating System: CentOS Linux 7 (Core)
> CPE OS Name: cpe:/o:centos:centos:7
> Kernel: Linux 3.10.0-862.2.3.el7.x86_64
> Architecture: x86-64
> $ java -version
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
> {code}
> I have the following entry in properties.xml for verbose logging from
> connectors:
> {code:java}
>
> {code}
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1512.patch, repro_delete.log
>
>
> I see an NPE from the Documentum Connector when I re-run a job after deleting
> a file from the Documentum repository using the Documentum Administrator:
> {code:java}
> FATAL 2018-06-29T08:36:54,847 (Worker thread '5') - Error tossed: null
> java.lang.NullPointerException
> at
> org.apache.manifoldcf.crawler.common.DCTM.DocumentumObjectImpl.getContentSize(DocumentumObjectImpl.java:181)
> ~[?:?]
> ...
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$ProcessDocumentThread.run(DCTM.java:1399)
> ~[?:?]
> {code}
> The job then appears not to terminate in the Manifold UI. If the job is not
> aborted, the NPE will show again after restarting the Manifold server and
> Documentum Connectors.
> I expect the connectors to handle deletion of source in the repository,
> passing the appropriate status to the output connector, and not failing like
> this.
> Reproduce:
> * Configure Manifold to use Documentum Connectors to talk to a Documentum
> instance
> * Sanity check your configuration by
> ** create a directory in Documentum, D, with two files F, G
> ** create a job in Manifold to take from D and write to the file system in
> directory L
> ** run the job and observe that F, G appear in L
> * delete G from D in Documentum
> ** in Documentum Administrator, right click on G and choose Delete, then OK
> * re-run your job
> * observe that the job doesn't end
> * inspect L
> * observe that F and G are both still present
> * inspect the Manifold logs
> * observe NPE in the logs
>
> In the log file I'll attach, F and G are Book1.xlsx and Book1.xls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (CONNECTORS-1512) Documentum Connector throws NPE when a job is run after a document is deleted from the repository
[
https://issues.apache.org/jira/browse/CONNECTORS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Thomas reopened CONNECTORS-1512:
--
I have just reported that I still see the issue after applying the proposed
patch.
> Documentum Connector throws NPE when a job is run after a document is deleted
> from the repository
> -
>
> Key: CONNECTORS-1512
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1512
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.8.1, ManifoldCF 2.10
> Environment: Documentum 16.4
> Manifold 2.10, using dfc jars 16.4
> Manifold server machine details (although also seen on a Windows-based MF
> 2.8.1):
> {code:java}
> $ hostnamectl
> ...
> Chassis: vm
> Virtualization: kvm
> Operating System: CentOS Linux 7 (Core)
> CPE OS Name: cpe:/o:centos:centos:7
> Kernel: Linux 3.10.0-862.2.3.el7.x86_64
> Architecture: x86-64
> $ java -version
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
> {code}
> I have the following entry in properties.xml for verbose logging from
> connectors:
> {code:java}
>
> {code}
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1512.patch, repro_delete.log
>
>
> I see an NPE from the Documentum Connector when I re-run a job after deleting
> a file from the Documentum repository using the Documentum Administrator:
> {code:java}
> FATAL 2018-06-29T08:36:54,847 (Worker thread '5') - Error tossed: null
> java.lang.NullPointerException
> at
> org.apache.manifoldcf.crawler.common.DCTM.DocumentumObjectImpl.getContentSize(DocumentumObjectImpl.java:181)
> ~[?:?]
> ...
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$ProcessDocumentThread.run(DCTM.java:1399)
> ~[?:?]
> {code}
> The job then appears not to terminate in the Manifold UI. If the job is not
> aborted, the NPE will show again after restarting the Manifold server and
> Documentum Connectors.
> I expect the connectors to handle deletion of source in the repository,
> passing the appropriate status to the output connector, and not failing like
> this.
> Reproduce:
> * Configure Manifold to use Documentum Connectors to talk to a Documentum
> instance
> * Sanity check your configuration by
> ** create a directory in Documentum, D, with two files F, G
> ** create a job in Manifold to take from D and write to the file system in
> directory L
> ** run the job and observe that F, G appear in L
> * delete G from D in Documentum
> ** in Documentum Administrator, right click on G and choose Delete, then OK
> * re-run your job
> * observe that the job doesn't end
> * inspect L
> * observe that F and G are both still present
> * inspect the Manifold logs
> * observe NPE in the logs
>
> In the log file I'll attach, F and G are Book1.xlsx and Book1.xls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1512) Documentum Connector throws NPE when a job is run after a document is deleted from the repository
[
https://issues.apache.org/jira/browse/CONNECTORS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530388#comment-16530388
]
James Thomas commented on CONNECTORS-1512:
--
You were right about line endings. I ran dos2unix on the two files that you
edited and was then able to patch and build.
These are the changed files (where dist.vanilla is the 2.10 build without the
patch):
{code:java}
$ diff -r dist dist.vanilla/
Binary files dist/connector-lib/mcf-documentum-connector.jar and
dist.vanilla/connector-lib/mcf-documentum-connector.jar differ
Binary files dist/connector-lib/mcf-documentum-connector-rmistub.jar and
dist.vanilla/connector-lib/mcf-documentum-connector-rmistub.jar differ
Binary files dist/connector-lib/mcf-filenet-connector-rmistub.jar and
dist.vanilla/connector-lib/mcf-filenet-connector-rmistub.jar differ
Binary files dist/connector-lib/mcf-meridio-connector.jar and
dist.vanilla/connector-lib/mcf-meridio-connector.jar differ
Binary files dist/connector-lib/mcf-sharepoint-connector.jar and
dist.vanilla/connector-lib/mcf-sharepoint-connector.jar differ
Binary files
dist/processes/documentum-server/lib/mcf-documentum-connector-implementation.jar
and
dist.vanilla/processes/documentum-server/lib/mcf-documentum-connector-implementation.jar
differ
Binary files
dist/processes/documentum-server/lib/mcf-documentum-connector-rmiskel.jar and
dist.vanilla/processes/documentum-server/lib/mcf-documentum-connector-rmiskel.jar
differ
Binary files
dist/processes/filenet-server/lib/mcf-filenet-connector-rmiskel.jar and
dist.vanilla/processes/filenet-server/lib/mcf-filenet-connector-rmiskel.jar
differ
Binary files dist/web/war/mcf-crawler-ui.war and
dist.vanilla/web/war/mcf-crawler-ui.war differ
Binary files dist/web-proprietary/war/mcf-crawler-ui.war and
dist.vanilla/web-proprietary/war/mcf-crawler-ui.war differ
{code}
I zipped up these files and deployed them into a copy of my 2.10 instance,
removed the dbname files (to clear any bad history) and restarted Manfiold and
the Documentum connector servers.
I re-ran my repro steps and am still seeing an NPE at the same lines in
DocumentumObjectImpl.java, DCTM.java at the same point:
{code:java}
FATAL 2018-07-02T20:46:25,935 (Worker thread '13') - Error tossed: null
java.lang.NullPointerException
at
org.apache.manifoldcf.crawler.common.DCTM.DocumentumObjectImpl.getContentSize(DocumentumObjectImpl.java:181)
~[?:?]
...
at
org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$ProcessDocumentThread.run(DCTM.java:1399)
~[?:?]
{code}
What information can I provide you with to help diagnose further or check my
build?
For example, as a sanity check I can see that DocumentumtImpl.java is changed
in the patched deployment:
{code:java}
$ unzip -l
apache-manifoldcf-2.10/processes/documentum-server/lib/mcf-documentum-connector-implementation.jar
Archive:
apache-manifoldcf-2.10/processes/documentum-server/lib/mcf-documentum-connector-implementation.jar
Length DateTimeName
- -- -
0 04-15-2018 11:43 META-INF/
103 04-15-2018 11:43 META-INF/MANIFEST.MF
0 04-15-2018 11:43 org/
0 04-15-2018 11:43 org/apache/
0 04-15-2018 11:43 org/apache/manifoldcf/
0 04-15-2018 11:43 org/apache/manifoldcf/crawler/
0 04-15-2018 11:43 org/apache/manifoldcf/crawler/common/
0 04-15-2018 11:43 org/apache/manifoldcf/crawler/common/DCTM/
1126 04-15-2018 11:43
org/apache/manifoldcf/crawler/common/DCTM/DocumentumFactoryImpl.class
10676 04-15-2018 11:43
org/apache/manifoldcf/crawler/common/DCTM/DocumentumImpl.class
9555 04-15-2018 11:43
org/apache/manifoldcf/crawler/common/DCTM/DocumentumObjectImpl.class
2487 04-15-2018 11:43
org/apache/manifoldcf/crawler/common/DCTM/DocumentumResultImpl.class
- ---
23947 12 files
$ unzip -l
apache-manifoldcf-2.10_1512/processes/documentum-server/lib/mcf-documentum-connector-implementation.jar
Archive:
apache-manifoldcf-2.10_1512/processes/documentum-server/lib/mcf-documentum-connector-implementation.jar
Length DateTimeName
- -- -
0 07-02-2018 10:25 META-INF/
105 07-02-2018 10:25 META-INF/MANIFEST.MF
0 07-01-2018 11:15 org/
0 07-01-2018 11:15 org/apache/
0 07-01-2018 11:15 org/apache/manifoldcf/
0 07-01-2018 11:15 org/apache/manifoldcf/crawler/
0 07-01-2018 11:15 org/apache/manifoldcf/crawler/common/
0 07-01-2018 11:15 org/apache/manifoldcf/crawler/common/DCTM/
1126 07-01-2018 11:15
org/apache/manifoldcf/crawler/common/DCTM/DocumentumFactoryImpl.class
10858 07-02-2018 10:25
org/apache/manifoldcf/crawler/common/DCTM/DocumentumImpl.class
9555 07-01-2018 11:15
[jira] [Commented] (CONNECTORS-1512) Documentum Connector throws NPE when a job is run after a document is deleted from the repository
[
https://issues.apache.org/jira/browse/CONNECTORS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529047#comment-16529047
]
James Thomas commented on CONNECTORS-1512:
--
I have followed your steps (without patching) and apparently successfully built
- I got a bunch of jars in the dist directory.
These are the tarballs I used:
{code:java}
$ wget
http://apache.mirror.anlx.net/manifoldcf/apache-manifoldcf-2.10/apache-manifoldcf-2.10-src.tar.gz
$ wget
http://mirrors.ukfast.co.uk/sites/ftp.apache.org/manifoldcf/apache-manifoldcf-2.10/apache-manifoldcf-2.10-lib.tar.gz
{code}
I then tried to patch, but it's not applying:
{code:java}
$ wget
https://issues.apache.org/jira/secure/attachment/12929706/CONNECTORS-1512.patch
--2018-07-01 11:36:34--
https://issues.apache.org/jira/secure/attachment/12929706/CONNECTORS-1512.patch
Resolving issues.apache.org... 207.244.88.139
Connecting to issues.apache.org|207.244.88.139|:443... connected.
HTTP request sent, awaiting response... 200 200
Length: 2562 (2.5K) [text/plain]
Saving to: “CONNECTORS-1512.patch”
100%[>]
2,562 --.-K/s in 0s
2018-07-01 11:36:34 (36.0 MB/s) - “CONNECTORS-1512.patch” saved [2562/2562]
$ patch -p0 < CONNECTORS-1512.patch
(Stripping trailing CRs from patch.)
patching file
connectors/documentum/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/DCTM/DCTM.java
Hunk #1 FAILED at 1396.
Hunk #2 FAILED at 1463.
2 out of 2 hunks FAILED -- saving rejects to file
connectors/documentum/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/DCTM/DCTM.java.rej
(Stripping trailing CRs from patch.)
patching file
connectors/documentum/implementation/src/main/java/org/apache/manifoldcf/crawler/common/DCTM/DocumentumImpl.java
Hunk #1 FAILED at 331.
1 out of 1 hunk FAILED -- saving rejects to file
connectors/documentum/implementation/src/main/java/org/apache/manifoldcf/crawler/common/DCTM/DocumentumImpl.java.rej
$ md5sum
connectors/documentum/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/DCTM/DCTM.java
a8669dbba38ee038353b1ed0b0bb4302
connectors/documentum/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/DCTM/DCTM.java
$ md5sum
connectors/documentum/implementation/src/main/java/org/apache/manifoldcf/crawler/common/DCTM/DocumentumImpl.java
303bee1eaf32ddb871c8fa4c4db636a7
connectors/documentum/implementation/src/main/java/org/apache/manifoldcf/crawler/common/DCTM/DocumentumImpl.java
{code}
I wonder whether you're working from a revision that isn't quite the 2.10
distro? Or am I doing something obviously wrong?
I tried -a few things with patch: -merge, -fuzz etc but I'm reluctant to monkey
around with your patch too much.
P.S. I found some inconsistency in README between the src and lib distros.
Where should I report that?
> Documentum Connector throws NPE when a job is run after a document is deleted
> from the repository
> -
>
> Key: CONNECTORS-1512
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1512
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.8.1, ManifoldCF 2.10
> Environment: Documentum 16.4
> Manifold 2.10, using dfc jars 16.4
> Manifold server machine details (although also seen on a Windows-based MF
> 2.8.1):
> {code:java}
> $ hostnamectl
> ...
> Chassis: vm
> Virtualization: kvm
> Operating System: CentOS Linux 7 (Core)
> CPE OS Name: cpe:/o:centos:centos:7
> Kernel: Linux 3.10.0-862.2.3.el7.x86_64
> Architecture: x86-64
> $ java -version
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
> {code}
> I have the following entry in properties.xml for verbose logging from
> connectors:
> {code:java}
>
> {code}
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1512.patch, repro_delete.log
>
>
> I see an NPE from the Documentum Connector when I re-run a job after deleting
> a file from the Documentum repository using the Documentum Administrator:
> {code:java}
> FATAL 2018-06-29T08:36:54,847 (Worker thread '5') - Error tossed: null
> java.lang.NullPointerException
> at
> org.apache.manifoldcf.crawler.common.DCTM.DocumentumObjectImpl.getContentSize(DocumentumObjectImpl.java:181)
> ~[?:?]
> ...
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$ProcessDocumentThread.run(DCTM.java:1399)
> ~[?:?]
> {code}
> The job then appears not to terminate in the Manifold UI.
[jira] [Commented] (CONNECTORS-1512) Documentum Connector throws NPE when a job is run after a document is deleted from the repository
[
https://issues.apache.org/jira/browse/CONNECTORS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527468#comment-16527468
]
James Thomas commented on CONNECTORS-1512:
--
I downloaded 2.10 from
http://www.apache.org/dyn/closer.lua/manifoldcf/apache-manifoldcf-2.10/apache-manifoldcf-2.10-bin.tar.gz
and uncompressed it.
I'm running from the command line in the example directory using
{code:java}
java -jar start.jar
{code}
I took the DFC distro for DM 16.4 from Open Text's forums and unpacked the
relevant files into processes/documentum-server/lib-proprietary
I found that I needed to set CLASSPATH to get the DM connectors to work so I've
edited the supplied run.sh scripts to do this by adding this line immediately
before calls to Java, effectively adding the location of dfc.properties to the
classpath:
{code:java}
CLASSPATH=$CLASSPATH:/opt/manifold/apache-manifoldcf-2.10/processes/documentum-server
{code}
So I've done no building from source. If it was possible to pick up a nightly
build of the DM connector jars only (I guess?) I'd like to do that.
If you need other specific information please say.
> Documentum Connector throws NPE when a job is run after a document is deleted
> from the repository
> -
>
> Key: CONNECTORS-1512
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1512
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.8.1, ManifoldCF 2.10
> Environment: Documentum 16.4
> Manifold 2.10, using dfc jars 16.4
> Manifold server machine details (although also seen on a Windows-based MF
> 2.8.1):
> {code:java}
> $ hostnamectl
> ...
> Chassis: vm
> Virtualization: kvm
> Operating System: CentOS Linux 7 (Core)
> CPE OS Name: cpe:/o:centos:centos:7
> Kernel: Linux 3.10.0-862.2.3.el7.x86_64
> Architecture: x86-64
> $ java -version
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
> {code}
> I have the following entry in properties.xml for verbose logging from
> connectors:
> {code:java}
>
> {code}
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1512.patch, repro_delete.log
>
>
> I see an NPE from the Documentum Connector when I re-run a job after deleting
> a file from the Documentum repository using the Documentum Administrator:
> {code:java}
> FATAL 2018-06-29T08:36:54,847 (Worker thread '5') - Error tossed: null
> java.lang.NullPointerException
> at
> org.apache.manifoldcf.crawler.common.DCTM.DocumentumObjectImpl.getContentSize(DocumentumObjectImpl.java:181)
> ~[?:?]
> ...
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$ProcessDocumentThread.run(DCTM.java:1399)
> ~[?:?]
> {code}
> The job then appears not to terminate in the Manifold UI. If the job is not
> aborted, the NPE will show again after restarting the Manifold server and
> Documentum Connectors.
> I expect the connectors to handle deletion of source in the repository,
> passing the appropriate status to the output connector, and not failing like
> this.
> Reproduce:
> * Configure Manifold to use Documentum Connectors to talk to a Documentum
> instance
> * Sanity check your configuration by
> ** create a directory in Documentum, D, with two files F, G
> ** create a job in Manifold to take from D and write to the file system in
> directory L
> ** run the job and observe that F, G appear in L
> * delete G from D in Documentum
> ** in Documentum Administrator, right click on G and choose Delete, then OK
> * re-run your job
> * observe that the job doesn't end
> * inspect L
> * observe that F and G are both still present
> * inspect the Manifold logs
> * observe NPE in the logs
>
> In the log file I'll attach, F and G are Book1.xlsx and Book1.xls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CONNECTORS-1512) Documentum Connector throws NPE when a job is run after a document is deleted from the repository
[
https://issues.apache.org/jira/browse/CONNECTORS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527458#comment-16527458
]
James Thomas commented on CONNECTORS-1512:
--
Thanks for the quick response (y)
I'm pretty new to Manifold. Where can I get a build containing your fix, and
what do I need to do to install it?
> Documentum Connector throws NPE when a job is run after a document is deleted
> from the repository
> -
>
> Key: CONNECTORS-1512
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1512
> Project: ManifoldCF
> Issue Type: Bug
> Components: Documentum connector
>Affects Versions: ManifoldCF 2.8.1, ManifoldCF 2.10
> Environment: Documentum 16.4
> Manifold 2.10, using dfc jars 16.4
> Manifold server machine details (although also seen on a Windows-based MF
> 2.8.1):
> {code:java}
> $ hostnamectl
> ...
> Chassis: vm
> Virtualization: kvm
> Operating System: CentOS Linux 7 (Core)
> CPE OS Name: cpe:/o:centos:centos:7
> Kernel: Linux 3.10.0-862.2.3.el7.x86_64
> Architecture: x86-64
> $ java -version
> openjdk version "1.8.0_171"
> OpenJDK Runtime Environment (build 1.8.0_171-b10)
> OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
> {code}
> I have the following entry in properties.xml for verbose logging from
> connectors:
> {code:java}
>
> {code}
>Reporter: James Thomas
>Assignee: Karl Wright
>Priority: Major
> Fix For: ManifoldCF 2.11
>
> Attachments: CONNECTORS-1512.patch, repro_delete.log
>
>
> I see an NPE from the Documentum Connector when I re-run a job after deleting
> a file from the Documentum repository using the Documentum Administrator:
> {code:java}
> FATAL 2018-06-29T08:36:54,847 (Worker thread '5') - Error tossed: null
> java.lang.NullPointerException
> at
> org.apache.manifoldcf.crawler.common.DCTM.DocumentumObjectImpl.getContentSize(DocumentumObjectImpl.java:181)
> ~[?:?]
> ...
> at
> org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$ProcessDocumentThread.run(DCTM.java:1399)
> ~[?:?]
> {code}
> The job then appears not to terminate in the Manifold UI. If the job is not
> aborted, the NPE will show again after restarting the Manifold server and
> Documentum Connectors.
> I expect the connectors to handle deletion of source in the repository,
> passing the appropriate status to the output connector, and not failing like
> this.
> Reproduce:
> * Configure Manifold to use Documentum Connectors to talk to a Documentum
> instance
> * Sanity check your configuration by
> ** create a directory in Documentum, D, with two files F, G
> ** create a job in Manifold to take from D and write to the file system in
> directory L
> ** run the job and observe that F, G appear in L
> * delete G from D in Documentum
> ** in Documentum Administrator, right click on G and choose Delete, then OK
> * re-run your job
> * observe that the job doesn't end
> * inspect L
> * observe that F and G are both still present
> * inspect the Manifold logs
> * observe NPE in the logs
>
> In the log file I'll attach, F and G are Book1.xlsx and Book1.xls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CONNECTORS-1512) Documentum Connector throws NPE when a job is run after a document is deleted from the repository
James Thomas created CONNECTORS-1512:
Summary: Documentum Connector throws NPE when a job is run after a
document is deleted from the repository
Key: CONNECTORS-1512
URL: https://issues.apache.org/jira/browse/CONNECTORS-1512
Project: ManifoldCF
Issue Type: Bug
Components: Documentum connector
Affects Versions: ManifoldCF 2.10, ManifoldCF 2.8.1
Environment: Documentum 16.4
Manifold 2.10, using dfc jars 16.4
Manifold server machine details (although also seen on a Windows-based MF
2.8.1):
{code:java}
$ hostnamectl
...
Chassis: vm
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-862.2.3.el7.x86_64
Architecture: x86-64
$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-b10)
OpenJDK 64-Bit Server VM (build 25.171-b10, mixed mode)
{code}
I have the following entry in properties.xml for verbose logging from
connectors:
{code:java}
{code}
Reporter: James Thomas
Attachments: repro_delete.log
I see an NPE from the Documentum Connector when I re-run a job after deleting a
file from the Documentum repository using the Documentum Administrator:
{code:java}
FATAL 2018-06-29T08:36:54,847 (Worker thread '5') - Error tossed: null
java.lang.NullPointerException
at
org.apache.manifoldcf.crawler.common.DCTM.DocumentumObjectImpl.getContentSize(DocumentumObjectImpl.java:181)
~[?:?]
...
at
org.apache.manifoldcf.crawler.connectors.DCTM.DCTM$ProcessDocumentThread.run(DCTM.java:1399)
~[?:?]
{code}
The job then appears not to terminate in the Manifold UI. If the job is not
aborted, the NPE will show again after restarting the Manifold server and
Documentum Connectors.
I expect the connectors to handle deletion of source in the repository, passing
the appropriate status to the output connector, and not failing like this.
Reproduce:
* Configure Manifold to use Documentum Connectors to talk to a Documentum
instance
* Sanity check your configuration by
** create a directory in Documentum, D, with two files F, G
** create a job in Manifold to take from D and write to the file system in
directory L
** run the job and observe that F, G appear in L
* delete G from D in Documentum
** in Documentum Administrator, right click on G and choose Delete, then OK
* re-run your job
* observe that the job doesn't end
* inspect L
* observe that F and G are both still present
* inspect the Manifold logs
* observe NPE in the logs
In the log file I'll attach, F and G are Book1.xlsx and Book1.xls
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
