[
https://issues.apache.org/jira/browse/CONNECTORS-16?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wright reassigned CONNECTORS-16:
-
Assignee: Karl Wright
JCIFS connector's document fingerprinting feature is not general enough
---
Key: CONNECTORS-16
URL: https://issues.apache.org/jira/browse/CONNECTORS-16
Project: Lucene Connector Framework
Issue Type: Improvement
Components: Framework agents process, Framework crawler agent, GTS
connector, JCIFS connector, LiveLink connector, Lucene/SOLR connector,
Meridio connector, RSS connector, SharePoint connector, Web connector
Reporter: Karl Wright
Assignee: Karl Wright
Priority: Minor
The JCIFS connector has a feature, called fingerprinting, which allows it
to classify documents according to ability of the back-end to index that
content. Right at the moment, this fingerprinter is capable of recognizing
PDFs, Microsoft Office files, and text files as being indexable. One could
imagine, though, that different SOLR plugins, etc. might have more capability
than that. Also, other connectors could potentially benefit from similar
technology, specifically any connector that deals with binary documents.
One approach to solving this problem would be to remove the feature entirely,
and allow whatever pipeline exists in SOLR determine the indexability after
the fact. The reason this feature was added at MetaCarta, however, is that
it may be possible to exclude an un-useful document without having to fetch
the whole thing, and (at least for MetaCarta clients) the number of
unindexable files of gigantic size was a big concern.
Another approach might be to tie the functionality in with the output
connector interface, so that an output connector would (somehow) determine
applicability of a document. This would require some care to make it
possible to fingerprint without having to download the entire document, but
would otherwise have the correct overall structure.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
