I already tried this example, I am currently working on this. I have complied
the code, it is indexing the documents. But, it is not adding any thing to the
field - _text_ . Also, not giving any metadata.
doc.addField("_text_", textHandler.toString()); --> here,
textHandler.toString() is blank for all the 40 documents. All I am getting is
the 'id' & 'version' field.
This is the code that I tried :
package mind.solr;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.HttpSolrClient;
import org.apache.solr.client.solrj.impl.XMLResponseParser;
import org.apache.solr.client.solrj.response.UpdateResponse;
import org.apache.solr.common.SolrInputDocument;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.sax.BodyContentHandler;
import org.xml.sax.ContentHandler;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collection;
public class solrJExtract {
private HttpSolrClient client;
private long start = System.currentTimeMillis();
private AutoDetectParser autoParser;
private int totalTika = 0;
private int totalSql = 0;
@SuppressWarnings("rawtypes")
private Collection docList = new ArrayList();
public static void main(String[] args) {
try {
solrJExtract idxer = new solrJExtract("http://localhost:8983/solr/tika";);
idxer.doTikaDocuments(new File("D:\\docs"));
idxer.endIndexing();
} catch (Exception e) {
e.printStackTrace();
}
}
private solrJExtract(String url) throws IOException, SolrServerException {
// Create a SolrCloud-aware client to send docs to Solr
// Use something like HttpSolrClient for stand-alone
client = new HttpSolrClient.Builder("http://localhost:8983/solr/tika";)
.withConnectionTimeout(10000)
.withSocketTimeout(60000)
.build();
// binary parser is used by default for responses
client.setParser(new XMLResponseParser());
// One of the ways Tika can be used to attempt to parse arbitrary files.
autoParser = new AutoDetectParser();
}
// Just a convenient place to wrap things up.
@SuppressWarnings("unchecked")
private void endIndexing() throws IOException, SolrServerException {
if ( docList.size() > 0) { // Are there any documents left over?
client.add(docList, 300000); // Commit within 5 minutes
}
client.commit(); // Only needs to be done at the end,
// commitWithin should do the rest.
// Could even be omitted
// assuming commitWithin was specified.
long endTime = System.currentTimeMillis();
System.out.println("Total Time Taken: " + (endTime - start) +
" milliseconds to index " + totalSql +
" SQL rows and " + totalTika + " documents");
}
/**
* ***************************Tika processing here
*/
// Recursively traverse the filesystem, parsing everything found.
private void doTikaDocuments(File root) throws IOException,
SolrServerException {
// Simple loop for recursively indexing all the files
// in the root directory passed in.
for (File file : root.listFiles()) {
if (file.isDirectory()) {
doTikaDocuments(file);
continue;
}
// Get ready to parse the file.
ContentHandler textHandler = new BodyContentHandler();
Metadata metadata = new Metadata();
ParseContext context = new ParseContext();
// Tim Allison noted the following, thanks Tim!
// If you want Tika to parse embedded files (attachments within your .doc
or any other embedded
// files), you need to send in the autodetectparser in the parsecontext:
// context.set(Parser.class, autoParser);
InputStream input = new FileInputStream(file);
// Try parsing the file. Note we haven't checked at all to
// see whether this file is a good candidate.
try {
autoParser.parse(input, textHandler, metadata, context);
} catch (Exception e) {
// Needs better logging of what went wrong in order to
// track down "bad" documents.
System.out.println(String.format("File %s failed",
file.getCanonicalPath()));
e.printStackTrace();
continue;
}
// Just to show how much meta-data and what form it's in.
dumpMetadata(file.getCanonicalPath(), metadata);
// Index just a couple of the meta-data fields.
SolrInputDocument doc = new SolrInputDocument();
doc.addField("id", file.getCanonicalPath());
// Crude way to get known meta-data fields.
// Also possible to write a simple loop to examine all the
// metadata returned and selectively index it and/or
// just get a list of them.
// One can also use the Lucidworks field mapping to
// accomplish much the same thing.
String author = metadata.get("Author");
/*
* if (author != null) { //doc.addField("author", author); }
*/
doc.addField("_text_", textHandler.toString());
//doc.addField("meta", metadata.get("Last_Modified"));
docList.add(doc);
++totalTika;
// Completely arbitrary, just batch up more than one document
// for throughput!
if ( docList.size() >= 1000) {
// Commit within 5 minutes.
UpdateResponse resp = client.add(docList, 300000);
if (resp.getStatus() != 0) {
System.out.println("Some horrible error has occurred, status is: " +
resp.getStatus());
}
docList.clear();
}
}
}
// Just to show all the metadata that's available.
private void dumpMetadata(String fileName, Metadata metadata) {
System.out.println("Dumping metadata for file: " + fileName);
for (String name : metadata.names()) {
System.out.println(name + ":" + metadata.get(name));
}
System.out.println("........xxxxxxxxxxxxxxxxxxxxxxxxx..........");
}
}
Also, I am attaching the scrollconfig.xml & Managed-schema.xml for my
collection. Please see to it & suggest where I am getting wrong.
I can't even get to see the _text_ field in the query result, instead of stored
parameter being true.
Any help would really be appreciated.
Thanks !
-----Original Message-----
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: 28 August 2019 14:18
To: solr-user@lucene.apache.org
Subject: Re: Require searching only for file content and not metadata
On 8/27/2019 7:18 AM, Khare, Kushal (MIND) wrote:
> Basically, what problem I am facing is - I am getting the textual content +
> other metadata in my _text_ field. But, I want only the textual content
> written inside the document.
> I tried various Request Handler Update Extract configurations, but none of
> them worked for me.
> Please help me resolve this as I am badly stuck in this.
Controlling exactly what gets indexed in which fields is likely going to
require that you write the indexing software yourself -- a program that
extracts the data you want and sends it to Solr for indexing.
We do not recommend running the Extracting Request Handler in production
-- Tika is known to crash when given some documents (usually PDF files are the
problematic ones, but other formats can cause it too), and if it crashes while
running inside Solr, it will take Solr down with it.
Here is an example program that uses Tika for rich document parsing. It also
talks to a database, but that part could be easily removed or modified:
https://lucidworks.com/post/indexing-with-solrj/
Thanks,
Shawn
________________________________
The information contained in this electronic message and any attachments to
this message are intended for the exclusive use of the addressee(s) and may
contain proprietary, confidential or privileged information. If you are not the
intended recipient, you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately and destroy all copies of this message and
any attachments. WARNING: Computer viruses can be transmitted via email. The
recipient should check this email and any attachments for the presence of
viruses. The company accepts no liability for any damage caused by any
virus/trojan/worms/malicious code transmitted by this email. www.motherson.com
solrconfig.xml
Description: solrconfig.xml
