Hi,
maybe it is the way you created the jar ? Why not applying the patch to
lucene/solr trunk and use ant jar instead to get the codecs jar created
for you ?
Also, I think the directory where you put the jars should be called
"lib" instead of "Lib".
you can try also to use the lib directives in your solrconfig.xml [1]
[1]
https://cwiki.apache.org/confluence/display/solr/Lib+Directives+in+SolrConfig
--
Renaud Delbru
On 20/06/16 15:42, Sidana, Mohit wrote:
Hello,
As Part of my studies I am exploring the solutions which can be used for
Lucene/Solr Index encryption.
I found the patch open on Apache JIRA- Codec for index-level encryption
<https://issues.apache.org/jira/browse/LUCENE-6966>(LUCENE-6966).
https://issues.apache.org/jira/browse/LUCENE-6966 and I am currently
trying to test this Custom codec with Solr to perform secure search over
some sensitive records.
I've decided to follow the path described in Solr wiki, setting up
Simple Text Codec and further tried to use Encrypted codec Source.
*Here are the additional details.*
I've created a basic jar file out of this source code (Build it as a jar
from Eclipse using Maven Plugin).
The Solr installation I'm using to test this is the Solr 6.0.0 unzipped,
and started via its embedded Jetty server and using the single core.
I've placed my jar with the codec in [My_Core\ instance Dir.]\ Lib
In:
[$SolrDir]\Solr\ My_Core \conf\*solrconfig.xml*
I've added the following lines:
|<lib dir="./lib/" regex=".*\.jar"/> |||
|<codecFactory class="solr.SchemaCodecFactory"/>|
Then in the *schema.xml* file, I've declared some field and field Types
that should use this codec:
|<field name="City" type="strings"/>|||
| <field name="Country" type="strings"/>|
| <field name="Name" type="strings"/>|
| <field name="Payment" type="strings"/>|
| <field name="Price" type="tlongs"/>|
| <field name="Product" type="strings"/>|
| <field name="State" type="strings"/>|
||
|<!-- set the per field posting format as below -->|
|<fieldType name="strings" class="solr.StrField" sortMissingLast="true"
docValues="true" multiValued="true" postingsFormat="EncryptedLucene50"/>|
|<fieldType name="tlongs" class="solr.TrieLongField"
positionIncrementGap="0" docValues="true" multiValued="true"
precisionStep="8" postingsFormat="EncryptedLucene50"/>|
I'm pretty sure I've followed all the steps described in Solr Wiki;
however, when I actually try to use custom codec implementation (named
"Encrypted Codec") to index some sample CSV data using simple post tool
java -Dtype=text/csv -Durl=http://localhost:8983/solr/My_Core /update
-jar post.jar Sales.csv
and I have also tried doing the same with SolrJ but I have faced the
same error.
SolrClient _server_=
*new*HttpSolrClient("http://localhost:8983/solr/My_Core ");
SolrInputDocument doc= *new*SolrInputDocument();
doc.addField("id", "1234");
doc.addField("name", "A lovely summer holiday");
*try*{
server.add(doc);
server.commit();
System.*/out/*.println("Document added!");
} *catch*(SolrServerException | IOException e) {
e.printStackTrace();
}
}
}
I get the attached errors in Solr log.
org.apache.solr.common.SolrException: Exception writing document id
b3e01ada-d0f1-4ddf-ad6a-2828bfe619a3 to the index; possible analysis error.
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:181)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:936)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1091)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:714)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)
at
org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)
at
org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:391)
at
org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:43)
at
org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:360)
at
org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:30)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
at
org.apache.solr.core.SolrCore.execute(SolrCore.java:2033)
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.AlreadyClosedException: this
IndexWriter is closed
at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724)
at
org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738)
at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1488)
at
org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)
... 58 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/lucene/codecs/encrypted/DummyCipherFactory
at
org.apache.lucene.codecs.encrypted.DummyEncryptedLucene60Codec$DummyEncryptedLucene50PostingsFormat.getCipherFactory(DummyEncryptedLucene60Codec.java:67)
at
org.apache.lucene.codecs.encrypted.EncryptedLucene50PostingsFormat.fieldsConsumer(EncryptedLucene50PostingsFormat.java:479)
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:196)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:107)
at
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134)
at
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:423)
at
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
at
org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2815)
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2989)
at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2956)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:619)
at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1453)
at
org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1256)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
Caused by: java.lang.ClassNotFoundException:
org.apache.lucene.codecs.encrypted.DummyCipherFactory
at
java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at
java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 22 more
From the output I assume that Solr is not picking up the jar with this
custom codec, and I don't get why?
What I find strange is that, I have also tried to run this codec with
Lucene and eclipse by writing some code for indexing and searching and
same codec jar is working just fine.
System.*/out/*.println("Indexing to directory '"+
indexPath+ "'...");
Directory dir=
FSDirectory./open/(Paths./get/(indexPath));
Analyzer analyzer= *new*StandardAnalyzer();
IndexWriterConfig iwc=
*new*IndexWriterConfig(analyzer);
// set encrypted codec //This encrypted Codec is being used for indexing
iwc.setCodec(*new*DummyEncryptedLucene60Codec());
*if*(create) {
// Create a new index in the directory, removing any
// previously indexed documents:
iwc.setOpenMode(OpenMode.*/CREATE/*);
} *else*{
// Add new documents to an existing index:
iwc.setOpenMode(OpenMode.*/CREATE_OR_APPEND/*);
}
IndexWriter writer= *new*IndexWriter(dir, iwc);
/indexDocs/(writer, docDir);
writer.close();
I'm guessing that As Solr uses Java SPI options to load this codec which
loads things differently, and there's something missing...?
My question is, what might I be doing wrong here or what's missing so
that codec jar can be picked up by Solr?
Thanks.
