Hi Renaud,
I apologize for a typo. I have the jars in directory called "lib" instead of
"Lib".(Mistyped while I posted).
I have also added the Encrypted codec Jar to the directory -
[$SolrDir]\server\solr-webapp\webapp\WEB-INF\lib and all of a sudden it seems
to be working fine now.
I don’t really understand this behavior of Solr as per Solr wiki all external
jars should be in [Solr_Core\ instance Dir.]\lib and when I check logs indeed
it is loading the jar from this directory.
INFO (coreLoadExecutor-6-thread-1) [ ] o.a.s.c.SolrResourceLoader new
SolrResourceLoader for directory: 'C:\solr-6.0.0\solr-6.0.0\server\solr\demo'
1190 INFO (coreLoadExecutor-6-thread-2) [ ] o.a.s.c.SolrResourceLoader new
SolrResourceLoader for directory: 'C:\solr-6.0.0\solr-6.0.0\server\solr\test'
1190 INFO (coreLoadExecutor-6-thread-1) [ ] o.a.s.c.SolrResourceLoader JNDI
not configured for solr (NoInitialContextEx)
1190 INFO (coreLoadExecutor-6-thread-2) [ ] o.a.s.c.SolrResourceLoader JNDI
not configured for solr (NoInitialContextEx)
1190 INFO (coreLoadExecutor-6-thread-1) [ ] o.a.s.c.SolrResourceLoader using
system property solr.solr.home: C:\solr-6.0.0\solr-6.0.0\server\solr
1190 INFO (coreLoadExecutor-6-thread-2) [ ] o.a.s.c.SolrResourceLoader using
system property solr.solr.home: C:\solr-6.0.0\solr-6.0.0\server\solr
1191 INFO (coreLoadExecutor-6-thread-2) [ ] o.a.s.c.SolrResourceLoader
Adding
'file:/C:/solr-6.0.0/solr-6.0.0/server/solr/test/lib/SecureCodec-SecureCodecSNAPSHOT.jar'
to classloader
1206 INFO (main) [ ] o.e.j.s.h.ContextHandler Started
o.e.j.w.WebAppContext@a67c67e{/solr,file:///C:/solr-6.0.0/solr-6.0.0/server/solr-webapp/webapp/,AVAILABLE}{C:\solr-6.0.0\solr-6.0.0\server/solr-webapp/webapp
After Indexing I get _6_EncryptedLucene50_0.doc ; _6_EncryptedLucene50_0.Pos
; _6_EncryptedLucene50_0.Tim ; _6_EncryptedLucene50_0.Tip etc. so I assume
Solr is working fine and performing indexing with Encrypted codec.
build lucene/solr trunk and applied the latest patch to the Branch 6_x and
after compiling all seems to be working fine now.
Thanks,
Mohit
-----Original Message-----
From: Renaud Delbru [mailto:renaud@siren.solutions]
Sent: Tuesday, June 21, 2016 11:39 AM
To: solr-user@lucene.apache.org
Subject: Re: Encryption to Solr indexes – Using Custom Codec
Hi,
maybe it is the way you created the jar ? Why not applying the patch to
lucene/solr trunk and use ant jar instead to get the codecs jar created for you
?
Also, I think the directory where you put the jars should be called "lib"
instead of "Lib".
you can try also to use the lib directives in your solrconfig.xml [1]
[1]
https://cwiki.apache.org/confluence/display/solr/Lib+Directives+in+SolrConfig
--
Renaud Delbru
On 20/06/16 15:42, Sidana, Mohit wrote:
> Hello,
>
> As Part of my studies I am exploring the solutions which can be used
> for Lucene/Solr Index encryption.
>
> I found the patch open on Apache JIRA- Codec for index-level
> encryption <https://issues.apache.org/jira/browse/LUCENE-6966>(LUCENE-6966).
> https://issues.apache.org/jira/browse/LUCENE-6966 and I am currently
> trying to test this Custom codec with Solr to perform secure search
> over some sensitive records.
>
> I've decided to follow the path described in Solr wiki, setting up
> Simple Text Codec and further tried to use Encrypted codec Source.
>
> *Here are the additional details.*
>
> I've created a basic jar file out of this source code (Build it as a
> jar from Eclipse using Maven Plugin).
>
> The Solr installation I'm using to test this is the Solr 6.0.0
> unzipped, and started via its embedded Jetty server and using the single core.
>
> I've placed my jar with the codec in [My_Core\ instance Dir.]\ Lib
>
> In:
>
> [$SolrDir]\Solr\ My_Core \conf\*solrconfig.xml*
>
> I've added the following lines:
>
> |<lib dir="./lib/" regex=".*\.jar"/> |||
>
> |<codecFactory class="solr.SchemaCodecFactory"/>|
>
> Then in the *schema.xml* file, I've declared some field and field
> Types that should use this codec:
>
> |<field name="City" type="strings"/>|||
>
> | <field name="Country" type="strings"/>|
>
> | <field name="Name" type="strings"/>|
>
> | <field name="Payment" type="strings"/>|
>
> | <field name="Price" type="tlongs"/>|
>
> | <field name="Product" type="strings"/>|
>
> | <field name="State" type="strings"/>|
>
> ||
>
> |<!-- set the per field posting format as below -->|
>
> |<fieldType name="strings" class="solr.StrField" sortMissingLast="true"
> docValues="true" multiValued="true"
> postingsFormat="EncryptedLucene50"/>|
>
> |<fieldType name="tlongs" class="solr.TrieLongField"
> positionIncrementGap="0" docValues="true" multiValued="true"
> precisionStep="8" postingsFormat="EncryptedLucene50"/>|
>
> I'm pretty sure I've followed all the steps described in Solr Wiki;
> however, when I actually try to use custom codec implementation (named
> "Encrypted Codec") to index some sample CSV data using simple post
> tool
>
> java -Dtype=text/csv -Durl=http://localhost:8983/solr/My_Core /update
> -jar post.jar Sales.csv
>
> and I have also tried doing the same with SolrJ but I have faced the
> same error.
>
> SolrClient _server_=
> *new*HttpSolrClient("http://localhost:8983/solr/My_Core ");
>
> SolrInputDocument doc= *new*SolrInputDocument();
>
> doc.addField("id", "1234");
>
> doc.addField("name", "A lovely summer holiday");
>
> *try*{
>
> server.add(doc);
>
> server.commit();
>
> System.*/out/*.println("Document added!");
>
> } *catch*(SolrServerException | IOException e) {
>
> e.printStackTrace();
>
> }
>
> }
>
> }
>
> I get the attached errors in Solr log.
>
> org.apache.solr.common.SolrException: Exception writing document id
> b3e01ada-d0f1-4ddf-ad6a-2828bfe619a3 to the index; possible analysis error.
>
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler
> 2.java:181)
>
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpda
> teProcessorFactory.java:68)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory
> $AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProces
> sorFactory.java:335)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processA
> dd(FieldMutatingUpdateProcessor.java:117)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processA
> dd(FieldMutatingUpdateProcessor.java:117)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processA
> dd(FieldMutatingUpdateProcessor.java:117)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processA
> dd(FieldMutatingUpdateProcessor.java:117)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFacto
> ry$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processA
> dd(FieldMutatingUpdateProcessor.java:117)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd
> (DistributedUpdateProcessor.java:936)
>
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd
> (DistributedUpdateProcessor.java:1091)
>
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd
> (DistributedUpdateProcessor.java:714)
>
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdatePr
> ocessor.processAdd(LogUpdateProcessorFactory.java:103)
>
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(Upd
> ateRequestProcessor.java:48)
>
> at
> org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFa
> ctory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdat
> eProcessorFactory.java:93)
>
> at
> org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:
> 391)
>
> at
> org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoade
> r.java:43)
>
> at
> org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:3
> 60)
>
> at
> org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:30)
>
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandl
> er.java:94)
>
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Con
> tentStreamHandlerBase.java:69)
>
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rBase.java:155)
>
> at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2033)
>
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
>
> at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter
> .java:229)
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter
> .java:184)
>
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletH
> andler.java:1668)
>
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:
> 581)
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.ja
> va:143)
>
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java
> :548)
>
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandle
> r.java:226)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandle
> r.java:1160)
>
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:5
> 11)
>
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler
> .java:185)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler
> .java:1092)
>
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.ja
> va:141)
>
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Conte
> xtHandlerCollection.java:213)
>
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColle
> ction.java:119)
>
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.
> java:134)
>
> at
> org.eclipse.jetty.server.Server.handle(Server.java:518)
>
> at
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java
> :244)
>
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(Abstrac
> tConnection.java:273)
>
> at
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint
> .java:93)
>
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAn
> dRun(ExecuteProduceConsume.java:246)
>
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(Execu
> teProduceConsume.java:156)
>
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool
> .java:654)
>
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.
> java:572)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.lucene.store.AlreadyClosedException: this
> IndexWriter is closed
>
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724)
>
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738)
>
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:14
> 88)
>
> at
> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdat
> eHandler2.java:282)
>
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandle
> r2.java:214)
>
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler
> 2.java:169)
>
> ... 58 more
>
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/lucene/codecs/encrypted/DummyCipherFactory
>
> at
> org.apache.lucene.codecs.encrypted.DummyEncryptedLucene60Codec$DummyEn
> cryptedLucene50PostingsFormat.getCipherFactory(DummyEncryptedLucene60C
> odec.java:67)
>
> at
> org.apache.lucene.codecs.encrypted.EncryptedLucene50PostingsFormat.fie
> ldsConsumer(EncryptedLucene50PostingsFormat.java:479)
>
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.
> write(PerFieldPostingsFormat.java:196)
>
> at
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.
> java:107)
>
> at
> org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChai
> n.java:134)
>
> at
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriter
> PerThread.java:423)
>
> at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:5
> 02)
>
> at
> org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWrite
> r.java:614)
>
> at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.
> java:2815)
>
> at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:29
> 89)
>
> at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2956)
>
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler
> 2.java:619)
>
> at
> org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1
> 453)
>
> at
> org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1256)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511
> )
>
> at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511
> )
>
> at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.la
> mbda$execute$0(ExecutorUtil.java:229)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> ava:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:617)
>
> ... 1 more
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.lucene.codecs.encrypted.DummyCipherFactory
>
> at
> java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
> at
> java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at
> java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)
>
> at
> java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> ... 22 more
>
> From the output I assume that Solr is not picking up the jar with
> this custom codec, and I don't get why?
>
> What I find strange is that, I have also tried to run this codec with
> Lucene and eclipse by writing some code for indexing and searching and
> same codec jar is working just fine.
>
> System.*/out/*.println("Indexing to directory
> '"+
> indexPath+ "'...");
>
> Directory dir=
> FSDirectory./open/(Paths./get/(indexPath));
>
> Analyzer analyzer= *new*StandardAnalyzer();
>
> IndexWriterConfig iwc=
> *new*IndexWriterConfig(analyzer);
>
> // set encrypted codec //This encrypted Codec is being used for
> indexing
>
> iwc.setCodec(*new*DummyEncryptedLucene60Codec());
>
> *if*(create) {
>
> // Create a new index in the directory, removing any
>
> // previously indexed documents:
>
> iwc.setOpenMode(OpenMode.*/CREATE/*);
>
> } *else*{
>
> // Add new documents to an existing index:
>
> iwc.setOpenMode(OpenMode.*/CREATE_OR_APPEND/*);
>
> }
>
> IndexWriter writer= *new*IndexWriter(dir, iwc);
>
> /indexDocs/(writer, docDir);
>
> writer.close();
>
> I'm guessing that As Solr uses Java SPI options to load this codec
> which loads things differently, and there's something missing...?
>
> My question is, what might I be doing wrong here or what's missing so
> that codec jar can be picked up by Solr?
>
> Thanks.
>
