Re: custom tokenizer error
baseTokenizer is reset in the #reset method.
Sarita
From: Jack Krupansky j...@basetechnology.com
To: solr-user@lucene.apache.org
Sent: Sunday, May 5, 2013 1:37 PM
Subject: Re: custom tokenizer error
I didn't notice any call to the reset method for your base tokenizer.
Is there any reason that you didn't just use char filters to replace colon
and periods with spaces?
-- Jack Krupansky
-Original Message-
From: Sarita Nair
Sent: Friday, May 03, 2013 2:43 PM
To: solr-user@lucene.apache.org
Subject: custom tokenizer error
I am using a custom Tokenizer, as part of analysis chain, for a Solr (4.2.1)
field. On trying to index, Solr throws a NullPointerException.
The unit tests for the custom tokenizer work fine. Any ideas as to what is
it that I am missing/doing incorrectly will be appreciated.
Here is the relevant schema.xml excerpt:
fieldType name=negated class=solr.TextField omitNorms=true
analyzer type=index
tokenizer
class=some.other.solr.analysis.EmbeddedPunctuationTokenizer$Factory/
filter class=solr.LowerCaseFilterFactory/
filter class=solr.EnglishPossessiveFilterFactory/
/analyzer
/fieldType
Here are the relevant pieces of the Tokenizer:
/**
* Intercepts each token produced by {@link
StandardTokenizer#incrementToken()}
* and checks for the presence of a colon or period. If found, splits
the token
* on the punctuation mark and adjusts the term and offset attributes of
the
* underlying {@link TokenStream} to create additional tokens.
*
*
*/
public class EmbeddedPunctuationTokenizer extends Tokenizer {
private static final Pattern PUNCTUATION_SYMBOLS = Pattern.compile([:.]);
private StandardTokenizer baseTokenizer;
private CharTermAttribute termAttr;
private OffsetAttribute offsetAttr;
private /*@Nullable*/ String tokenAfterPunctuation = null;
private int currentOffset = 0;
public EmbeddedPunctuationTokenizer(final Reader reader) {
super(reader);
baseTokenizer = new StandardTokenizer(Version.MINIMUM_LUCENE_VERSION,
reader);
// Two TokenStreams are in play here: the one underlying the current
// instance and the one underlying the StandardTokenizer. The attribute
// instances must be associated with both.
termAttr = baseTokenizer.addAttribute(CharTermAttribute.class);
offsetAttr = baseTokenizer.addAttribute(OffsetAttribute.class);
this.addAttributeImpl((CharTermAttributeImpl)termAttr);
this.addAttributeImpl((OffsetAttributeImpl)offsetAttr);
}
@Override
public void end() throws IOException {
baseTokenizer.end();
super.end();
}
@Override
public void close() throws IOException {
baseTokenizer.close();
super.close();
}
@Override
public void reset() throws IOException {
super.reset();
baseTokenizer.reset();
currentOffset = 0;
tokenAfterPunctuation = null;
}
@Override
public final boolean incrementToken() throws IOException {
clearAttributes();
if (tokenAfterPunctuation != null) {
// Do not advance the underlying TokenStream if the previous call
// found an embedded punctuation mark and set aside the substring
// that follows it. Set the attributes instead from the substring,
// bearing in mind that the substring could contain more embedded
// punctuation marks.
adjustAttributes(tokenAfterPunctuation);
} else if (baseTokenizer.incrementToken()) {
// No remaining substring from a token with embedded punctuation: save
// the starting offset reported by the base tokenizer as the current
// offset, then proceed with the analysis of token it returned.
currentOffset = offsetAttr.startOffset();
adjustAttributes(termAttr.toString());
} else {
// No more tokens in the underlying token stream: return false
return false;
}
return true;
}
private void adjustAttributes(final String token) {
Matcher m = PUNCTUATION_SYMBOLS.matcher(token);
if (m.find()) {
int index = m.start();
offsetAttr.setOffset(currentOffset, currentOffset + index);
termAttr.copyBuffer(token.toCharArray(), 0, index);
tokenAfterPunctuation = token.substring(index + 1);
// Given that the incoming token had an embedded punctuation mark,
// the starting offset for the substring following the punctuation
// mark will be 1 beyond the end of the current token, which is the
// substring preceding embedded punctuation mark.
currentOffset = offsetAttr.endOffset() + 1;
} else if (tokenAfterPunctuation != null) {
// Last remaining substring following a previously detected embedded
// punctuation mark: adjust attributes based on its values.
int length = tokenAfterPunctuation.length();
termAttr.copyBuffer(tokenAfterPunctuation.toCharArray(), 0, length);
offsetAttr.setOffset(currentOffset, currentOffset + length);
tokenAfterPunctuation = null;
}
// Implied else: neither is true so attributes from base tokenizer need
// no adjustments.
}
}
}
Solr throws the following error, in the 'else if' block of #incrementToken
2013-04-29 14:19:48,920 [http-thread-pool-8080(3)] ERROR
org.apache.solr.core.SolrCore
custom tokenizer error
I am using a custom Tokenizer, as part of analysis chain, for a Solr (4.2.1)
field. On trying to index, Solr throws a NullPointerException.
The unit tests for the custom tokenizer work fine. Any ideas as to what is it
that I am missing/doing incorrectly will be appreciated.
Here is the relevant schema.xml excerpt:
fieldType name=negated class=solr.TextField omitNorms=true
analyzer type=index
tokenizer
class=some.other.solr.analysis.EmbeddedPunctuationTokenizer$Factory/
filter class=solr.LowerCaseFilterFactory/
filter class=solr.EnglishPossessiveFilterFactory/
/analyzer
/fieldType
Here are the relevant pieces of the Tokenizer:
/**
* Intercepts each token produced by {@link
StandardTokenizer#incrementToken()}
* and checks for the presence of a colon or period. If found, splits the
token
* on the punctuation mark and adjusts the term and offset attributes of
the
* underlying {@link TokenStream} to create additional tokens.
*
*
*/
public class EmbeddedPunctuationTokenizer extends Tokenizer {
private static final Pattern PUNCTUATION_SYMBOLS = Pattern.compile([:.]);
private StandardTokenizer baseTokenizer;
private CharTermAttribute termAttr;
private OffsetAttribute offsetAttr;
private /*@Nullable*/ String tokenAfterPunctuation = null;
private int currentOffset = 0;
public EmbeddedPunctuationTokenizer(final Reader reader) {
super(reader);
baseTokenizer = new StandardTokenizer(Version.MINIMUM_LUCENE_VERSION, reader);
// Two TokenStreams are in play here: the one underlying the current
// instance and the one underlying the StandardTokenizer. The attribute
// instances must be associated with both.
termAttr = baseTokenizer.addAttribute(CharTermAttribute.class);
offsetAttr = baseTokenizer.addAttribute(OffsetAttribute.class);
this.addAttributeImpl((CharTermAttributeImpl)termAttr);
this.addAttributeImpl((OffsetAttributeImpl)offsetAttr);
}
@Override
public void end() throws IOException {
baseTokenizer.end();
super.end();
}
@Override
public void close() throws IOException {
baseTokenizer.close();
super.close();
}
@Override
public void reset() throws IOException {
super.reset();
baseTokenizer.reset();
currentOffset = 0;
tokenAfterPunctuation = null;
}
@Override
public final boolean incrementToken() throws IOException {
clearAttributes();
if (tokenAfterPunctuation != null) {
// Do not advance the underlying TokenStream if the previous call
// found an embedded punctuation mark and set aside the substring
// that follows it. Set the attributes instead from the substring,
// bearing in mind that the substring could contain more embedded
// punctuation marks.
adjustAttributes(tokenAfterPunctuation);
} else if (baseTokenizer.incrementToken()) {
// No remaining substring from a token with embedded punctuation: save
// the starting offset reported by the base tokenizer as the current
// offset, then proceed with the analysis of token it returned.
currentOffset = offsetAttr.startOffset();
adjustAttributes(termAttr.toString());
} else {
// No more tokens in the underlying token stream: return false
return false;
}
return true;
}
private void adjustAttributes(final String token) {
Matcher m = PUNCTUATION_SYMBOLS.matcher(token);
if (m.find()) {
int index = m.start();
offsetAttr.setOffset(currentOffset, currentOffset + index);
termAttr.copyBuffer(token.toCharArray(), 0, index);
tokenAfterPunctuation = token.substring(index + 1);
// Given that the incoming token had an embedded punctuation mark,
// the starting offset for the substring following the punctuation
// mark will be 1 beyond the end of the current token, which is the
// substring preceding embedded punctuation mark.
currentOffset = offsetAttr.endOffset() + 1;
} else if (tokenAfterPunctuation != null) {
// Last remaining substring following a previously detected embedded
// punctuation mark: adjust attributes based on its values.
int length = tokenAfterPunctuation.length();
termAttr.copyBuffer(tokenAfterPunctuation.toCharArray(), 0, length);
offsetAttr.setOffset(currentOffset, currentOffset + length);
tokenAfterPunctuation = null;
}
// Implied else: neither is true so attributes from base tokenizer need
// no adjustments.
}
}
}
Solr throws the following error, in the 'else if' block of #incrementToken
2013-04-29 14:19:48,920 [http-thread-pool-8080(3)] ERROR
org.apache.solr.core.SolrCore - java.lang.NullPointerException
at
org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:923)
at
org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1133)
at
org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:180)
at
some.other.solr.analysis.EmbeddedPunctuationTokenizer.incrementToken(EmbeddedPunctuationTokenizer.java:83)
at
org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:54)
Re: Solr 4.2.1 SSLInitializationException
:I'm confused ... it seems that you (or GlassFish) has created a :Catch-22... Glassfish specifies keystore as a system property, but does not require specifying the password for the keystore as a system property. GF uses a keychain mechanism, which requires the password to be passed from the DAS to access the keystore. :In SolrJ client code you can specify whatever HttpClient implementation :you want. In Solr (for it's use of talking to other nodes in distributed :search, which is what is indicated in your stack trace) :SystemDefaultHttpClient is hard coded. Thanks for the clarification. I am using DefaultHttpClient on the client side, but was hoping that there is a way around SystemDefaultHttpClient, in Solr. Looks like there is isn't any. Thanks for your help! From: Chris Hostetter hossman_luc...@fucit.org To: solr-user@lucene.apache.org solr-user@lucene.apache.org; Sarita Nair sarita...@yahoo.com Sent: Friday, April 12, 2013 3:07 PM Subject: Re: Solr 4.2.1 SSLInitializationException : Thanks for your response. As I mentioned in my email, I would prefer : the application to not have access to the keystore. Do you know if there I'm confused ... it seems that you (or GlassFish) has created a Catch-22... You say you don't want the application to have access to the keystore, but aparently you (or glassfish) is explicitly setting javax.net.ssl.keyStore to tell the application what keystore to use. The keystore you specify has a password set on it, but you are not telling the application what the password is, so it can't use that keystore. If you don't wnat to application to have access to the keystore at all, have you tried unsetting javax.net.ssl.keyStore ? : is a way of specifying a different HttpClient implementation (e.g. : DefaultHttpClient rather than SystemDefaultHttpClient) ? In SolrJ client code you can specify whatever HttpClient implementation you want. In Solr (for it's use of talking to other nodes in distributed search, which is what is indicated in your stack trace) SystemDefaultHttpClient is hard coded. -Hoss
IOexception, when using Solr 4.2.1 for indexing
Hi All,
I get the error below on trying to index using Solr 4.2.1. I have a single
core setup and use HttpSolrServer with DefaultHttpClient to talk to Solr.
#Here is how HttpSolrServer is instantiated:
solrServer = new HttpSolrServer( baseURL,
configurator.createHttpClient( new BasicHttpParams( ) ) );
#DefaultHttpClient creation:
public DefaultHttpClient createHttpClient( HttpParams parameters ) {
DefaultHttpClient httpClient = new DefaultHttpClient( connectionManager,
parameters );
httpClient.setRoutePlanner( routePlanner );
return httpClient;
}
Any ideas on what is it that I am doing incorrectly will be appreciated.
Thanks!
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException
occured when talking to server at: http://example.com:8080/solr-server-4.2.1
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:416)
~[solr-solrj-4.2.1.jar:4.2.1 1461071 - mark - 2013-03-26 08:28:42]
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
~[solr-solrj-4.2.1.jar:4.2.1 1461071 - mark - 2013-03-26 08:28:42]
at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
~[solr-solrj-4.2.1.jar:4.2.1 1461071 - mark - 2013-03-26 08:28:42]
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
~[solr-solrj-4.2.1.jar:4.2.1 1461071 - mark - 2013-03-26 08:28:42]
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
~[solr-solrj-4.2.1.jar:4.2.1 1461071 - mark - 2013-03-26 08:28:42]
at
com.qpidhealth.qpid.solr.SolrService.saveOrUpdate(SolrService.java:117)
~[classes/:na]
... 84 common frames omitted
Caused by: org.apache.http.client.ClientProtocolException: null
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
~[httpclient-4.2.2.jar:4.2.2]
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
~[httpclient-4.2.2.jar:4.2.2]
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
~[httpclient-4.2.2.jar:4.2.2]
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353)
~[solr-solrj-4.2.1.jar:4.2.1 1461071 - mark - 2013-03-26 08:28:42]
... 89 common frames omitted
Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot retry
request with a non-repeatable request entity. The cause lists the reason the
original request fail$
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
~[httpclient-4.2.2.jar:4.2.2]
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
~[httpclient-4.2.2.jar:4.2.2]
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
~[httpclient-4.2.2.jar:4.2.2]
... 92 common frames omitted
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
~[na:1.7.0_05]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
~[na:1.7.0_05]
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
~[na:1.7.0_05]
at
org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:167)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:110)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:165)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:92)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
~[httpclient-4.2.2.jar:4.2.2]
at
org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
~[httpclient-4.2.2.jar:4.2.2]
at
org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
~[httpcore-4.2.2.jar:4.2.2]
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
~[httpcore-4.2.2.jar:4.2.2]
Re: Solr 4.2.1 SSLInitializationException
Hi Uwe, Thanks for your response. As I mentioned in my email, I would prefer the application to not have access to the keystore. Do you know if there is a way of specifying a different HttpClient implementation (e.g. DefaultHttpClient rather than SystemDefaultHttpClient) ? From: Uwe Klosa uwe.kl...@gmail.com To: solr-user@lucene.apache.org; Sarita Nair sarita...@yahoo.com Sent: Wednesday, April 10, 2013 2:58 AM Subject: Re: Solr 4.2.1 SSLInitializationException You have to add two new Java options to your Glassfish config (example if you use the standard keystore and truststore): asadmin create-jvm-options -- -Djavax.net.ssl.keyStorePassword=changeit asadmin create-jvm-options -- -Djavax.net.ssl.trustStorePassword=changeit /Uwe On 10 April 2013 03:59, Sarita Nair sarita...@yahoo.com wrote: Hi Chris, Thanks for your response. My understanding is that GlassFish specifies the keystore as a system property, but does not specify the password in order to protect it from snooping. There's a keychain that requires a password to be passed from the DAS in order to unlock the key for the keystore. Is there some way to specify a different HttpClient implementation (e.g. DefaultHttpClient rather than SystemDefaultHttpClient), as we don't want the application to have access to the keystore? I have also pasted the entire stack trace below: 2013-04-09 10:45:06,144 [main] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Solr. Check solr/home property and the logs 2013-04-09 10:45:06,224 [main] ERROR org.apache.solr.core.SolrCore - null:org.apache.http.conn.ssl.SSLInitializationException: Failure initializing default system SSL context at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:368) at org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:204) at org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:82) at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:118) at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:466) at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:179) at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:33) at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:115) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:105) at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:134) at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79) at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63) at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55) Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) at java.security.KeyStore.load(KeyStore.java:1214) at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:281) at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:366) ... 50 more Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770) ... 54 more From: Chris Hostetter hossman_luc...@fucit.org To: solr-user@lucene.apache.org solr-user@lucene.apache.org; Sarita Nair sarita...@yahoo.com Sent: Tuesday, April 9, 2013 1:31 PM Subject: Re: Solr 4.2.1 SSLInitializationException : Deploying Solr 4.2.1 to GlassFish 3.1.1 results in the error below. I : have seen similar problems being reported with Solr 4.2 Are you trying to use server SSL with glassfish? can you please post the full stack trace so we can see where this error is coming from. My best guess is that this is coming from the changes made in SOLR-4451
Solr 4.2.1 SSLInitializationException
Hi All, Deploying Solr 4.2.1 to GlassFish 3.1.1 results in the error below. I have seen similar problems being reported with Solr 4.2 and my take-away was that 4.2.1 contains the necessary fix. Any help with this will be appreciated. Thanks! 2013-04-09 10:45:06,144 [main] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Solr. Check solr/home property and the logs 2013-04-09 10:45:06,224 [main] ERROR org.apache.solr.core.SolrCore - null:org.apache.http.conn.ssl.SSLInitializationException: Failure initializing default system SSL context Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) at java.security.KeyStore.load(KeyStore.java:1214) at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:281) at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:366) ... 50 more Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
Re: Solr 4.2.1 SSLInitializationException
Hi Chris, Thanks for your response. My understanding is that GlassFish specifies the keystore as a system property, but does not specify the password in order to protect it from snooping. There's a keychain that requires a password to be passed from the DAS in order to unlock the key for the keystore. Is there some way to specify a different HttpClient implementation (e.g. DefaultHttpClient rather than SystemDefaultHttpClient), as we don't want the application to have access to the keystore? I have also pasted the entire stack trace below: 2013-04-09 10:45:06,144 [main] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Solr. Check solr/home property and the logs 2013-04-09 10:45:06,224 [main] ERROR org.apache.solr.core.SolrCore - null:org.apache.http.conn.ssl.SSLInitializationException: Failure initializing default system SSL context at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:368) at org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:204) at org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:82) at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:118) at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:466) at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:179) at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:33) at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:115) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:105) at org.apache.solr.handler.component.HttpShardHandlerFactory.init(HttpShardHandlerFactory.java:134) at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79) at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63) at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55) Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) at java.security.KeyStore.load(KeyStore.java:1214) at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:281) at org.apache.http.conn.ssl.SSLSocketFactory.createSystemSSLContext(SSLSocketFactory.java:366) ... 50 more Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770) ... 54 more From: Chris Hostetter hossman_luc...@fucit.org To: solr-user@lucene.apache.org solr-user@lucene.apache.org; Sarita Nair sarita...@yahoo.com Sent: Tuesday, April 9, 2013 1:31 PM Subject: Re: Solr 4.2.1 SSLInitializationException : Deploying Solr 4.2.1 to GlassFish 3.1.1 results in the error below. I : have seen similar problems being reported with Solr 4.2 Are you trying to use server SSL with glassfish? can you please post the full stack trace so we can see where this error is coming from. My best guess is that this is coming from the changes made in SOLR-4451 to use system defaults correctly when initializing HttpClient, which suggets that your problem is exactly what the error message says... Keystore was tampered with, or password was incorrect Is it possible that the default keystore password for your JVM (or as overridden by glassfish defaults - possibly using the javax.net.ssl.keyStore sysprop) has a password set on it? If so you need to confiure your JVM with the standard java system properties to specify what that password is. http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201303.mbox/%3c1364232676233-4051159.p...@n3.nabble.com%3E : 2013-04-09 10:45:06,144 [main] ERROR : org.apache.solr.servlet.SolrDispatchFilter - Could not start Solr. Check solr/home property and the logs : 2013-04-09 10:45:06,224 [main] ERROR : org.apache.solr.core.SolrCore
Re: Retrieving Term vectors
Thanks much, Koji. That was indeed the problem. I did not realize that one has to do all three of these fieldType.setStoreTermVectors(true); fieldType.setStoreTermVectorPositions(true); fieldType.setStoreTermVectorOffsets(true); in order to store positions and offsets. -Sarita From: Koji Sekiguchi k...@r.email.ne.jp To: solr-user@lucene.apache.org Sent: Tuesday, March 19, 2013 7:02 PM Subject: Re: Retrieving Term vectors Hi Sarita, I've not dug into your code detail but my first impression is that you are missing store term positions? FieldType fieldType = new FieldType(); IndexOptions indexOptions = IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS; fieldType.setIndexOptions(indexOptions); fieldType.setIndexed(true); fieldType.setStoreTermVectors(true); fieldType.setStored(true); Document doc = new Document(); doc.add(new Field(content, one quick brown fox jumped over one lazy dog, fieldType)); I think you need: fieldType.setStoreTermVectorPositions(true); if you want term vector positions later. koji -- http://soleami.com/blog/lucene-4-is-super-convenient-for-developing-nlp-tools.html
Retrieving Term vectors
Hi All,
I am in the process upgrading from Solr 3.6.2 to Solr 4.1 and have been running
into
problems with retrieving Term vector information.
Below is the test and source code. The tests fails with a
NullPointerException, because DocsAndPositionsEnum is always null, despite the
fact that I have stored term vectors. Any ideas on what is it that I am doing
incorrectly, will be greatly appreciated.
public class LuceneUtilTest {
private final RAMDirectory ramDirectory = new RAMDirectory();
private static final String TERM_POSITION_PROVIDER = term position provider;
private AtomicReader atomicReader;
private DirectoryReader dr;
@BeforeClass
public void init() throws IOException {
IndexWriterConfig iwc = new IndexWriterConfig(Version.LUCENE_41, new
StandardAnalyzer(
Version.LUCENE_41));
iwc.setOpenMode(OpenMode.CREATE);
IndexWriter writer = new IndexWriter(ramDirectory, iwc);
FieldType fieldType = new FieldType();
IndexOptions indexOptions =
IndexOptions.DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS;
fieldType.setIndexOptions(indexOptions);
fieldType.setIndexed(true);
fieldType.setStoreTermVectors(true);
fieldType.setStored(true);
Document doc = new Document();
doc.add(new Field(content, one quick brown fox jumped over one lazy dog,
fieldType));
writer.addDocument(doc);
writer.commit();
writer.close();
dr = DirectoryReader.open(ramDirectory);
atomicReader = dr.leaves().get(0).reader();
}
@DataProvider(name = TERM_POSITION_PROVIDER)
public Object[][] getTermPositions() {
ListObject[] data = new ArrayListObject[]();
data.add(new Object[] { brown, new int[] { 2 } });
data.add(new Object[] { dog, new int[] { 8 } });
data.add(new Object[] { fox, new int[] { 3 } });
data.add(new Object[] { jumped, new int[] { 4 } });
data.add(new Object[] { lazy, new int[] { 7 } });
data.add(new Object[] { one, new int[] { 0, 6 } });
data.add(new Object[] { over, new int[] { 5 } });
data.add(new Object[] { quick, new int[] { 1 } });
return data.toArray(new Object[data.size()][]);
}
@AfterClass
public void destroy() throws IOException {
atomicReader.close();
}
@Test(dataProvider = TERM_POSITION_PROVIDER)
public void testGetTermPositionMap(String query, int[] position) throws
IOException,
ParseException {
MapString, int[] posMap = LuceneUtil.getTermPositionMap(atomicReader, 0,
content);
Assert.assertEquals(posMap.get(query)[0], position[0]);
}
}
#Method being tested
public final class LuceneUtil {
/**
* Private constructor to prevent instantiation.
*/
private LuceneUtil() {}
/**
*
* @param reader the index reader
* @param fieldName name of the field of interest
* @param docId internal doc ID of the document of interest
* @return all Terms present in the requested field
* @throws IOException on IndexReader error
*/
public static Terms getTerms(final AtomicReader reader, final String fieldName,
final int docId) throws IOException{
return reader.getTermVector(docId, fieldName);
}
/**
* Returns a map of the terms and their token positions for a field in a
* document. The map may be empty because vector information is not available
* for the requested field, or because the analyzer assigned to it found no
* terms in the original document field at index time.
*
* @param reader Lucene index reader (for access to term vector info)
* @param docId the internal Lucene ID of the document of interest
* @param fieldName name of the field of interest
* @return a map of term/positions pairs; the map may be empty.
* @throws IOException on IndexReader error
*/
public static MapString, int[] getTermPositionMap(final AtomicReader reader,
final int docId, final String fieldName) throws IOException {
MapString, int[] termPosMap = new HashMapString, int[]();
Terms terms = LuceneUtil.getTerms(reader, fieldName, docId);
if(terms!=null) {
TermsEnum termsEnum = terms.iterator(TermsEnum.EMPTY);
BytesRef term;
while ((term=termsEnum.next())!=null) {
String docTerm = term.utf8ToString();
DocsAndPositionsEnum docPosEnum =
termsEnum.docsAndPositions(reader.getLiveDocs(),
null,
DocsAndPositionsEnum.FLAG_OFFSETS);
docPosEnum.nextDoc();
int freq = docPosEnum.freq();
int[] posArray = new int[freq];
for (int i = 0; i freq; i++) {
int position = docPosEnum.nextPosition();
posArray[i]=position;
}
termPosMap.put(docTerm, posArray);
}
}
return termPosMap;
}
}
