NPE from UpdateLog.lookup(UpdateLog.java:706) (Version 4.8.1)

[Shay Sofer]

Hi all,

During daily work I got NPE at mention line. full method below.
Anyone know if it's a known bug ? should I open a ticket for Solr ?

Thanks in advance, 
Shay.

public Object More ...lookup(BytesRef indexedId) {
683 LogPtr entry;
684 TransactionLog lookupLog;
685 
686 synchronized (this) {
687   entry = map.get(indexedId);
688   lookupLog = tlog;  // something found in "map" will always be in 
"tlog"
689   // SolrCore.verbose("TLOG: lookup: for id 
",indexedId.utf8ToString(),"in 
map",System.identityHashCode(map),"got",entry,"lookupLog=",lookupLog);
690   if (entry == null && prevMap != null) {
691 entry = prevMap.get(indexedId);
692 // something found in prevMap will always be found in preMapLog 
(which could be tlog or prevTlog)
693 lookupLog = prevMapLog;
694 // SolrCore.verbose("TLOG: lookup: for id 
",indexedId.utf8ToString(),"in 
prevMap",System.identityHashCode(map),"got",entry,"lookupLog=",lookupLog);
695   }
696   if (entry == null && prevMap2 != null) {
697 entry = prevMap2.get(indexedId);
698 // something found in prevMap2 will always be found in preMapLog2 
(which could be tlog or prevTlog)
699 lookupLog = prevMapLog2;
700 // SolrCore.verbose("TLOG: lookup: for id 
",indexedId.utf8ToString(),"in 
prevMap2",System.identityHashCode(map),"got",entry,"lookupLog=",lookupLog);
701   }
702 
703   if (entry == null) {
704 return null;
705   }
706   lookupLog.incref();
707 }
708 
709 try {
710   // now do the lookup outside of the sync block for concurrency
711   return lookupLog.lookup(entry.pointer);
712 } finally {
713   lookupLog.decref();
714 }
715 
716   }
717

RE: Solr authentication

[Shay Sofer]

Thanks for the quick response.

1.  I'm using Solr with Jetty.
2. I'm using Java to access Solr, so I need a way to pass / add this 
authentication as well.




-Original Message-
From: Tim Dunphy [mailto:bluethu...@gmail.com] 
Sent: Tuesday, November 04, 2014 3:22 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr authentication

Hi Shay,

I'm new to using Solr myself. But what I've done to solve this problem is to 
run Solr via Tomcat. Then I put Apache in front of Tomcat using mod_jk and made 
Solr accessible via SSL on port 443. I also put basic authentication in front 
of Apache. That way you have to enter a username and password to log in.

Then I made port 8080 (the native port for Apache Tomcat) inaccessible using 
the firewall. So that the only way to access the Solr instance was through 
Apache and entering your password. With everything going over SSL. It's very 
secure.

>From what I read about Solr, there are no security considerations (such as 
>using a password for access) built in. So the only way to achieve some level 
>of security without doing what I just did is to secure it with a firewall. 
>Making your Solr instance accessible only from certain IPs. Please someone 
>correct me if I'm wrong about that.

But the way I did it with running Solr with Apache and SSL and mod_jk over 
tomcat is pretty easy. If you google it you will find plenty of useful guides 
out there on how to do this. I'd recommend taking that approach.

Tim

Sent from my iPhone

> On Nov 4, 2014, at 7:53 AM, Shay Sofer  wrote:
> 
> Hi,
> 
> I want that my Solr web connection will be protected by username and password.
> 
> When someone try to get to - 1.1.1.1:8983/Solr, he can do it only after login 
> (with known users).
> 
> Is it possible ?
> 
> Thanks,
> Shay.

Email secured by Check Point

Solr authentication

[Shay Sofer]

Hi,

I want that my Solr web connection will be protected by username and password.

When someone try to get to - 1.1.1.1:8983/Solr, he can do it only after login 
(with known users).

Is it possible ?

Thanks,
Shay.

Solr source code

[Shay Sofer]

Hi,

What is the process regarding modify Solr source code (legal part)?

In addition, who should I update for this bug and fix so Solr team will 
consider using it.

Thanks,
Shay.

RE: Solr issue

[Shay Sofer]

Hi,

None of the above fix my problem.

The problem is with QueryComponent.java:

  private void groupedFinishStage(final ResponseBuilder rb) {
// To have same response as non-distributed request.
GroupingSpecification groupSpec = rb.getGroupingSpec();
if (rb.mergedTopGroups.isEmpty()) {
  for (String field : groupSpec.getFields()) {
rb.mergedTopGroups.put(field, new TopGroups(null, null, 0, 0, new 
GroupDocs[]{}, Float.NaN));
  }
  rb.resultIds = new HashMap<>();
}

We are cleaning rb.resultIds, and in HighlightComponent we are trying to read 
from it and NPE is established.



-Original Message-
From: Patanachai Tangchaisin [mailto:patanachai.tangchai...@wizecommerce.com] 
Sent: Thursday, August 28, 2014 10:42 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr issue

Hi Shay,

I'm not quite sure about this.
But, I think it is get fixed with this.

https://issues.apache.org/jira/browse/SOLR-6223
https://issues.apache.org/jira/browse/SOLR-4186
https://issues.apache.org/jira/browse/SOLR-4049

Could you try 4.10 from a svn branch and see if your problem is fixed?

Thanks,
Patanachai



On 08/28/2014 03:23 AM, Shay Sofer wrote:
> Hi,
>
> Version - 4.8.1
>
> While executing this solr query (from solr web UI):
>
> http://localhost:8983/solr/Global_A/select?q=%2Btext%3A%28shay*%29+&rows=100&fl=id%2CobjId%2Cnull&shards=http%3A%2F%2F127.0.0.1%3A8983%2Fsolr%2F0_A%2Chttp%3A%2F%2F127.0.0.1%3A8983%2Fsolr%2FGlobal_A&group=true&group.query=name__s%3Ashay&sort=name__s_sort+asc&hl=true<http://localhost:8983/solr/cpm_Global_A/select?q=%2Btext%3A%28shay*%29+&rows=100&fl=id%2CobjId%2Cnull&shards=http%3A%2F%2F127.0.0.1%3A8983%2Fsolr%2Fcpm_0_A%2Chttp%3A%2F%2F127.0.0.1%3A8983%2Fsolr%2Fcpm_Global_A&group=true&group.query=name__s%3Ashay&sort=name__s_sort+asc&hl=true>
>
> We got NullPointerException:
>
> java.lang.NullPointerException at 
> org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:189)
>  at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:330)
>  at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>  at org.apache.solr.core.SolrCore.execute(SolrCore.java:1952) at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:774)
>  at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
>  at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
>  at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>  at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) 
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) 
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
>  at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
>  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) 
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>  at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>  at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) 
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
>  at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>  at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>  at org.eclipse.jetty.server.Server.handle(Server.java:368) at 
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>  at 
> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
>  at 
> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
>  at 
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
>  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at 
> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at 
> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
>  at 
> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
>  at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>  at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>  at java.lang.Thread.run(Threa

Solr issue

[Shay Sofer]

Hi,

Version - 4.8.1

While executing this solr query (from solr web UI):

http://localhost:8983/solr/Global_A/select?q=%2Btext%3A%28shay*%29+&rows=100&fl=id%2CobjId%2Cnull&shards=http%3A%2F%2F127.0.0.1%3A8983%2Fsolr%2F0_A%2Chttp%3A%2F%2F127.0.0.1%3A8983%2Fsolr%2FGlobal_A&group=true&group.query=name__s%3Ashay&sort=name__s_sort+asc&hl=true

We got NullPointerException:

java.lang.NullPointerException at 
org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:189)
 at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:330)
 at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:1952) at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:774) 
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
 at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
 at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) 
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) 
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) 
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) 
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
 at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
 at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) 
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
 at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
 at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) 
at org.eclipse.jetty.server.Server.handle(Server.java:368) at 
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
 at 
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
 at 
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
 at 
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at 
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at 
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
 at 
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) 
at java.lang.Thread.run(Thread.java:722)

Seems like integration of Grouping + shards + highlighting cause this 
NullPointerException.

Anyone familiar with this issue?

Thanks,
Shay.

Null pointer on multi-core search

[Shay Sofer]

Hi,

I'm using solr 4.8.1and with following scenario I got a null pointer exception:


1.   I'm trying to search over multi-cores and group search.

2.   SearchHandler is called and when executing
for(SearchComponent c : components) {
 c.finishStage(rb);
}


3.   With QueryComponent.finishStage, this code is called:

if (rb.grouping()) {
  groupedFinishStage(rb);
} else {
  regularFinishStage(rb);
}

4.   And then -

  private void groupedFinishStage(final ResponseBuilder rb) {

// To have same response as non-distributed request.

GroupingSpecification groupSpec = rb.getGroupingSpec();

if (rb.mergedTopGroups.isEmpty()) {

  for (String field : groupSpec.getFields()) {

rb.mergedTopGroups.put(field, new TopGroups(null, null, 0, 0, new 
GroupDocs[]{}, Float.NaN));

  }

  rb.resultIds = new HashMap<>();

}


5.   As you can see the marked line is initializing resultsIds.

6.   And then when get to HighlightComponent.finishStage in and trying to 
execute:

ShardDoc sdoc = rb.resultIds.get(id);

int idx = sdoc.positionInResponse;

arr[idx] = new NamedList.NamedListEntry<>(id, hl.getVal(i));

7.   resultsIds is empty and then sdoc.positionInResponse get a null 
pointer exception.

Hope that my description is clear.

Thanks,
Shay.

RE: Exact search with special characters

[Shay Sofer]

Hi,

Thanks for your reply.

I thought that google search work the same (quotes stand for exact match).

Example for my demands: 
Objects:
- test host
- test_host
-test $host
-test-host

When I'll search for test host I'll get all above  results.

When I'll search for "test host" Ill get only test host

Also, when search for partial string like test / host I'll get all above 
results.

Thanks.

-Original Message-
From: Jack Krupansky [mailto:j...@basetechnology.com] 
Sent: Sunday, August 24, 2014 3:34 PM
To: solr-user@lucene.apache.org
Subject: Re: Exact search with special characters 

What precisely do you mean by the term "exact search". I mean, Solr (and
Lucene) do not have that concept for tokenized text fields.

Or did you simply mean "quoted phrase". In which case, you need to be aware 
that all the quotes do is assure that the terms occur in that order or in close 
proximity according to the default or specified "phrase slop" 
distance. But each term is still analyzed according to the analyzer for the 
field.

Technically, Lucene will in fact analyze the full quoted phrase as one stream, 
which for non-tokenized fields will be one term, but for any tokenized fields 
which split on white space, the phrase will be broken into separate tokens and 
special characters will tend to be removed as well. The keyword tokenizer will 
indeed treat the entire phrase as a single token, and the white space tokenizer 
will preserve special characters, but the standard tokenizer will not preserve 
either white space or special characters.

Nominally, the keyword tokenizer does generate a single term at least at the 
tokenization stage, but the world delimiter filter then splits individual terms 
into multiple terms, thus guaranteeing that a phrase with white space will be 
multiple terms and special characters are removed as well.

The other technicality is that quoting a phrase does prevent the phrase from 
being interpreted as query parser syntax, such as AND and OR operators or 
treating special characters as query parser operators.

But, the fact remains that a quoted phrase is not treated as an "exact" 
string literal for any normal tokenized fields.

Out of curiosity, what references have lead you to believe that a quoted phrase 
is an "exact match"?

Use a "string" (not "tokenized text") field if you wish to make an "exact 
match" on a literal string, but the concept of "exact match" is not supported 
for tokenized and filtered text fields.

So, please describe, in plain English, plus examples, exactly what you expect 
your analyzer to do, both in terms of how it treats text to be indexed and how 
you expect to be able to query that text.

-- Jack Krupansky

-Original Message-
From: Shay Sofer
Sent: Sunday, August 24, 2014 5:58 AM
To: solr-user@lucene.apache.org
Subject: Exact search with special characters

Hi all,

I have a docs that's indexed by text field with mention schema.

I have those docs names:

-  Test host

-  Test_host

-  Test-host

-  Test $host

When I'm trying to do exact search like: "test host"
All the results from above are shown as a results.

How can I use exact match so I'll will get only one result?

I prefer to do my changes in search time but if I need to change my schema 
please offer that.

Thanks,
Shay.


This is my schema:















Email secured by Check Point

Exact search with special characters

[Shay Sofer]

Hi all,

I have a docs that's indexed by text field with mention schema.

I have those docs names:

-  Test host

-  Test_host

-  Test-host

-  Test $host

When I'm trying to do exact search like: "test host"
All the results from above are shown as a results.

How can I use exact match so I'll will get only one result?

I prefer to do my changes in search time but if I need to change my schema 
please offer that.

Thanks,
Shay.


This is my schema:

RE: Solr search \ special cases

[Shay Sofer]

I call directly from Solr web api. Field type is string.

* should bring more results ? this is suffix search ? am I wrong ?

Thanks !

-Original Message-
From: Harshvardhan Ojha [mailto:ojha.harshvard...@gmail.com] 
Sent: Monday, August 11, 2014 1:40 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr search \ special cases

Hi Shay,

I believe + is treated as space, is it a rest call or api ? what is your field 
type ?

Regards
Harshvardhan Ojha


On Mon, Aug 11, 2014 at 4:04 PM, Shay Sofer  wrote:

> Hi,
>
> I have some strange cases while search with Solr.
>
> I have doc with names like: rule #22, rule +33, rule %44.
>
> When search for #22 or %55 or +33 Solr bring me as expected:  rule #22 
> and rule +33 and rule %44.
>
> But when appending star (*) to each search (#22*, +33*, %55*), just 
> the one with + sign bring rule +33, all other result none.
>
> Can someone explain?
>
> Thanks,
> Shay.
>


Email secured by Check Point

Solr search \ special cases

[Shay Sofer]

Hi,

I have some strange cases while search with Solr.

I have doc with names like: rule #22, rule +33, rule %44.

When search for #22 or %55 or +33 Solr bring me as expected:  rule #22 and rule 
+33 and rule %44.

But when appending star (*) to each search (#22*, +33*, %55*), just the one 
with + sign bring rule +33, all other result none.

Can someone explain?

Thanks,
Shay.

Solr score manager

[Shay Sofer]

Hi All,

I need a specific score mechanism.

I would like to sort my results based on customize scoring field.
scoring for example -



1.   If this is a new object - 100

2.   Edited - 80

3.   Recent search - 50

4.   Opened - 40
and some more actions...

And then when execute a new search they sorted based on score field.

Example:
Object 1 : opened  = 40.
Object 2: New = 100
Object 3: edited X 2 + recent search X 1 = 210.

Result:

Object 3
Object 2
Object 1

Any good article for this? Examples?
I'm using Solr with Java.

Thanks in advance,
Shay.

RE: Solr search

[Shay Sofer]

Thanks for your reply.

How can I support suffix search?

Name: Hello_world
Search: *world

And I'll get hello_world as a result.

Thanks in advance.



-Original Message-
From: Shawn Heisey [mailto:s...@elyograg.org] 
Sent: Wednesday, June 11, 2014 5:47 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr search

> Hi,
>
> Any suggestion for tokenizer / filter / other solutions that support 
> search in Solr as following -
>
> Use Case
>
> Input
>
> Solr should return
>
> All Results
>
> *
>
> All results
>
> Prefix Search
>
> Text*
>
> All data started by Text* (Prefix search)
>
> Exact Search
>
> "Auto Text"
>
> Exact match. Only Auto Text
>
> Partial (substring)
>
> *Text*
>
> All strings contains the text
>
>
> Now I'm using KeywordTokenizerFactory and WordDelimiterFilterFactory.
>
> My issue is with exact search:
> When I have document named hello_world, and I'm trying to do Exact 
> Search of hello, I got "hello_world" as a result (I want to get only 
> hello named doccuments).

The WordDelimeterFilter will split on the underscore, which means that the term 
"hello" is in the index for that document. Leave that filter out if you really 
do want an exact match.

Searching for "*" by itself is not how you match all documents. It may work, 
but it is a wildcard search, which means under the covers that it's a search 
for every term in the index for that field. It's SLOW. The special shortcut *:* 
(this must be the entire query with no field name, and I'm assuming the 
standard query parser here) is what you want for all documents. In terms of 
user input, this is what you want to use when the user leaves the search box 
empty. If you're using dismax or edismax, then you would send an empty q 
parameter or leave it off entirely, and define a default q.alt parameter in 
solrconfig.xml, set to *:* for all docs.

Thanks,
Shawn





Email secured by Check Point

Solr search

[Shay Sofer]

Hi,

Any suggestion for tokenizer / filter / other solutions that support search in 
Solr as following -

Use Case

Input

Solr should return

All Results

*

All results

Prefix Search

Text*

All data started by Text* (Prefix search)

Exact Search

"Auto Text"

Exact match. Only Auto Text

Partial (substring)

*Text*

All strings contains the text


Now I'm using KeywordTokenizerFactory and WordDelimiterFilterFactory.

My issue is with exact search:
When I have document named hello_world, and I'm trying to do Exact Search of 
hello, I got "hello_world" as a result (I want to get only hello named 
documents).

Thanks in advance,
Shay.