[jira] [Assigned] (LUCENE-7091) Add doc values support to MemoryIndex

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley reassigned LUCENE-7091:


Assignee: David Smiley

> Add doc values support to MemoryIndex
> -
>
> Key: LUCENE-7091
> URL: https://issues.apache.org/jira/browse/LUCENE-7091
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Martijn van Groningen
>Assignee: David Smiley
> Attachments: LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch, 
> LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch
>
>
> Sometimes queries executed via the MemoryIndex require certain things to be 
> stored as doc values. Today this isn't possible because the memory index 
> doesn't support this and these queries silently return no results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr in Linux Platform

[Shawn Heisey]

On 3/14/2016 8:31 AM, Adel Mohamed Khalifa wrote:
> I build a website (Java EE ) and want to search in some json files so
> I installed the solr server in an Ubuntu server and create a new core
> then indexing json files and the web searched correctly when I moved
> my code from windows to the server it stopped and cannot connect to
> solr server I try to debug using netbeans in Ubuntu it’s stopped and
> there is no exception on this statement (SolrServer server = new
> HttpSolrServer(“localhost:8983/solr/SearchCore”) ).
>
>  
>
> I need for some help Please.
>
>  
>
> Note :- I attached the servlet I used to search and connect to solr
> server.
>

Don't set the "wt" or "indent" parameters.  You won't be interacting
with the actual text of the response -- all response access with SolrJ
is through Java objects.  Changing wt might just confuse SolrJ -- let it
use its normal binary response format.

The gson/json code you've got probably isn't going to do what you
expect.  If the "wt" parameter did not break the request (which might
happen), then what you get with the getResults method on the response
object will NOT be any standardized format like gson or json.  You will
need to access the SolrDocument object(s) from the SolrDocumentList, and
then access fields from SolrDocument.

Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-8844) Date math silently ignored for date strings

[Markus Jelsma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Markus Jelsma closed SOLR-8844.
---
   Resolution: Not A Problem
Fix Version/s: (was: 6.1)

There was a bug elsewhere in my code that caused this to pop-up. Date math 
works fine with full date strings.

> Date math silently ignored for date strings
> ---
>
> Key: SOLR-8844
> URL: https://issues.apache.org/jira/browse/SOLR-8844
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.5
>Reporter: Markus Jelsma
>Priority: Minor
>
> Consider the following query, ordered by date ascending: {code}
> http://localhost:8983/solr/logs/select?q=*:*=[2011-05-26T08:15:36Z%2B3DAY%20TO%20NOW/DAY]=time%20asc
> {code}
> Should not have a result set where the first entry entry has 
> 2011-05-26T08:15:36Z for the time field.
> It appears date math is just ignored, while i would expect it to work or 
> throw an error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-3475) Performance hit due to unnecessary syncronization

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley closed SOLR-3475.
--
Resolution: Fixed

This has long since been fixed.

> Performance hit due to unnecessary syncronization
> -
>
> Key: SOLR-3475
> URL: https://issues.apache.org/jira/browse/SOLR-3475
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
>Reporter: Varun Srivastava
>
> Method org.apache.solr.request.UnInvertedField.getUnInvertedField unnecessary 
> syncronize agiant cache object. The SolrCache implementations should them 
> selves be thread safe. For example FastLRUCache uses ConcurrentLRUCache which 
> in turn use java.util.concurrency structs to make the implementation 
> threadsafe.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8836) /update should return BAD REQUEST when invalid JSON provided

[Anshum Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193418#comment-15193418
 ] 

Anshum Gupta commented on SOLR-8836:


Thanks Jason. 
LGTM, I'll run the tests and commit.

> /update should return BAD REQUEST when invalid JSON provided
> 
>
> Key: SOLR-8836
> URL: https://issues.apache.org/jira/browse/SOLR-8836
> Project: Solr
>  Issue Type: Bug
>Affects Versions: master
>Reporter: Jason Gerlowski
>Priority: Trivial
> Fix For: master
>
> Attachments: SOLR-8836.patch, SOLR-8836.patch
>
>
> When a user provides invalid JSON to the /update endpoint, the request fails 
> with a 500 (INTERNAL SERVER ERROR).  If the user looks at the response body, 
> they'll see a message indicating that the provided JSON was invalid.  
> However, going from the status code alone, the failure is indistinguishable 
> from a true server-error.  This is a bit misleading.  It would be nice if 
> Solr instead returned a 400 (BAD REQUEST) status code when it detects invalid 
> arguments.
> Reproduction Steps:
> 1.) bin/solr start -e cloud -noprompt
> 2.) curl -i -l -k -X POST -H 'Content-Type: application/json' 
> 'http://localhost:8983/solr/gettingstarted/update' --data-binary '}{'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-8836) /update should return BAD REQUEST when invalid JSON provided

[Anshum Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anshum Gupta reassigned SOLR-8836:
--

Assignee: Anshum Gupta

> /update should return BAD REQUEST when invalid JSON provided
> 
>
> Key: SOLR-8836
> URL: https://issues.apache.org/jira/browse/SOLR-8836
> Project: Solr
>  Issue Type: Bug
>Affects Versions: master
>Reporter: Jason Gerlowski
>Assignee: Anshum Gupta
>Priority: Trivial
> Fix For: master
>
> Attachments: SOLR-8836.patch, SOLR-8836.patch
>
>
> When a user provides invalid JSON to the /update endpoint, the request fails 
> with a 500 (INTERNAL SERVER ERROR).  If the user looks at the response body, 
> they'll see a message indicating that the provided JSON was invalid.  
> However, going from the status code alone, the failure is indistinguishable 
> from a true server-error.  This is a bit misleading.  It would be nice if 
> Solr instead returned a 400 (BAD REQUEST) status code when it detects invalid 
> arguments.
> Reproduction Steps:
> 1.) bin/solr start -e cloud -noprompt
> 2.) curl -i -l -k -X POST -H 'Content-Type: application/json' 
> 'http://localhost:8983/solr/gettingstarted/update' --data-binary '}{'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr in Linux Platform

[Ishan Chattopadhyaya]

Hi Abdel,
I think asking in the solr-users mailing list would be more appropriate. In
this mailing list, we discuss development related issues.
Hope your problem is solved soon,
Regards,
Ishan

On Mon, Mar 14, 2016 at 8:01 PM, Adel Mohamed Khalifa <
a.moha...@saudisoft.com> wrote:

> Hello everybody,
>
>
>
> I build a website (Java EE ) and want to search in some json files so I
> installed the solr server in an Ubuntu server and create a new core then
> indexing json files and the web searched correctly when I moved my code
> from windows to the server it stopped and cannot connect to solr server I
> try to debug using netbeans in Ubuntu it’s stopped and there is no
> exception on this statement (SolrServer server = new
> HttpSolrServer(“localhost:8983/solr/SearchCore”) ).
>
>
>
> I need for some help Please.
>
>
>
> Note :- I attached the servlet I used to search and connect to solr server.
>
>
>
>
> *Regards,**Adel Khalifa*
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

[jira] [Commented] (LUCENE-7101) OfflineSorter's merging is O(N^2) cost for large sorts

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193402#comment-15193402
 ] 

Michael McCandless commented on LUCENE-7101:


To get log(N) behavior you need to merge N segments that are "close" to the 
same size on each merge...

E.g., say you are sorting "plenty" of data, with only 16 MB heap buffer to work 
with.

Today, we will write 128 (default merge factor) 16 MB segments, and then merge 
those down to a single 2 GB (= 128 * 16 MB) segment.  Next we write another 127 
16 MB segments, and merge the 2 GB segment, plus 127 16 MB segments, into a 
3.98 GB segment.  Then another 127 16 MB segments, and merge that into a 5.97 
GB segment, etc.

So the net number of MB copied by merging will be something like:

  (128 + 0*127)*16 + (128 + 1*127)*16 + (128 + 2*127)*16 + (128 + 3*127)*16 + 
...

The equation has a linear number of terms vs your input size (input data 
divided by 1.98 GB, or so?).

But this equation asymptotes towards O(N^2):

It rewrites to:

  16 * 128 + 16 * (0*127 + 1*127 + 2*127 + 3*127 + ...)

and rewrites to:

  16 * 128 + 16 * 127 * (0 + 1 + 2 + 3 + ...)

where the number of terms in that last part is linear vs your input data size.

So this bug only happens if you sort enough data, vs. your allowed heap usage, 
to require many merges, so the too-large 128 merge factor "hides" this problem 
for most use cases.


> OfflineSorter's merging is O(N^2) cost for large sorts
> --
>
> Key: LUCENE-7101
> URL: https://issues.apache.org/jira/browse/LUCENE-7101
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Blocker
> Fix For: master, 6.0
>
>
> Our {{OfflineSorter}} acts just like Lucene, writing small initial
> segments of sorted values (from what it was able to sort at once in
> heap), periodically merging them when there are too many, and doing a
> {{forceMerge(1)}} in the end.
> But the merge logic is too simplistic today, resulting in O(N^2)
> cost.  Smallish sorts usually won't hit it, because the default 128
> merge factor is so high, but e.g. the new 2B points tests do hit the
> N^2 behavior.  I suspect the high merge factor hurts performance (OS
> struggles to use what free RAM it has to read-ahead on 128 files), and
> also risks file descriptor exhaustion.
> I think we should implement a simple log merge policy for it, and drop
> its default merge factor to 10.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr in Linux Platform

[Adel Mohamed Khalifa]

Hello everybody,

 

I build a website (Java EE ) and want to search in some json files so I 
installed the solr server in an Ubuntu server and create a new core then 
indexing json files and the web searched correctly when I moved my code from 
windows to the server it stopped and cannot connect to solr server I try to 
debug using netbeans in Ubuntu it’s stopped and there is no exception on this 
statement (SolrServer server = new 
HttpSolrServer(“localhost:8983/solr/SearchCore”) ).

 

I need for some help Please.

 

Note :- I attached the servlet I used to search and connect to solr server.

 

Regards,
Adel Khalifa

 

protected void processRequest(HttpServletRequest request, HttpServletResponse 
response)
throws ServletException, IOException {
response.setContentType("text/html;charset=UTF-8");
try {
SolrServer server = new 
HttpSolrServer("http://localhost:8983/solr/SearchCore;);

ModifiableSolrParams params = new ModifiableSolrParams();
String qqq = new 
String(request.getParameter("qu").getBytes("ISO-8859-1"), "UTF-8");
params.set("q", new 
String(request.getParameter("qu").getBytes("ISO-8859-1"), "UTF-8"));
params.set("rows", "3");


params.set("spellcheck", "on");
params.set("spellcheck.build", "true");
params.set("wt", "json");
params.set("indent", "true");
   
PrintWriter out = response.getWriter();
QueryResponse res = server.query(params);

SolrDocumentList results = res.getResults();
Iterator i = results.iterator();
request.setAttribute("size", results.size());
String json = new Gson().toJson(res.getResults());
out.write(json);
} catch (SolrServerException ex) {
System.out.println(ex.toString());
}

}

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7091) Add doc values support to MemoryIndex

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193364#comment-15193364
 ] 

David Smiley commented on LUCENE-7091:
--

Almost there...

* (still applies to the _other_ addField): I think the javadocs sentence you 
added to addField meant to use "if" not "is".
* At first I thought there might have been a bug for double-applying the boost 
since I see you're still passing "boost" as a constructor argument to Info.  
But now I see you only apply when numTokens > 0.  I think it would be much 
clearer (and simpler) to remove boost from the constructor to Info, and simply 
apply it in storeTerms (no matter what numTokens is).  It's hard to judge the 
testDocValuesDoNotAffectBoostPositionsOrOffset for this problem... it'd get 
encoded in the norm and I have no idea what norm it should be; your test 
asserts -127.  H.  Perhaps simply leave a check of that nature to the test 
that asserts parity with the real index in RAMDirectory
* in {{storeDocValues()}} SORTED_NUMERIC: you call ArrayUtil.grow with only the 
array.  This results in the same {{O(N^2)}} we're trying to avoid!  Pass in a 
second argument of the desired length.

> Add doc values support to MemoryIndex
> -
>
> Key: LUCENE-7091
> URL: https://issues.apache.org/jira/browse/LUCENE-7091
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Martijn van Groningen
> Attachments: LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch, 
> LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch
>
>
> Sometimes queries executed via the MemoryIndex require certain things to be 
> stored as doc values. Today this isn't possible because the memory index 
> doesn't support this and these queries silently return no results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8844) Date math silently ignored for date strings

[Markus Jelsma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193360#comment-15193360
 ] 

Markus Jelsma commented on SOLR-8844:
-

Crap, you're right. I stumbled upon such an issue via a unit test but this is a 
bad reproduction. I'll look into it again.

> Date math silently ignored for date strings
> ---
>
> Key: SOLR-8844
> URL: https://issues.apache.org/jira/browse/SOLR-8844
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.5
>Reporter: Markus Jelsma
>Priority: Minor
> Fix For: 6.1
>
>
> Consider the following query, ordered by date ascending: {code}
> http://localhost:8983/solr/logs/select?q=*:*=[2011-05-26T08:15:36Z%2B3DAY%20TO%20NOW/DAY]=time%20asc
> {code}
> Should not have a result set where the first entry entry has 
> 2011-05-26T08:15:36Z for the time field.
> It appears date math is just ignored, while i would expect it to work or 
> throw an error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7101) OfflineSorter's merging is O(N^2) cost for large sorts

[Dawid Weiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193359#comment-15193359
 ] 

Dawid Weiss commented on LUCENE-7101:
-

Sorry for being dim, what's the scenario of hitting N^2 complexity here?

> OfflineSorter's merging is O(N^2) cost for large sorts
> --
>
> Key: LUCENE-7101
> URL: https://issues.apache.org/jira/browse/LUCENE-7101
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Blocker
> Fix For: master, 6.0
>
>
> Our {{OfflineSorter}} acts just like Lucene, writing small initial
> segments of sorted values (from what it was able to sort at once in
> heap), periodically merging them when there are too many, and doing a
> {{forceMerge(1)}} in the end.
> But the merge logic is too simplistic today, resulting in O(N^2)
> cost.  Smallish sorts usually won't hit it, because the default 128
> merge factor is so high, but e.g. the new 2B points tests do hit the
> N^2 behavior.  I suspect the high merge factor hurts performance (OS
> struggles to use what free RAM it has to read-ahead on 128 files), and
> also risks file descriptor exhaustion.
> I think we should implement a simple log merge policy for it, and drop
> its default merge factor to 10.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-master - Build # 437 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/437/

No tests ran.

Build Log:
[...truncated 14 lines...]
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
git://git.apache.org/lucene-solr.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true 
fetch --tags --progress git://git.apache.org/lucene-solr.git 
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: read error: Connection reset by peer

at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1441)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:62)
at 
org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:313)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
at hudson.remoting.UserRequest.perform(UserRequest.java:120)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:326)
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ..remote call to lucene(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
at hudson.remoting.Channel.call(Channel.java:781)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145)
at sun.reflect.GeneratedMethodAccessor407.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
at com.sun.proxy.$Proxy64.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:764)
... 11 more
ERROR: null
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git://git.apache.org/lucene-solr.git # 
 > timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Fetching upstream changes from git://git.apache.org/lucene-solr.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > git://git.apache.org/lucene-solr.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from 
git://git.apache.org/lucene-solr.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
  

Re: [JENKINS] Lucene-Solr-NightlyTests-master - Build # 960 - Still Failing

[Michael McCandless]

I'll fix.

Mike McCandless

http://blog.mikemccandless.com


On Mon, Mar 14, 2016 at 10:13 AM, Apache Jenkins Server
 wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/960/
>
> 1 tests failed.
> FAILED:  org.apache.lucene.search.spans.TestSpanCollection.testSpanNotQuery
>
> Error Message:
> StandardDirectoryReader(segments_1:6 _0(7.0.0):C3 _1(7.0.0):c1) has 2 
> segments instead of exactly one
>
> Stack Trace:
> java.lang.IllegalArgumentException: StandardDirectoryReader(segments_1:6 
> _0(7.0.0):C3 _1(7.0.0):c1) has 2 segments instead of exactly one
> at 
> __randomizedtesting.SeedInfo.seed([D0A81CBB92B6824A:ABBE7B2F085F75C9]:0)
> at 
> org.apache.lucene.util.LuceneTestCase.getOnlyLeafReader(LuceneTestCase.java:786)
> at 
> org.apache.lucene.search.spans.TestSpanCollection.setUp(TestSpanCollection.java:73)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:905)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
> at 
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at 
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
> at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
> at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
> at 
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
> at 
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at 
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at 
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at 
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at 
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at 
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 1545 lines...]
>[junit4] Suite: org.apache.lucene.search.spans.TestSpanCollection
>[junit4]   2> NOTE: download the large Jenkins line-docs file by running 
> 'ant get-jenkins-line-docs' in the lucene directory.
>[junit4]   2> NOTE: reproduce with: ant test  
> -Dtestcase=TestSpanCollection -Dtests.method=testSpanNotQuery 
> -Dtests.seed=D0A81CBB92B6824A -Dtests.multiplier=2 -Dtests.nightly=true 

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 960 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/960/

1 tests failed.
FAILED:  org.apache.lucene.search.spans.TestSpanCollection.testSpanNotQuery

Error Message:
StandardDirectoryReader(segments_1:6 _0(7.0.0):C3 _1(7.0.0):c1) has 2 segments 
instead of exactly one

Stack Trace:
java.lang.IllegalArgumentException: StandardDirectoryReader(segments_1:6 
_0(7.0.0):C3 _1(7.0.0):c1) has 2 segments instead of exactly one
at 
__randomizedtesting.SeedInfo.seed([D0A81CBB92B6824A:ABBE7B2F085F75C9]:0)
at 
org.apache.lucene.util.LuceneTestCase.getOnlyLeafReader(LuceneTestCase.java:786)
at 
org.apache.lucene.search.spans.TestSpanCollection.setUp(TestSpanCollection.java:73)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:905)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 1545 lines...]
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanCollection
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSpanCollection 
-Dtests.method=testSpanNotQuery -Dtests.seed=D0A81CBB92B6824A 
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt 
-Dtests.locale=ar-KW -Dtests.timezone=Australia/Brisbane -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.01s J1 | TestSpanCollection.testSpanNotQuery <<<
   [junit4]> Throwable #1: 

[jira] [Commented] (SOLR-8844) Date math silently ignored for date strings

[Jack Krupansky (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193332#comment-15193332
 ] 

Jack Krupansky commented on SOLR-8844:
--

1. No field is specified for the fq parameter here. What is df?
2. Do any matching date/time values occur as literal strings in the default 
search field?


> Date math silently ignored for date strings
> ---
>
> Key: SOLR-8844
> URL: https://issues.apache.org/jira/browse/SOLR-8844
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.5
>Reporter: Markus Jelsma
>Priority: Minor
> Fix For: 6.1
>
>
> Consider the following query, ordered by date ascending: {code}
> http://localhost:8983/solr/logs/select?q=*:*=[2011-05-26T08:15:36Z%2B3DAY%20TO%20NOW/DAY]=time%20asc
> {code}
> Should not have a result set where the first entry entry has 
> 2011-05-26T08:15:36Z for the time field.
> It appears date math is just ignored, while i would expect it to work or 
> throw an error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-7101) OfflineSorter's merging is O(N^2) cost for large sorts

[Michael McCandless (JIRA)]

Michael McCandless created LUCENE-7101:
--

 Summary: OfflineSorter's merging is O(N^2) cost for large sorts
 Key: LUCENE-7101
 URL: https://issues.apache.org/jira/browse/LUCENE-7101
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Blocker
 Fix For: master, 6.0


Our {{OfflineSorter}} acts just like Lucene, writing small initial
segments of sorted values (from what it was able to sort at once in
heap), periodically merging them when there are too many, and doing a
{{forceMerge(1)}} in the end.

But the merge logic is too simplistic today, resulting in O(N^2)
cost.  Smallish sorts usually won't hit it, because the default 128
merge factor is so high, but e.g. the new 2B points tests do hit the
N^2 behavior.  I suspect the high merge factor hurts performance (OS
struggles to use what free RAM it has to read-ahead on 128 files), and
also risks file descriptor exhaustion.

I think we should implement a simple log merge policy for it, and drop
its default merge factor to 10.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8844) Date math silently ignored for date strings

[Markus Jelsma (JIRA)]

Markus Jelsma created SOLR-8844:
---

 Summary: Date math silently ignored for date strings
 Key: SOLR-8844
 URL: https://issues.apache.org/jira/browse/SOLR-8844
 Project: Solr
  Issue Type: Bug
Affects Versions: 5.5
Reporter: Markus Jelsma
Priority: Minor
 Fix For: 6.1


Consider the following query, ordered by date ascending: {code}
http://localhost:8983/solr/logs/select?q=*:*=[2011-05-26T08:15:36Z%2B3DAY%20TO%20NOW/DAY]=time%20asc
{code}

Should not have a result set where the first entry entry has 
2011-05-26T08:15:36Z for the time field.

It appears date math is just ignored, while i would expect it to work or throw 
an error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8843) Missing fallback for NoChildrenForEphemeralsException on ZkController.getLeaderPropsWithFallback for rolling upgrade

[Enrico Hartung (JIRA)]

Enrico Hartung created SOLR-8843:


 Summary: Missing fallback for NoChildrenForEphemeralsException on 
ZkController.getLeaderPropsWithFallback for rolling upgrade
 Key: SOLR-8843
 URL: https://issues.apache.org/jira/browse/SOLR-8843
 Project: Solr
  Issue Type: Bug
  Components: SolrCloud
Affects Versions: 5.4.1, 5.5
Reporter: Enrico Hartung


When doing a rolling upgrade from 5.3.2 to 5.4.1 (or 5.5.0) leader election 
fails with the following error (NoChildrenForEphemeralsException):
{code}
ERROR org.apache.solr.cloud.ShardLeaderElectionContext  [c:collection s:shard1 
r:core_node1 x:collection_shard1_replica1] – There was a problem trying to 
register as the leader:org.apache.solr.common.SolrException: Could not register 
as the leader because creating the ephemeral registration node in ZooKeeper 
failed
#011at 
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:214)
#011at 
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:406)
#011at 
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
#011at 
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:158)
#011at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:59)
#011at 
org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:389)
#011at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:264)
#011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
#011at java.util.concurrent.FutureTask.run(FutureTask.java:266)
#011at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
#011at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
#011at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
#011at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.zookeeper.KeeperException$NoChildrenForEphemeralsException: 
KeeperErrorCode = NoChildrenForEphemerals
#011at org.apache.zookeeper.KeeperException.create(KeeperException.java:117)
#011at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:949)
#011at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
#011at 
org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:570)
#011at 
org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:567)
#011at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
#011at org.apache.solr.common.cloud.SolrZkClient.multi(SolrZkClient.java:567)
#011at 
org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:197)
#011at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:50)
#011at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:43)
#011at 
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:179)
#011... 12 more
{code}

A similar issues has been resolved with SOLR-8561, but it is not handling the 
case *NoChildrenForEphemeralsException*.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8842) security should use an API to expose the permission name instead of using HTTP params

[Noble Paul (JIRA)]

Noble Paul created SOLR-8842:


 Summary: security should use an API to expose the permission name 
instead of using HTTP params
 Key: SOLR-8842
 URL: https://issues.apache.org/jira/browse/SOLR-8842
 Project: Solr
  Issue Type: Improvement
Reporter: Noble Paul
Assignee: Noble Paul


Currently the well-known permissions are using the HTTP atributes, such as 
method, uri, params etc to identify the corresponding permission name such as 
'read', 'update' etc. Expose this value through an API so that it can be more 
accurate and handle various versions of the API

RequestHandlers will be able to implement an interface to provide the name
{code}
interface PermissionNameProvider {
String getPermissionName(SolrQueryRequest req)
}
{code} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-3475) Performance hit due to unnecessary syncronization

[Steve Shabino (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-3475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193227#comment-15193227
 ] 

Steve Shabino commented on SOLR-3475:
-

I believe the synchronization exists so that multiple, concurrent requests for 
the same cache key do not cause redundant construction of the (likely 
expensive) cached value. This ought to be a feature of the cache, as done by 
Guava or Caffeine via their LoadingCache interfaces.

> Performance hit due to unnecessary syncronization
> -
>
> Key: SOLR-3475
> URL: https://issues.apache.org/jira/browse/SOLR-3475
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 4.1
>Reporter: Varun Srivastava
>
> Method org.apache.solr.request.UnInvertedField.getUnInvertedField unnecessary 
> syncronize agiant cache object. The SolrCache implementations should them 
> selves be thread safe. For example FastLRUCache uses ConcurrentLRUCache which 
> in turn use java.util.concurrency structs to make the implementation 
> threadsafe.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8029) Modernize and standardize Solr APIs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193221#comment-15193221
 ] 

ASF subversion and git services commented on SOLR-8029:
---

Commit 6c768cebca2f2ca6baa3aa7546d63ad4fc6fe37f in lucene-solr's branch 
refs/heads/apiv2 from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6c768ce ]

SOLR-8029: test should randomly run on v2


> Modernize and standardize Solr APIs
> ---
>
> Key: SOLR-8029
> URL: https://issues.apache.org/jira/browse/SOLR-8029
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: master
>Reporter: Noble Paul
>Assignee: Noble Paul
>  Labels: API, EaseOfUse
> Fix For: master
>
> Attachments: SOLR-8029.patch, SOLR-8029.patch, SOLR-8029.patch, 
> SOLR-8029.patch
>
>
> Solr APIs have organically evolved and they are sometimes inconsistent with 
> each other or not in sync with the widely followed conventions of HTTP 
> protocol. Trying to make incremental changes to make them modern is like 
> applying band-aid. So, we have done a complete rethink of what the APIs 
> should be. The most notable aspects of the API are as follows:
> The new set of APIs will be placed under a new path {{/solr2}}. The legacy 
> APIs will continue to work under the {{/solr}} path as they used to and they 
> will be eventually deprecated.
> There are 4 types of requests in the new API 
> * {{/v2//*}} : Hit a collection directly or manage 
> collections/shards/replicas 
> * {{/v2//*}} : Hit a core directly or manage cores 
> * {{/v2/cluster/*}} : Operations on cluster not pertaining to any collection 
> or core. e.g: security, overseer ops etc
> This will be released as part of a major release. Check the link given below 
> for the full specification.  Your comments are welcome
> [Solr API version 2 Specification | http://bit.ly/1JYsBMQ]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8029) Modernize and standardize Solr APIs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193218#comment-15193218
 ] 

ASF subversion and git services commented on SOLR-8029:
---

Commit e802a4d61ed26ce43fc5bdffa3ab5391c9724dc6 in lucene-solr's branch 
refs/heads/apiv2 from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e802a4d ]

SOLR-8029: Add V2 paths for security permissions and refactored and documented 
security APIs


> Modernize and standardize Solr APIs
> ---
>
> Key: SOLR-8029
> URL: https://issues.apache.org/jira/browse/SOLR-8029
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: master
>Reporter: Noble Paul
>Assignee: Noble Paul
>  Labels: API, EaseOfUse
> Fix For: master
>
> Attachments: SOLR-8029.patch, SOLR-8029.patch, SOLR-8029.patch, 
> SOLR-8029.patch
>
>
> Solr APIs have organically evolved and they are sometimes inconsistent with 
> each other or not in sync with the widely followed conventions of HTTP 
> protocol. Trying to make incremental changes to make them modern is like 
> applying band-aid. So, we have done a complete rethink of what the APIs 
> should be. The most notable aspects of the API are as follows:
> The new set of APIs will be placed under a new path {{/solr2}}. The legacy 
> APIs will continue to work under the {{/solr}} path as they used to and they 
> will be eventually deprecated.
> There are 4 types of requests in the new API 
> * {{/v2//*}} : Hit a collection directly or manage 
> collections/shards/replicas 
> * {{/v2//*}} : Hit a core directly or manage cores 
> * {{/v2/cluster/*}} : Operations on cluster not pertaining to any collection 
> or core. e.g: security, overseer ops etc
> This will be released as part of a major release. Check the link given below 
> for the full specification.  Your comments are welcome
> [Solr API version 2 Specification | http://bit.ly/1JYsBMQ]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-8210) Admin UI menu does not scroll

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira resolved SOLR-8210.
-
   Resolution: Fixed
Fix Version/s: 5.4

This was resolved for 5.4

> Admin UI menu does not scroll
> -
>
> Key: SOLR-8210
> URL: https://issues.apache.org/jira/browse/SOLR-8210
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 5.3
>Reporter: Upayavira
>Assignee: Upayavira
>Priority: Minor
> Fix For: 5.4
>
>
> When you view the UI on a projector - or a small screen (e.g. with dev tools 
> open), some menu options might be obscured at the bottom of the screen. The 
> menu doesn't scroll though, meaning the only way to get to these entries is 
> to use another screen, or change the text size in the browser temporarily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-8210) Admin UI menu does not scroll

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira closed SOLR-8210.
---

> Admin UI menu does not scroll
> -
>
> Key: SOLR-8210
> URL: https://issues.apache.org/jira/browse/SOLR-8210
> Project: Solr
>  Issue Type: Improvement
>  Components: web gui
>Affects Versions: 5.3
>Reporter: Upayavira
>Assignee: Upayavira
>Priority: Minor
> Fix For: 5.4
>
>
> When you view the UI on a projector - or a small screen (e.g. with dev tools 
> open), some menu options might be obscured at the bottom of the screen. The 
> menu doesn't scroll though, meaning the only way to get to these entries is 
> to use another screen, or change the text size in the browser temporarily.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-8199) Text specifying which UI a user is looking at is incorrect

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira resolved SOLR-8199.
-
Resolution: Fixed

This was resolved before release

> Text specifying which UI a user is looking at is incorrect
> --
>
> Key: SOLR-8199
> URL: https://issues.apache.org/jira/browse/SOLR-8199
> Project: Solr
>  Issue Type: Bug
>  Components: UI
>Reporter: Youssef Chaker
>Assignee: Upayavira
>Priority: Trivial
> Attachments: Screen_Shot_2015-10-24_at_10_21_08_AM.png, 
> Screen_Shot_2015-10-24_at_10_21_41_AM.png
>
>
> In the top right corner of the admin UI, a text is available to indicate 
> whether the user is looking at the original UI or the new one.
> But it currently says "New UI" for http://localhost:8983/solr/#/ and 
> "Original UI" for http://localhost:8983/solr/index.html#/ when it should be 
> the other way around.
> This issue is tied to #SOLR-7666



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-7857) Update CHANGES.txt to mention angular UI changes

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira closed SOLR-7857.
---

> Update CHANGES.txt to mention angular UI changes
> 
>
> Key: SOLR-7857
> URL: https://issues.apache.org/jira/browse/SOLR-7857
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Reporter: Upayavira
>Assignee: Upayavira
>Priority: Minor
> Fix For: 5.5, master
>
>
> Many tweaks to AngularUI made since 5.2.1 - CHANGES.txt should describe the 
> nature of those changes and set expectations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-7666) Umbrella ticket for Angular JS post-5.2.1 issues

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira closed SOLR-7666.
---

> Umbrella ticket for Angular JS post-5.2.1 issues
> 
>
> Key: SOLR-7666
> URL: https://issues.apache.org/jira/browse/SOLR-7666
> Project: Solr
>  Issue Type: New Feature
>  Components: UI
>Affects Versions: 5.3, master
>Reporter: Erick Erickson
>Assignee: Upayavira
> Attachments: SOLR-7666-3.patch, SOLR-7666-4.patch, SOLR-7666-5.patch, 
> SOLR-7666-part2.patch, SOLR-7666-part2.patch, SOLR-7666.patch, 
> admin-ui-7666.zip
>
>
> As of Solr 5.2.1, there's a new admin UI available that has been written 
> almost entirely by Upayavira (thanks!) over the last several months. It's 
> written in Angular JS with an eye towards enhancement/maintainability. The 
> default UI is still the old version, but you can access the new version by 
> going to http:///solr/index.html. There are a couple of fixes 
> between 5.2.0 and 5.2.1, so please use either a fresh 5x checkout, trunk, or 
> 5.2.1
> The expectation is that in Solr 5.3, the new code will become the default 
> with the old UI deprecated and eventually removed.
> So it would be a great help if volunteers could give the new UI a spin. It 
> should look much the same as the current one at the start, but evolve into 
> something much more interesting and more cloud-friendly. Of course all the 
> new UI code will always be available on trunk/6.0 too, and the up-to-date 
> code will always be on both the trunk and 5x branches.
> Please provide feedback on the user's (or dev) lists about anything you find 
> that doesn't work, or enhancements you'd like to see (or, even better, 
> contribute). If you raise a JIRA, please link it to this one so I can keep 
> track of what needs to be committed. If linking JIRAs is a mystery just add a 
> comment to this JIRA referencing the new JIRA and we can take care of it.
> Please do _not_ attach patches to this JIRA, it'll be much easier to keep 
> track of everything if the patches are attached to sub-JIRAs.
> And a big thanks to Upayavira for this work!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-7857) Update CHANGES.txt to mention angular UI changes

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira resolved SOLR-7857.
-
Resolution: Fixed

Either this was done, or it is now irrelevant. Either way, it is resolved

> Update CHANGES.txt to mention angular UI changes
> 
>
> Key: SOLR-7857
> URL: https://issues.apache.org/jira/browse/SOLR-7857
> Project: Solr
>  Issue Type: Bug
>  Components: web gui
>Reporter: Upayavira
>Assignee: Upayavira
>Priority: Minor
> Fix For: master, 5.5
>
>
> Many tweaks to AngularUI made since 5.2.1 - CHANGES.txt should describe the 
> nature of those changes and set expectations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-7666) Umbrella ticket for Angular JS post-5.2.1 issues

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira resolved SOLR-7666.
-
Resolution: Fixed

As Erick says, this ticket is rather long, any new issues should be in new 
tickets.

> Umbrella ticket for Angular JS post-5.2.1 issues
> 
>
> Key: SOLR-7666
> URL: https://issues.apache.org/jira/browse/SOLR-7666
> Project: Solr
>  Issue Type: New Feature
>  Components: UI
>Affects Versions: 5.3, master
>Reporter: Erick Erickson
>Assignee: Upayavira
> Attachments: SOLR-7666-3.patch, SOLR-7666-4.patch, SOLR-7666-5.patch, 
> SOLR-7666-part2.patch, SOLR-7666-part2.patch, SOLR-7666.patch, 
> admin-ui-7666.zip
>
>
> As of Solr 5.2.1, there's a new admin UI available that has been written 
> almost entirely by Upayavira (thanks!) over the last several months. It's 
> written in Angular JS with an eye towards enhancement/maintainability. The 
> default UI is still the old version, but you can access the new version by 
> going to http:///solr/index.html. There are a couple of fixes 
> between 5.2.0 and 5.2.1, so please use either a fresh 5x checkout, trunk, or 
> 5.2.1
> The expectation is that in Solr 5.3, the new code will become the default 
> with the old UI deprecated and eventually removed.
> So it would be a great help if volunteers could give the new UI a spin. It 
> should look much the same as the current one at the start, but evolve into 
> something much more interesting and more cloud-friendly. Of course all the 
> new UI code will always be available on trunk/6.0 too, and the up-to-date 
> code will always be on both the trunk and 5x branches.
> Please provide feedback on the user's (or dev) lists about anything you find 
> that doesn't work, or enhancements you'd like to see (or, even better, 
> contribute). If you raise a JIRA, please link it to this one so I can keep 
> track of what needs to be committed. If linking JIRAs is a mystery just add a 
> comment to this JIRA referencing the new JIRA and we can take care of it.
> Please do _not_ attach patches to this JIRA, it'll be much easier to keep 
> track of everything if the patches are attached to sub-JIRAs.
> And a big thanks to Upayavira for this work!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7093) MemoryIndex does not support points

[Martijn van Groningen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martijn van Groningen updated LUCENE-7093:
--
Attachment: LUCENE-7093.patch

Synched patch with LUCENE-7091 and removed the an added constructor to 
BytesRefArray (after realising that BytesRefArray can only work with its own 
privately constructed byte pool) .

> MemoryIndex does not support points
> ---
>
> Key: LUCENE-7093
> URL: https://issues.apache.org/jira/browse/LUCENE-7093
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Martijn van Groningen
> Attachments: LUCENE-7093.patch, LUCENE-7093.patch
>
>
> I realized this glancing at LUCENE-7091.
> I think this should have points support or else people cannot move off of the 
> deprecated LegacyXXX encodings?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8841) edismax: minimum match and compound words

[Christian Winkler (JIRA)]

Christian Winkler created SOLR-8841:
---

 Summary: edismax: minimum match and compound words
 Key: SOLR-8841
 URL: https://issues.apache.org/jira/browse/SOLR-8841
 Project: Solr
  Issue Type: Bug
  Components: search
Affects Versions: 5.5, trunk
 Environment: all
Reporter: Christian Winkler


Hi,

when searching for a single word which is split by a compound word splitter 
(very common in German), minimum match is not handled correctly. It is always 
set to 1 (only a single search term), but as the word is split into several 
single parts, one matching part is enough

This also happens if mm is set to 100%.

Probably mm should be set after the split has been performed. Similar problems 
might arise with synonymization at search time.

Regards
Christian 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6336) AnalyzingInfixSuggester needs duplicate handling

[JIRA]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated LUCENE-6336:

Labels: lookup suggester  (was: )

> AnalyzingInfixSuggester needs duplicate handling
> 
>
> Key: LUCENE-6336
> URL: https://issues.apache.org/jira/browse/LUCENE-6336
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 4.10.3, 5.0
>Reporter: Jan Høydahl
>  Labels: lookup, suggester
> Attachments: LUCENE-6336.patch
>
>
> Spinoff from LUCENE-5833 but else unrelated.
> Using {{AnalyzingInfixSuggester}} which is backed by a Lucene index and 
> stores payload and score together with the suggest text.
> I did some testing with Solr, producing the DocumentDictionary from an index 
> with multiple documents containing the same text, but with random weights 
> between 0-100. Then I got duplicate identical suggestions sorted by weight:
> {code}
> {
>   "suggest":{"languages":{
>   "engl":{
> "numFound":101,
> "suggestions":[{
> "term":"English",
> "weight":100,
> "payload":"0"},
>   {
> "term":"English",
> "weight":99,
> "payload":"0"},
>   {
> "term":"English",
> "weight":98,
> "payload":"0"},
> ---etc all the way down to 0---
> {code}
> I also reproduced the same behavior in AnalyzingInfixSuggester directly. So 
> there is a need for some duplicate removal here, either while building the 
> local suggest index or during lookup. Only the highest weight suggestion for 
> a given term should be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6336) AnalyzingInfixSuggester needs duplicate handling

[JIRA]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated LUCENE-6336:

Fix Version/s: (was: 5.2)
   (was: master)

> AnalyzingInfixSuggester needs duplicate handling
> 
>
> Key: LUCENE-6336
> URL: https://issues.apache.org/jira/browse/LUCENE-6336
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 4.10.3, 5.0
>Reporter: Jan Høydahl
> Attachments: LUCENE-6336.patch
>
>
> Spinoff from LUCENE-5833 but else unrelated.
> Using {{AnalyzingInfixSuggester}} which is backed by a Lucene index and 
> stores payload and score together with the suggest text.
> I did some testing with Solr, producing the DocumentDictionary from an index 
> with multiple documents containing the same text, but with random weights 
> between 0-100. Then I got duplicate identical suggestions sorted by weight:
> {code}
> {
>   "suggest":{"languages":{
>   "engl":{
> "numFound":101,
> "suggestions":[{
> "term":"English",
> "weight":100,
> "payload":"0"},
>   {
> "term":"English",
> "weight":99,
> "payload":"0"},
>   {
> "term":"English",
> "weight":98,
> "payload":"0"},
> ---etc all the way down to 0---
> {code}
> I also reproduced the same behavior in AnalyzingInfixSuggester directly. So 
> there is a need for some duplicate removal here, either while building the 
> local suggest index or during lookup. Only the highest weight suggestion for 
> a given term should be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7091) Add doc values support to MemoryIndex

[Martijn van Groningen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martijn van Groningen updated LUCENE-7091:
--
Attachment: LUCENE-7091.patch

Updated the patch with the good points that you've raised.

bq. It's a shame that SORTED & BINARY use a BytesRefHash (adds overhead) and 
ultimately get sorted when, really, it's not necessary of course. The 
ByteBlockPool could be used directly to store it (see BytesRefArray for 
examples) with a little bit of code. This isn't a blocker but it would sure be 
nice.

Agreed, that would be nicer. I think we should do this in a follow up issue.

bq. Add term text here too, and under same field names as DV ones at that.

I think this is covered in 
TestMemoryIndexAgainstRAMDir#testDocValuesMemoryIndexVsNormalIndex() test, in 
this test regular fields are randomilly added.

> Add doc values support to MemoryIndex
> -
>
> Key: LUCENE-7091
> URL: https://issues.apache.org/jira/browse/LUCENE-7091
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Martijn van Groningen
> Attachments: LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch, 
> LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch, LUCENE-7091.patch
>
>
> Sometimes queries executed via the MemoryIndex require certain things to be 
> stored as doc values. Today this isn't possible because the memory index 
> doesn't support this and these queries silently return no results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6336) AnalyzingInfixSuggester needs duplicate handling

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193150#comment-15193150
 ] 

Alessandro Benedetti commented on LUCENE-6336:
--

Initially I liked the idea of adding a component responsible of the 
de-duplication .
But I would like to raise some considerations, what about the number of the 
suggestions ?

At the moment the number of suggestions upbound the search in the auxiliary 
lucene index 
( see this 
org/apache/lucene/search/suggest/analyzing/AnalyzingInfixSuggester.java:591 .)

This means that retrieving a max of 5 suggestions could bring the return of 5 
duplicates ( leaving other values in the remaining results) .
Then the dedupe wrapper will dedupe  and return only 1 suggestion ( we forget 
about other 4 good suggestions that were low in the ranking)
We potentially risk to not cover the top N we wants in the configuration.

I was thinking we should solve this Lucene side, building a better query using 
field collapsing.
In particular I think we should add a couple of parameters ( unique=true and 
weightCalculus =max|min|avg ect ) and play with something similar to : 
https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results .

What do you think [~janhoy], [~mikemccand]? I think with field collapsing we 
could be more consistent.
I will study this more, please inform me if my reasoning lacks of some 
important assumption :)




> AnalyzingInfixSuggester needs duplicate handling
> 
>
> Key: LUCENE-6336
> URL: https://issues.apache.org/jira/browse/LUCENE-6336
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 4.10.3, 5.0
>Reporter: Jan Høydahl
> Fix For: 5.2, master
>
> Attachments: LUCENE-6336.patch
>
>
> Spinoff from LUCENE-5833 but else unrelated.
> Using {{AnalyzingInfixSuggester}} which is backed by a Lucene index and 
> stores payload and score together with the suggest text.
> I did some testing with Solr, producing the DocumentDictionary from an index 
> with multiple documents containing the same text, but with random weights 
> between 0-100. Then I got duplicate identical suggestions sorted by weight:
> {code}
> {
>   "suggest":{"languages":{
>   "engl":{
> "numFound":101,
> "suggestions":[{
> "term":"English",
> "weight":100,
> "payload":"0"},
>   {
> "term":"English",
> "weight":99,
> "payload":"0"},
>   {
> "term":"English",
> "weight":98,
> "payload":"0"},
> ---etc all the way down to 0---
> {code}
> I also reproduced the same behavior in AnalyzingInfixSuggester directly. So 
> there is a need for some duplicate removal here, either while building the 
> local suggest index or during lookup. Only the highest weight suggestion for 
> a given term should be returned.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8029) Modernize and standardize Solr APIs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193151#comment-15193151
 ] 

ASF subversion and git services commented on SOLR-8029:
---

Commit bbb81e0bb6881b8233a99650afe59a270334b9fc in lucene-solr's branch 
refs/heads/apiv2 from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bbb81e0 ]

SOLR-8029: FIxed failing tests and added a testcase


> Modernize and standardize Solr APIs
> ---
>
> Key: SOLR-8029
> URL: https://issues.apache.org/jira/browse/SOLR-8029
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: master
>Reporter: Noble Paul
>Assignee: Noble Paul
>  Labels: API, EaseOfUse
> Fix For: master
>
> Attachments: SOLR-8029.patch, SOLR-8029.patch, SOLR-8029.patch, 
> SOLR-8029.patch
>
>
> Solr APIs have organically evolved and they are sometimes inconsistent with 
> each other or not in sync with the widely followed conventions of HTTP 
> protocol. Trying to make incremental changes to make them modern is like 
> applying band-aid. So, we have done a complete rethink of what the APIs 
> should be. The most notable aspects of the API are as follows:
> The new set of APIs will be placed under a new path {{/solr2}}. The legacy 
> APIs will continue to work under the {{/solr}} path as they used to and they 
> will be eventually deprecated.
> There are 4 types of requests in the new API 
> * {{/v2//*}} : Hit a collection directly or manage 
> collections/shards/replicas 
> * {{/v2//*}} : Hit a core directly or manage cores 
> * {{/v2/cluster/*}} : Operations on cluster not pertaining to any collection 
> or core. e.g: security, overseer ops etc
> This will be released as part of a major release. Check the link given below 
> for the full specification.  Your comments are welcome
> [Solr API version 2 Specification | http://bit.ly/1JYsBMQ]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-8839) Angular admin/segments display: display of deleted docs not proportional

[Upayavira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Upayavira reassigned SOLR-8839:
---

Assignee: Upayavira

> Angular admin/segments display: display of deleted docs not proportional
> 
>
> Key: SOLR-8839
> URL: https://issues.apache.org/jira/browse/SOLR-8839
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.4.1
>Reporter: Luc Vanlerberghe
>Assignee: Upayavira
>Priority: Minor
>
> In the /segments portion of the admin site, the segments are displayed as a 
> bar graph with the size of each bar proportional to the logarithm of the 
> segment size.
> Within each bar the number of deleted documents is shown as a dark-gray 
> portion at the end.
> Before the angular version, the size of this part was directly proportional 
> to the number of deleted documents with respect to the total number of 
> documents in the segment
> In the angular version, the dark-gray portion is way too large.
> In the previous version, the result was odd as well since it displayed a 
> proportional percentage within in a logarithmic graph.
> I'll add a PR shortly that changes the calculation so the dark-gray part 
> looks approximately proportional to the size the segment would shrink if 
> optimized.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8672) Unique Suggestions getter in Solrj

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192984#comment-15192984
 ] 

Alessandro Benedetti edited comment on SOLR-8672 at 3/14/16 11:09 AM:
--

Hi David,
I agree this was a simple contribution to try to have a quick fix to the 
problem at least at SolrJ level.
I thought more about the problem, and I agree with Tomas.
I even think we should NOT include my own patch here, this is my reason :

if we add now this SolrJ support and in the future we add the suggester 
solrconfig.xml parameter, users are going to be confused.
The solrconfig.xml approach will be the most efficient and recommended but the 
presence of this method SolrJ side could grow confusion in people minds and 
generate some scenario where you use the Solrj method but actually going to the 
solr configuration would be optimal.
Does this make sense ?

I will proceed in studying a little bit LUCENE-6336 and some internals, I will 
end up with a proposal here and in the mailing list ( with related Jira) .

Not sure if a Dedup Wrapper is the best way, ideally I don't want to generate 
at all duplicates, avoid  a wrapper reading the list again and removing 
duplicates.
I will spend some time on that!

Cheers


was (Author: alessandro.benedetti):
Hi David,
I agree this was a simple contribution to try to have a quick fix to the 
problem at least at SolrJ level.
I thought more about the problem, and I agree with Tomas.
I even think we should NOT include my own patch here, this is my reason :

if we add now this SolrJ support and in the future we add the suggester 
solrconfig.xml parameter, users are going to be confused.
The solrconfig.xml approach will be the most efficient and recommended but the 
presence of this method SolrJ side could grow confusion in people minds and 
generate some scenario where you use the Solrj method but actually going to the 
solr configuration would be optimal.
Does this make sense ?

I will proceed in studying a little bit LUCENE-6636 and some internals, I will 
end up with a proposal here and in the mailing list ( with related Jira) .

Not sure if a Dedup Wrapper is the best way, ideally I don't want to generate 
at all duplicates, avoid  a wrapper reading the list again and removing 
duplicates.
I will spend some time on that!

Cheers

> Unique Suggestions getter in Solrj
> --
>
> Key: SOLR-8672
> URL: https://issues.apache.org/jira/browse/SOLR-8672
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.4.1
>Reporter: Alessandro Benedetti
> Attachments: SOLR-8672.patch
>
>
> Currently all the suggesters based on full field content retrieval gives back 
> possibly duplicated suggestions.
> First observation , related the suggester domain, is that is unlikely we need 
> to return duplicates ( actually we don't return the id of the document, or 
> anything else, so having duplicates is arguably not a benefit) .
> I propose at least to offer via SolrJ  the possibility of getting the 
> suggestions without duplicates.
> Any feedback is welcome.
> The patch provided is really simple.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Description: 
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning 
this component on, i.e. {{olap=\{!ex=tag_name\}true}}.

Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.

  was:
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning 
this component on, i.e. _olap={!ex=tag_name}true_.

Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.


> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). 
> The Analytics Component was added in SOLR-5302.
> I tried to add this feature in simplified way, where the tags are specified 
> in param value turning 
> this component on, i.e. {{olap=\{!ex=tag_name\}true}}.
> Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Description: 
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning 
this component on, i.e. olap={!ex=tag_name}true.

Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.

  was:
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning this component on, i.e. olap={!ex=tag_name}true.
I was inspired by changes made in SOLR-3177 and I created a patch base on 
_branch_5_4_.


> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). 
> The Analytics Component was added in SOLR-5302.
> I tried to add this feature in simplified way, where the tags are specified 
> in param value turning 
> this component on, i.e. olap={!ex=tag_name}true.
> Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Description: 
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning 
this component on, i.e. _olap={!ex=tag_name}true_.

Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.

  was:
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning 
this component on, i.e. olap={!ex=tag_name}true.

Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.


> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). 
> The Analytics Component was added in SOLR-5302.
> I tried to add this feature in simplified way, where the tags are specified 
> in param value turning 
> this component on, i.e. _olap={!ex=tag_name}true_.
> Changes were inspired by SOLR-3177 and patch created is based on _branch_5_4_.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Description: 
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning this component on, i.e. olap={!ex=tag_name}true.
I was inspired by changes made in SOLR-3177 and I created a patch base on 
_branch_5_4_.

  was:
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning this component on, i.e. olap={!ex=tag_name}true. 
I was inspired by changes made in SOLR-3177 and I created a patch base on 
_branch_5_4_.


> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). 
> The Analytics Component was added in SOLR-5302.
> I tried to add this feature in simplified way, where the tags are specified 
> in param value turning this component on, i.e. olap={!ex=tag_name}true.
> I was inspired by changes made in SOLR-3177 and I created a patch base on 
> _branch_5_4_.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Description: 
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). 
The Analytics Component was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning this component on, i.e. olap={!ex=tag_name}true. 
I was inspired by changes made in SOLR-3177 and I created a patch base on 
_branch_5_4_.

  was:
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). The Analytics Component 
was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning this component on, i.e. _olap={!ex=tag_name}true_. I was 
inspired by changes made in SOLR-3177 and I created a patch base on 
_branch_5_4_.


> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). 
> The Analytics Component was added in SOLR-5302.
> I tried to add this feature in simplified way, where the tags are specified 
> in param value turning this component on, i.e. olap={!ex=tag_name}true. 
> I was inspired by changes made in SOLR-3177 and I created a patch base on 
> _branch_5_4_.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Attachment: (was: SOLR-5302_filter_exclusion.patch)

> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). The Analytics 
> Component was added in SOLR-5302.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Attachment: SOLR-8840_filter_exclusion.patch

> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). The Analytics 
> Component was added in SOLR-5302.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Attachment: SOLR-5302_filter_exclusion.patch

> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). The Analytics 
> Component was added in SOLR-5302.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lubomír Pokorný updated SOLR-8840:
--
Description: 
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). The Analytics Component 
was added in SOLR-5302.

I tried to add this feature in simplified way, where the tags are specified in 
param value turning this component on, i.e. _olap={!ex=tag_name}true_. I was 
inspired by changes made in SOLR-3177 and I created a patch base on 
_branch_5_4_.

  was:
There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). The Analytics Component 
was added in SOLR-5302.



> Analytics Component - tagged filters exclusion support
> --
>
> Key: SOLR-8840
> URL: https://issues.apache.org/jira/browse/SOLR-8840
> Project: Solr
>  Issue Type: Wish
>  Components: SearchComponents - other
>Reporter: Lubomír Pokorný
>Priority: Minor
> Attachments: SOLR-8840_filter_exclusion.patch
>
>
> There is a really nice component for gaining advanced statistics, but it 
> doesn`t support tagged filters exclusion. This would be very nice feature, 
> since we use it for gaining data for price histogram, which should be 
> independent on filters selected by user (of our eshop). The Analytics 
> Component was added in SOLR-5302.
> I tried to add this feature in simplified way, where the tags are specified 
> in param value turning this component on, i.e. _olap={!ex=tag_name}true_. I 
> was inspired by changes made in SOLR-3177 and I created a patch base on 
> _branch_5_4_.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8840) Analytics Component - tagged filters exclusion support

[JIRA]

Lubomír Pokorný created SOLR-8840:
-

 Summary: Analytics Component - tagged filters exclusion support
 Key: SOLR-8840
 URL: https://issues.apache.org/jira/browse/SOLR-8840
 Project: Solr
  Issue Type: Wish
  Components: SearchComponents - other
Reporter: Lubomír Pokorný
Priority: Minor


There is a really nice component for gaining advanced statistics, but it 
doesn`t support tagged filters exclusion. This would be very nice feature, 
since we use it for gaining data for price histogram, which should be 
independent on filters selected by user (of our eshop). The Analytics Component 
was added in SOLR-5302.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8839) Angular admin/segments display: display of deleted docs not proportional

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193079#comment-15193079
 ] 

ASF GitHub Bot commented on SOLR-8839:
--

GitHub user LucVL opened a pull request:

https://github.com/apache/lucene-solr/pull/20

SOLR-8839: Angular admin/segments display: display of deleted docs no…

…t proportional

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/LucVL/lucene-solr solr-8839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/20.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20


commit c187ffaca74ba6d95f70fd85569879c034b66617
Author: Luc Vanlerberghe 
Date:   2016-03-14T10:38:56Z

SOLR-8839: Angular admin/segments display: display of deleted docs not 
proportional




> Angular admin/segments display: display of deleted docs not proportional
> 
>
> Key: SOLR-8839
> URL: https://issues.apache.org/jira/browse/SOLR-8839
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.4.1
>Reporter: Luc Vanlerberghe
>Priority: Minor
>
> In the /segments portion of the admin site, the segments are displayed as a 
> bar graph with the size of each bar proportional to the logarithm of the 
> segment size.
> Within each bar the number of deleted documents is shown as a dark-gray 
> portion at the end.
> Before the angular version, the size of this part was directly proportional 
> to the number of deleted documents with respect to the total number of 
> documents in the segment
> In the angular version, the dark-gray portion is way too large.
> In the previous version, the result was odd as well since it displayed a 
> proportional percentage within in a logarithmic graph.
> I'll add a PR shortly that changes the calculation so the dark-gray part 
> looks approximately proportional to the size the segment would shrink if 
> optimized.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request: SOLR-8839: Angular admin/segments displa...

[LucVL]

GitHub user LucVL opened a pull request:

https://github.com/apache/lucene-solr/pull/20

SOLR-8839: Angular admin/segments display: display of deleted docs no…

…t proportional

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/LucVL/lucene-solr solr-8839

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/20.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20


commit c187ffaca74ba6d95f70fd85569879c034b66617
Author: Luc Vanlerberghe 
Date:   2016-03-14T10:38:56Z

SOLR-8839: Angular admin/segments display: display of deleted docs not 
proportional




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8839) Angular admin/segments display: display of deleted docs not proportional

[Luc Vanlerberghe (JIRA)]

Luc Vanlerberghe created SOLR-8839:
--

 Summary: Angular admin/segments display: display of deleted docs 
not proportional
 Key: SOLR-8839
 URL: https://issues.apache.org/jira/browse/SOLR-8839
 Project: Solr
  Issue Type: Bug
Affects Versions: 5.4.1
Reporter: Luc Vanlerberghe
Priority: Minor


In the /segments portion of the admin site, the segments are displayed as a bar 
graph with the size of each bar proportional to the logarithm of the segment 
size.
Within each bar the number of deleted documents is shown as a dark-gray portion 
at the end.
Before the angular version, the size of this part was directly proportional to 
the number of deleted documents with respect to the total number of documents 
in the segment
In the angular version, the dark-gray portion is way too large.

In the previous version, the result was odd as well since it displayed a 
proportional percentage within in a logarithmic graph.

I'll add a PR shortly that changes the calculation so the dark-gray part looks 
approximately proportional to the size the segment would shrink if optimized.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7093) MemoryIndex does not support points

[Martijn van Groningen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martijn van Groningen updated LUCENE-7093:
--
Attachment: LUCENE-7093.patch

Attached a patch that adds point values support for the memory index.
This patch is based on LUCENE-7091, which will need to get merged first before 
this one. 

> MemoryIndex does not support points
> ---
>
> Key: LUCENE-7093
> URL: https://issues.apache.org/jira/browse/LUCENE-7093
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
>Assignee: Martijn van Groningen
> Attachments: LUCENE-7093.patch
>
>
> I realized this glancing at LUCENE-7091.
> I think this should have points support or else people cannot move off of the 
> deprecated LegacyXXX encodings?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6954) More Like This Query Generation

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193024#comment-15193024
 ] 

Alessandro Benedetti commented on LUCENE-6954:
--

Thank you Anshum,
would be great to have a second opinion and possibly improve the patch/MLT .
Let me know, 

Cheers

> More Like This Query Generation 
> 
>
> Key: LUCENE-6954
> URL: https://issues.apache.org/jira/browse/LUCENE-6954
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/other
>Affects Versions: 5.4
>Reporter: Alessandro Benedetti
>  Labels: morelikethis
> Attachments: LUCENE-6954.patch
>
>
> Currently the query is generated : 
> org.apache.lucene.queries.mlt.MoreLikeThis#retrieveTerms(int)
> 1) we extract the terms from the interesting fields, adding them to a map :
> Map termFreqMap = new HashMap<>();
> ( we lose the relation field-> term, we don't know anymore where the term was 
> coming ! )
> org.apache.lucene.queries.mlt.MoreLikeThis#createQueue
> 2) we build the queue that will contain the query terms, at this point we 
> connect again there terms to some field, but :
> ...
> // go through all the fields and find the largest document frequency
> String topField = fieldNames[0];
> int docFreq = 0;
> for (String fieldName : fieldNames) {
>   int freq = ir.docFreq(new Term(fieldName, word));
>   topField = (freq > docFreq) ? fieldName : topField;
>   docFreq = (freq > docFreq) ? freq : docFreq;
> }
> ...
> We identify the topField as the field with the highest document frequency for 
> the term t .
> Then we build the termQuery :
> queue.add(new ScoreTerm(word, topField, score, idf, docFreq, tf));
> In this way we lose a lot of precision.
> Not sure why we do that.
> I would prefer to keep the relation between terms and fields.
> The MLT query can improve a lot the quality.
> If i run the MLT on 2 fields : weSell and weDontSell for example.
> It is likely I want to find documents with similar terms in the weSell and 
> similar terms in the weDontSell, without mixing up the things and loosing the 
> semantic of the terms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7097) Can we increase the stack depth before Introsorter switches to heapsort?

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193020#comment-15193020
 ] 

ASF subversion and git services commented on LUCENE-7097:
-

Commit d1ba89959137c120be5cf116e72f43b26339cb6e in lucene-solr's branch 
refs/heads/branch_6x from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d1ba899 ]

LUCENE-7097: let IntroSorter go 2X deeper in quicksort before switching to 
heapsort


> Can we increase the stack depth before Introsorter switches to heapsort?
> 
>
> Key: LUCENE-7097
> URL: https://issues.apache.org/jira/browse/LUCENE-7097
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: trunk, 6.1
>
> Attachments: LUCENE-7097.patch
>
>
> Introsort is a "safe" quicksort: it uses quicksort but detects when an 
> adversary is at work and cuts over to heapsort at that point.
> The description at https://en.wikipedia.org/wiki/Introsort shows the cutover 
> as 2X log_2(N) but our impl ({{IntroSorter}}) currently uses just log_2.
> So I tested using 2X log_2 instead, and I see a decent (~5.6%, from 98.2 sec 
> to 92.7 sec) speedup in the time for offline sorter to sort when doing the 
> force merge of 6.1 LatLonPoints from the London UK benchmark.
> Is there any reason not to switch?  I know this means 2X the stack required, 
> but since this is log_2 space that seems fine?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-7097) Can we increase the stack depth before Introsorter switches to heapsort?

[Michael McCandless (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless resolved LUCENE-7097.

Resolution: Fixed

Thanks [~jpountz].

> Can we increase the stack depth before Introsorter switches to heapsort?
> 
>
> Key: LUCENE-7097
> URL: https://issues.apache.org/jira/browse/LUCENE-7097
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: trunk, 6.1
>
> Attachments: LUCENE-7097.patch
>
>
> Introsort is a "safe" quicksort: it uses quicksort but detects when an 
> adversary is at work and cuts over to heapsort at that point.
> The description at https://en.wikipedia.org/wiki/Introsort shows the cutover 
> as 2X log_2(N) but our impl ({{IntroSorter}}) currently uses just log_2.
> So I tested using 2X log_2 instead, and I see a decent (~5.6%, from 98.2 sec 
> to 92.7 sec) speedup in the time for offline sorter to sort when doing the 
> force merge of 6.1 LatLonPoints from the London UK benchmark.
> Is there any reason not to switch?  I know this means 2X the stack required, 
> but since this is log_2 space that seems fine?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8542) Integrate Learning to Rank into Solr

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193017#comment-15193017
 ] 

Alessandro Benedetti commented on SOLR-8542:


As I briefly discussed with Diego, about how to include the training in Solr as 
well :
A simple integration could be :

1) select a supported training library for linear SVM and one for the 
LambdaMart ( basically the libraries that you already suggest in the README 
could be a starting point)

2) create an Update Request handler that accepts the training set ( and the 
format of the training set will be clearly described in the documentation like 
: LETOR )
This update handler will basically take the training set file and related 
parameters supported by the related library to proceed with the training. 
Trying to use the default configuration parameter where possible, in the way to 
make it as easy as possible the user interaction.
The update handler  will then extract the document features ( a revisit of the 
cache could be interesting in here, to improve the rycicling of feature 
extraction)

3) update request handler will train the model calling internally the selected 
library , using all the parameters provided. The model generated will be 
converted in the supported Json format and stored in the model store.

This sample approach could be complicated as much as we want ( we can add 
flexibility in the library to be used and make it easy to extend) .
A further next step could be to add a layer of signal processing directly in 
Solr , to build the training set as well .
( a sort of REST Api that takes in input the  document, queryId, rating score) 
and automatically create an entry of the training set stored in some smart way.
Than we can trigger the model generation or set up  schedule to refresh the 
model automatically.
We could even take into account only certain periods, store training data in 
different places, clean the training set automatically from time to time ect 
ext :)
Now I am going off topic, but there are a lot of things to do with the training 
, to ease the integration :)
Happy to discuss them and get new ideas to improve the plugin which I think is 
going to be really , really valuable for the Solr community

> Integrate Learning to Rank into Solr
> 
>
> Key: SOLR-8542
> URL: https://issues.apache.org/jira/browse/SOLR-8542
> Project: Solr
>  Issue Type: New Feature
>Reporter: Joshua Pantony
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: README.md, README.md, SOLR-8542-branch_5x.patch, 
> SOLR-8542-trunk.patch
>
>
> This is a ticket to integrate learning to rank machine learning models into 
> Solr. Solr Learning to Rank (LTR) provides a way for you to extract features 
> directly inside Solr for use in training a machine learned model. You can 
> then deploy that model to Solr and use it to rerank your top X search 
> results. This concept was previously presented by the authors at Lucene/Solr 
> Revolution 2015 ( 
> http://www.slideshare.net/lucidworks/learning-to-rank-in-solr-presented-by-michael-nilsson-diego-ceccarelli-bloomberg-lp
>  ).
> The attached code was jointly worked on by Joshua Pantony, Michael Nilsson, 
> David Grohmann and Diego Ceccarelli.
> Any chance this could make it into a 5x release? We've also attached 
> documentation as a github MD file, but are happy to convert to a desired 
> format.
> h3. Test the plugin with solr/example/techproducts in 6 steps
> Solr provides some simple example of indices. In order to test the plugin 
> with 
> the techproducts example please follow these steps
> h4. 1. compile solr and the examples 
> cd solr
> ant dist
> ant example
> h4. 2. run the example
> ./bin/solr -e techproducts 
> h4. 3. stop it and install the plugin:
>
> ./bin/solr stop
> mkdir example/techproducts/solr/techproducts/lib
> cp build/contrib/ltr/lucene-ltr-6.0.0-SNAPSHOT.jar 
> example/techproducts/solr/techproducts/lib/
> cp contrib/ltr/example/solrconfig.xml 
> example/techproducts/solr/techproducts/conf/
> h4. 4. run the example again
> 
> ./bin/solr -e techproducts
> h4. 5. index some features and a model
> curl -XPUT 'http://localhost:8983/solr/techproducts/schema/fstore'  
> --data-binary "@./contrib/ltr/example/techproducts-features.json"  -H 
> 'Content-type:application/json'
> curl -XPUT 'http://localhost:8983/solr/techproducts/schema/mstore'  
> --data-binary "@./contrib/ltr/example/techproducts-model.json"  -H 
> 'Content-type:application/json'
> h4. 6. have fun !
> *access to the default feature store*
> http://localhost:8983/solr/techproducts/schema/fstore/_DEFAULT_ 
> *access to the model store*
> http://localhost:8983/solr/techproducts/schema/mstore
> *perform a query using the model, and retrieve the features*
> 

[jira] [Commented] (LUCENE-7097) Can we increase the stack depth before Introsorter switches to heapsort?

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193018#comment-15193018
 ] 

ASF subversion and git services commented on LUCENE-7097:
-

Commit 8cbe4713775565a3194e29b90747f59fe2ffe3f1 in lucene-solr's branch 
refs/heads/master from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8cbe471 ]

LUCENE-7097: let IntroSorter go 2X deeper in quicksort before switching to 
heapsort


> Can we increase the stack depth before Introsorter switches to heapsort?
> 
>
> Key: LUCENE-7097
> URL: https://issues.apache.org/jira/browse/LUCENE-7097
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: trunk, 6.1
>
> Attachments: LUCENE-7097.patch
>
>
> Introsort is a "safe" quicksort: it uses quicksort but detects when an 
> adversary is at work and cuts over to heapsort at that point.
> The description at https://en.wikipedia.org/wiki/Introsort shows the cutover 
> as 2X log_2(N) but our impl ({{IntroSorter}}) currently uses just log_2.
> So I tested using 2X log_2 instead, and I see a decent (~5.6%, from 98.2 sec 
> to 92.7 sec) speedup in the time for offline sorter to sort when doing the 
> force merge of 6.1 LatLonPoints from the London UK benchmark.
> Is there any reason not to switch?  I know this means 2X the stack required, 
> but since this is log_2 space that seems fine?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8542) Integrate Learning to Rank into Solr

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193007#comment-15193007
 ] 

Alessandro Benedetti commented on SOLR-8542:


Hi Joshua,
I was assuming you had some sort of script/app tranformer to parse the xml and 
build your Json :)
I think could be definitely useful to have it as well.

I understand and I agree you didn't want to force the user to any specific 
training library ( and related model in output) .
But in the end, the plugin supports ( at the moment) 2 possible learned model ( 
linear SVM and LambdaMart), so I think can be really helpful to provide users 
with a step by step guide to run an example end to end.

I think the next step could be to add the training component in Solr as well.
I will describe in another post in this issue, a possible basic approach :)


> Integrate Learning to Rank into Solr
> 
>
> Key: SOLR-8542
> URL: https://issues.apache.org/jira/browse/SOLR-8542
> Project: Solr
>  Issue Type: New Feature
>Reporter: Joshua Pantony
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: README.md, README.md, SOLR-8542-branch_5x.patch, 
> SOLR-8542-trunk.patch
>
>
> This is a ticket to integrate learning to rank machine learning models into 
> Solr. Solr Learning to Rank (LTR) provides a way for you to extract features 
> directly inside Solr for use in training a machine learned model. You can 
> then deploy that model to Solr and use it to rerank your top X search 
> results. This concept was previously presented by the authors at Lucene/Solr 
> Revolution 2015 ( 
> http://www.slideshare.net/lucidworks/learning-to-rank-in-solr-presented-by-michael-nilsson-diego-ceccarelli-bloomberg-lp
>  ).
> The attached code was jointly worked on by Joshua Pantony, Michael Nilsson, 
> David Grohmann and Diego Ceccarelli.
> Any chance this could make it into a 5x release? We've also attached 
> documentation as a github MD file, but are happy to convert to a desired 
> format.
> h3. Test the plugin with solr/example/techproducts in 6 steps
> Solr provides some simple example of indices. In order to test the plugin 
> with 
> the techproducts example please follow these steps
> h4. 1. compile solr and the examples 
> cd solr
> ant dist
> ant example
> h4. 2. run the example
> ./bin/solr -e techproducts 
> h4. 3. stop it and install the plugin:
>
> ./bin/solr stop
> mkdir example/techproducts/solr/techproducts/lib
> cp build/contrib/ltr/lucene-ltr-6.0.0-SNAPSHOT.jar 
> example/techproducts/solr/techproducts/lib/
> cp contrib/ltr/example/solrconfig.xml 
> example/techproducts/solr/techproducts/conf/
> h4. 4. run the example again
> 
> ./bin/solr -e techproducts
> h4. 5. index some features and a model
> curl -XPUT 'http://localhost:8983/solr/techproducts/schema/fstore'  
> --data-binary "@./contrib/ltr/example/techproducts-features.json"  -H 
> 'Content-type:application/json'
> curl -XPUT 'http://localhost:8983/solr/techproducts/schema/mstore'  
> --data-binary "@./contrib/ltr/example/techproducts-model.json"  -H 
> 'Content-type:application/json'
> h4. 6. have fun !
> *access to the default feature store*
> http://localhost:8983/solr/techproducts/schema/fstore/_DEFAULT_ 
> *access to the model store*
> http://localhost:8983/solr/techproducts/schema/mstore
> *perform a query using the model, and retrieve the features*
> http://localhost:8983/solr/techproducts/query?indent=on=test=json={!ltr%20model=svm%20reRankDocs=25%20efi.query=%27test%27}=*,[features],price,score,name=true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6644) DataImportHandler holds on to each DB connection until the end

[Mahmoud Jalajel (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193006#comment-15193006
 ] 

Mahmoud Jalajel commented on SOLR-6644:
---

We're seeing similar symptoms on version 5.3.1 Any updates on this issue?
cc: [~gthb]

> DataImportHandler holds on to each DB connection until the end
> --
>
> Key: SOLR-6644
> URL: https://issues.apache.org/jira/browse/SOLR-6644
> Project: Solr
>  Issue Type: Bug
>  Components: contrib - DataImportHandler
>Affects Versions: 4.7, 4.10.1
>Reporter: Gunnlaugur Thor Briem
>  Labels: dataimport, jdbc
>
> DataImportHandler with a JDBC data source opens one DB connection per entity, 
> and then holds on to that DB connection with an open transaction after it's 
> finished processing that entity ... right until the whole DataImportHandler 
> operation is finished.
> So this can mean dozens of DB connections tied up for hours, unnecessarily 
> --- with each connection staying in "idle in transaction" state, holding (in 
> PostgreSQL) an AccessShareLock on each relation it has looked at. Not ideal 
> for production operations, of course.
> Here are the connections from Solr to the DB when a large import has been 
> running for a while:
> {code}
>  backend_start | xact_start | query_start |state| minutes 
> idle 
> ---++-+-+--
>  20:03:20  | 20:03:20   | 20:03:21| idle in transaction |   32
>  20:03:22  | 20:03:22   | 20:03:22| idle in transaction |   32
>  20:03:22  | 20:03:22   | 20:03:22| idle in transaction |   32
>  20:03:22  | 20:03:22   | 20:03:23| idle in transaction |   32
>  20:03:21  | 20:03:21   | 20:16:35| idle in transaction |   19
>  20:03:21  | 20:03:21   | 20:16:35| idle in transaction |   19
>  20:03:22  | 20:03:22   | 20:16:35| idle in transaction |   19
>  20:03:22  | 20:03:22   | 20:16:35| idle in transaction |   19
>  20:03:22  | 20:03:22   | 20:16:35| idle in transaction |   19
>  20:16:37  | 20:16:37   | 20:16:38| idle in transaction |   19
>  20:03:21  | 20:03:21   | 20:16:35| idle in transaction |   19
>  20:03:21  | 20:03:21   | 20:16:35| idle in transaction |   19
>  20:03:21  | 20:03:21   | 20:16:35| idle in transaction |   19
>  20:16:36  | 20:16:36   | 20:16:37| idle in transaction |   19
>  20:03:20  | 20:03:20   | 20:16:35| idle in transaction |   19
>  20:16:36  | 20:16:36   | 20:35:49| idle in transaction |0
>  20:16:36  | 20:16:36   | 20:35:49| idle in transaction |0
>  20:16:37  | 20:16:37   | 20:35:49| idle in transaction |0
>  20:16:35  | 20:16:35   | 20:35:41| idle in transaction |0
>  20:16:36  | 20:16:36   | 20:35:49| idle in transaction |0
>  20:16:37  | 20:16:37   | 20:35:49| active  |0
> {code}
> Most of these haven't been touched for a long time, and will not be needed 
> again (because DataImportHandler is done with that top-level entity). They 
> should be released as soon as possible.
> Noticed in production in Solr 4.7.0, then reproduced in 4.10.1 (so probably 
> also true of all versions inbetween).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8672) Unique Suggestions getter in Solrj

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192984#comment-15192984
 ] 

Alessandro Benedetti commented on SOLR-8672:


Hi David,
I agree this was a simple contribution to try to have a quick fix to the 
problem at least at SolrJ level.
I thought more about the problem, and I agree with Tomas.
I even think we should NOT include my own patch here, this is my reason :

if we add now this SolrJ support and in the future we add the suggester 
solrconfig.xml parameter, users are going to be confused.
The solrconfig.xml approach will be the most efficient and recommended but the 
presence of this method SolrJ side could grow confusion in people minds and 
generate some scenario where you use the Solrj method but actually going to the 
solr configuration would be optimal.
Does this make sense ?

I will proceed in studying a little bit LUCENE-6636 and some internals, I will 
end up with a proposal here and in the mailing list ( with related Jira) .

Not sure if a Dedup Wrapper is the best way, ideally I don't want to generate 
at all duplicates, avoid  a wrapper reading the list again and removing 
duplicates.
I will spend some time on that!

Cheers

> Unique Suggestions getter in Solrj
> --
>
> Key: SOLR-8672
> URL: https://issues.apache.org/jira/browse/SOLR-8672
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.4.1
>Reporter: Alessandro Benedetti
> Attachments: SOLR-8672.patch
>
>
> Currently all the suggesters based on full field content retrieval gives back 
> possibly duplicated suggestions.
> First observation , related the suggester domain, is that is unlikely we need 
> to return duplicates ( actually we don't return the id of the document, or 
> anything else, so having duplicates is arguably not a benefit) .
> I propose at least to offer via SolrJ  the possibility of getting the 
> suggestions without duplicates.
> Any feedback is welcome.
> The patch provided is really simple.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7097) Can we increase the stack depth before Introsorter switches to heapsort?

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192959#comment-15192959
 ] 

Adrien Grand commented on LUCENE-7097:
--

+1

> Can we increase the stack depth before Introsorter switches to heapsort?
> 
>
> Key: LUCENE-7097
> URL: https://issues.apache.org/jira/browse/LUCENE-7097
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: trunk, 6.1
>
> Attachments: LUCENE-7097.patch
>
>
> Introsort is a "safe" quicksort: it uses quicksort but detects when an 
> adversary is at work and cuts over to heapsort at that point.
> The description at https://en.wikipedia.org/wiki/Introsort shows the cutover 
> as 2X log_2(N) but our impl ({{IntroSorter}}) currently uses just log_2.
> So I tested using 2X log_2 instead, and I see a decent (~5.6%, from 98.2 sec 
> to 92.7 sec) speedup in the time for offline sorter to sort when doing the 
> force merge of 6.1 LatLonPoints from the London UK benchmark.
> Is there any reason not to switch?  I know this means 2X the stack required, 
> but since this is log_2 space that seems fine?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 11 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/11/

2 tests failed.
FAILED:  org.apache.solr.update.AutoCommitTest.testMaxDocs

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([842077131ABD72FC:3DA1A1CC36577676]:0)
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:764)
at 
org.apache.solr.update.AutoCommitTest.testMaxDocs(AutoCommitTest.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result[@numFound=1]
xml response was: 

00


request was:q=id:14=standard=0=20=2.2
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:757)
... 40 more


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=67900, name=collection4, 

[JENKINS-EA] Lucene-Solr-6.x-Linux (32bit/jdk-9-ea+109) - Build # 126 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/126/
Java: 32bit/jdk-9-ea+109 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.UnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=3566, 
name=testExecutor-1590-thread-12, state=RUNNABLE, 
group=TGRP-UnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=3566, name=testExecutor-1590-thread-12, 
state=RUNNABLE, group=TGRP-UnloadDistributedZkTest]
at 
__randomizedtesting.SeedInfo.seed([ED45E9A91CFC5F5C:6511D673B20032A4]:0)
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:34206/nf
at __randomizedtesting.SeedInfo.seed([ED45E9A91CFC5F5C]:0)
at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:583)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)
at java.lang.Thread.run(Thread.java:804)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured 
while waiting response from server at: http://127.0.0.1:34206/nf
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:588)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:581)
... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:482)
... 8 more




Build Log:
[...truncated 10846 lines...]
   [junit4] Suite: org.apache.solr.cloud.UnloadDistributedZkTest
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.UnloadDistributedZkTest_ED45E9A91CFC5F5C-001/init-core-data-001
   [junit4]   2> 300117 INFO  
(SUITE-UnloadDistributedZkTest-seed#[ED45E9A91CFC5F5C]-worker) [] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /nf/
   [junit4]   2> 300119 INFO  
(TEST-UnloadDistributedZkTest.test-seed#[ED45E9A91CFC5F5C]) [] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 300119 INFO  (Thread-970) [] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   

[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+109) - Build # 16208 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/16208/
Java: 32bit/jdk-9-ea+109 -client -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.UnloadDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=906, 
name=testExecutor-502-thread-1, state=RUNNABLE, 
group=TGRP-UnloadDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=906, name=testExecutor-502-thread-1, 
state=RUNNABLE, group=TGRP-UnloadDistributedZkTest]
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:37764
at __randomizedtesting.SeedInfo.seed([8FEC13EBAE93288E]:0)
at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:583)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1158)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632)
at java.lang.Thread.run(Thread.java:804)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured 
while waiting response from server at: http://127.0.0.1:37764
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:588)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.cloud.BasicDistributedZkTest.lambda$createCores$0(BasicDistributedZkTest.java:581)
... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:482)
... 8 more




Build Log:
[...truncated 10631 lines...]
   [junit4] Suite: org.apache.solr.cloud.UnloadDistributedZkTest
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.UnloadDistributedZkTest_8FEC13EBAE93288E-001/init-core-data-001
   [junit4]   2> 78840 INFO  
(SUITE-UnloadDistributedZkTest-seed#[8FEC13EBAE93288E]-worker) [] 
o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 78844 INFO  
(TEST-UnloadDistributedZkTest.test-seed#[8FEC13EBAE93288E]) [] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 78844 INFO  (Thread-258) [] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 78844 INFO  (Thread-258) [] o.a.s.c.ZkTestServer Starting 
server
   [junit4]   2> 78944 

[jira] [Commented] (LUCENE-6966) Contribution: Codec for index-level encryption

[Karl Sanders (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15192849#comment-15192849
 ] 

Karl Sanders commented on LUCENE-6966:
--

There's an apparently abandoned project that might be of interest:
https://code.google.com/archive/p/lucenetransform/

It appears to be implementing compression and encryption for Lucene indexes.
I also found a couple of related links.

- Some considerations about how it's being used in another project:
https://github.com/muzima/documentation/wiki/Security-regarding-stored-data-by-using-Lucene

- A discussion about ensuring that indexes aren't tampered with:
http://permalink.gmane.org/gmane.comp.jakarta.lucene.user/50495

> Contribution: Codec for index-level encryption
> --
>
> Key: LUCENE-6966
> URL: https://issues.apache.org/jira/browse/LUCENE-6966
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: modules/other
>Reporter: Renaud Delbru
>  Labels: codec, contrib
>
> We would like to contribute a codec that enables the encryption of sensitive 
> data in the index that has been developed as part of an engagement with a 
> customer. We think that this could be of interest for the community.
> Below is a description of the project.
> h1. Introduction
> In comparison with approaches where all data is encrypted (e.g., file system 
> encryption, index output / directory encryption), encryption at a codec level 
> enables more fine-grained control on which block of data is encrypted. This 
> is more efficient since less data has to be encrypted. This also gives more 
> flexibility such as the ability to select which field to encrypt.
> Some of the requirements for this project were:
> * The performance impact of the encryption should be reasonable.
> * The user can choose which field to encrypt.
> * Key management: During the life cycle of the index, the user can provide a 
> new version of his encryption key. Multiple key versions should co-exist in 
> one index.
> h1. What is supported ?
> - Block tree terms index and dictionary
> - Compressed stored fields format
> - Compressed term vectors format
> - Doc values format (prototype based on an encrypted index output) - this 
> will be submitted as a separated patch
> - Index upgrader: command to upgrade all the index segments with the latest 
> key version available.
> h1. How it is implemented ?
> h2. Key Management
> One index segment is encrypted with a single key version. An index can have 
> multiple segments, each one encrypted using a different key version. The key 
> version for a segment is stored in the segment info.
> The provided codec is abstract, and a subclass is responsible in providing an 
> implementation of the cipher factory. The cipher factory is responsible of 
> the creation of a cipher instance based on a given key version.
> h2. Encryption Model
> The encryption model is based on AES/CBC with padding. Initialisation vector 
> (IV) is reused for performance reason, but only on a per format and per 
> segment basis.
> While IV reuse is usually considered a bad practice, the CBC mode is somehow 
> resilient to IV reuse. The only "leak" of information that this could lead to 
> is being able to know that two encrypted blocks of data starts with the same 
> prefix. However, it is unlikely that two data blocks in an index segment will 
> start with the same data:
> - Stored Fields Format: Each encrypted data block is a compressed block 
> (~4kb) of one or more documents. It is unlikely that two compressed blocks 
> start with the same data prefix.
> - Term Vectors: Each encrypted data block is a compressed block (~4kb) of 
> terms and payloads from one or more documents. It is unlikely that two 
> compressed blocks start with the same data prefix.
> - Term Dictionary Index: The term dictionary index is encoded and encrypted 
> in one single data block.
> - Term Dictionary Data: Each data block of the term dictionary encodes a set 
> of suffixes. It is unlikely to have two dictionary data blocks sharing the 
> same prefix within the same segment.
> - DocValues: A DocValues file will be composed of multiple encrypted data 
> blocks. It is unlikely to have two data blocks sharing the same prefix within 
> the same segment (each one will encodes a list of values associated to a 
> field).
> To the best of our knowledge, this model should be safe. However, it would be 
> good if someone with security expertise in the community could review and 
> validate it. 
> h1. Performance
> We report here a performance benchmark we did on an early prototype based on 
> Lucene 4.x. The benchmark was performed on the Wikipedia dataset where all 
> the fields (id, title, body, date) were encrypted. Only the block tree terms 
> and compressed stored fields format were tested at that 

[JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+109) - Build # 125 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/125/
Java: 64bit/jdk-9-ea+109 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.lucene.codecs.lucene50.TestLucene50NormsFormat.testOutliers

Error Message:
Unable to unmap the mapped buffer: 
MMapIndexInput(path="/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/build/backward-codecs/test/J1/temp/lucene.codecs.lucene50.TestLucene50NormsFormat_2D8861A4DA56AC49-001/index-MMapDirectory-001/_y.fdx")

Stack Trace:
java.io.IOException: Unable to unmap the mapped buffer: 
MMapIndexInput(path="/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/build/backward-codecs/test/J1/temp/lucene.codecs.lucene50.TestLucene50NormsFormat_2D8861A4DA56AC49-001/index-MMapDirectory-001/_y.fdx")
at 
__randomizedtesting.SeedInfo.seed([2D8861A4DA56AC49:DCB1FF5577A70A98]:0)
at 
org.apache.lucene.store.MMapDirectory.lambda$unmapHackImpl$1(MMapDirectory.java:384)
at 
org.apache.lucene.store.ByteBufferIndexInput.freeBuffer(ByteBufferIndexInput.java:376)
at 
org.apache.lucene.store.ByteBufferIndexInput.close(ByteBufferIndexInput.java:355)
at 
org.apache.lucene.store.MockIndexInputWrapper.close(MockIndexInputWrapper.java:61)
at 
org.apache.lucene.index.IndexWriter.slowFileExists(IndexWriter.java:4815)
at org.apache.lucene.index.IndexWriter.filesExist(IndexWriter.java:4352)
at 
org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4423)
at 
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2876)
at 
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2989)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2956)
at 
org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:288)
at 
org.apache.lucene.index.BaseNormsFormatTestCase.doTestNormsVersusDocValues(BaseNormsFormatTestCase.java:262)
at 
org.apache.lucene.index.BaseNormsFormatTestCase.testOutliers(BaseNormsFormatTestCase.java:161)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:520)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_72) - Build # 16207 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/16207/
Java: 32bit/jdk1.8.0_72 -client -XX:+UseConcMarkSweepGC

20 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestCloudPivotFacet

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([A5358077E1BED06C]:0)


FAILED:  org.apache.solr.TestSolrCoreProperties.testSimple

Error Message:
IOException occured when talking to server at: 
http://127.0.0.1:39767/solr/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: http://127.0.0.1:39767/solr/collection1
at 
__randomizedtesting.SeedInfo.seed([A5358077E1BED06C:9D86A489C64D04BD]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
at 
org.apache.solr.TestSolrCoreProperties.testSimple(TestSolrCoreProperties.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at