Re: Old programmers do fade away

[Bram Van Dam]

Your presence on the user mailing list will be sorely missed. You've
been an invaluable source of good advice for years. I owe you at least
$bignum beers.

Best of luck with the squirrel problem (they are rather tasty) and any
other fun projects!

Take care, and thank you,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

versions = true in SolrCloud (SOLR-8733)

[Bram Van Dam]

Hey folks,

Been taking a look at SOLR-8733. Basically the versions=true param
doesn't work in SolrCloud unless you manage to direct each update to the
leader. The docs don't mention that this is broken in SolrCloud, and
according to Jira it seems like @yonik never intended this to be a
public feature in the first place.

It still seems like a useful feature, and it would make our life easier.

>From poking around in the code for the last hour or so, I think this
could be made to work by taking a similar approach to how the
Replication Factor is tracked across nodes (using
RollupRequestReplicationTracker and LeaderRequestReplicationTracker).

I think I'm going to spend some time implementing this and I'll see if I
can submit a patch. Given the age of the jira issue (2016) I figured I'd
check in first. Is there any reason why this would be a terrible idea or
why this might end up not getting merged?

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Cross Data-Center Replication in Apache Solr

[Bram Van Dam]

We've had some experience with this. As per usual, customers tend to
want the holy grail: no data loss when one data centre blows up, but no
increased latency when updating data. This then somehow, magically, has
to work over a slow uplink between two data centres without saturating
the link.

Currently we use NRT replicas across data centres. Which does add some
latency, but consistency is a bit more important for us. Overall, this
works pretty well.

The biggest problems we've experienced have all been related to
recovering replicas across a slow data centre uplink. A saturated link
can cause multiple replicas to lag behind, and when this gets too bad,
they too will go into recovery, and then shit really hits the fan.

I'm not sure whether there are any easy ways of improving that
behaviour. Limiting max bandwidth per solr instance during recovery?
Slow recovery is better than destructive recovery.

External tools like Kafka add a lot of operational overhead. One of the
great things about SolrCloud is how simple the whole replication setup is.



On 06/12/2020 14:46, Erick Erickson wrote:
>> I can see at least two different approaches here, your mention of SolrJ 
>> seems to hint at the first one:
>> 1. Get the data as it comes from the client and fork it to local and remote 
>> data centers,
>> 2. Create (an asynchronous) stream replicating local data center data to 
>> remote.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Thinking about upgrading indexes to X+2

[Bram Van Dam]

On 20/11/2020 19:18, Uwe Schindler wrote:
> Thanks for bringing this again.
> 
> I tend to say: Let us just allow also IndexUpgrader beyodn 2 versions! If 
> somebody complains about incorrect offsets, oh man - It's their problem.

Is there any particular reason why the IndexUpgrader couldn't simply
warn about non-upgradable changes? If you have a data type that's no
longer supported and there's no migration path: error out.

I understand the whole "x != f(x)" upgrade problem, but many scenarios
should be pretty straightforward to upgrade. If a field is stored or has
docvalues, f(x) can simply be recomputed, no?

Am I missing something glaringly obvious here?

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: timeAllowed NPE and possible patch

[Bram Van Dam]

On 19/08/2020 16:11, David Smiley wrote:
> Please file a JIRA issue.  Also mention which version of JIRA you use
> for the "Affects version".

Thanks. Created https://issues.apache.org/jira/browse/SOLR-14758 and
added a patch.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

timeAllowed NPE and possible patch

[Bram Van Dam]

Hey folks,

I would occasionally bump into an NPE in QueryComponent.mergeIds when
using timeAllowed. Here's a little patch which seems to fix the issue. I
suspect sortFieldValues can be null depending on when the query
execution is aborted?

 - Bram

diff --git
a/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
b/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
index 853da1c3a38..c7e7c1e6b86 100644
---
a/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
+++
b/solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
@@ -932,7 +932,7 @@ public class QueryComponent extends SearchComponent

 @SuppressWarnings({"rawtypes"})
 NamedList sortFieldValues =
(NamedList)(srsp.getSolrResponse().getResponse().get("sort_values"));
-if (sortFieldValues.size()==0 && // we bypass merging this
response only if it's partial itself
+if ((null == sortFieldValues || sortFieldValues.size()==0) &&
// we bypass merging this response only if it's partial itself
 thisResponseIsPartial) { // but not the
previous one!!
   continue; //fsv timeout yields empty sort_vlaues
 }

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

SOLR-14413 - timeAllowed/cursorMark

[Bram Van Dam]

Hey folks,

Someone submitted a patch for SOLR-14413 which allows timeAllowed to
work in conjunction with cursor marks. There's a patch in Jira and a PR
(#1436) on github, but it looks like it hasn't been merged.

Is there anything I can do to help get this merged? Or are there any
fundamental objections against merging this? The diff is tiny, not much
more than removing the check which prevents this combination of
parameters from working.

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SolrClient and making requests asynchronously

[Bram Van Dam]

> public CompletableFuture>
requestAsync(SolrRequest request);

NamedList aside, this looks like a great async API to me. But I
would still like some control over the thread pool/executor that's being
used.

Maybe that doesn't have to be part of the requestAysnc method signature,
maybe it could be part of the SolrClient constructor/builder, but then
maybe I'll want a different one for reads and writes? So maybe an
overloaded version of requestAsync with an extra paramater for a thread
pool/executor?

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: clue

[Bram Van Dam]

This looks great! I'm assuming that the 8.5.0 in the release name
indicates that this will only work with Lucene 8.5.0 indexes?

On 10/06/2020 20:15, John Wang wrote:
> Hi folks:
> 
> A while ago I wrote a tool to manage a Lucene
> index: https://github.com/javasoze/clue.
> 
> If there is any interest, I'd like to donate it to the Lucene community
> as I don't find myself having time to properly support it.
> 
> Thank you
> 
> -John


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: StringBuffer usage

[Bram Van Dam]

On 06/06/2020 22:48, Erick Erickson wrote:
> When is there a good reason to use StringBuffer rather than StringBuilder? 
> While going through some of the warnings I happened to run across a few of 
> these. I haven’t changed them, and at least one (AuditEvent) has a comment 
> about back-compat and is quite recent…

StringBuffer hardly ever makes sense. Sure, it's synchronized, but it
usually doesn't do what you want.

If Thread 1 and Thread 2 each call

sb.append("1");
sb.append("2");
sb.append("3");

the result likely won't be "123123". Appending each individual append
operation is synchronized and "safe", but multiple consecutive calls are
not. So unless you're either using an external synchronization
mechanism, or are appending things "atomically", you can still get
screwed. With that in mind, StringBuffer's synchronization is almost
always useless overhead.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Lucene-Solr split (Solr promoted to TLP)

[Bram Van Dam]

On 11/05/2020 15:09, Dawid Weiss wrote:
>> Maybe I'm alone in this, but (better) Lucene compatibility is one of the
>> reasons why our company chose Solr over ElasticSearch.
> 
> I fail to see anything supporting superior Lucene
> compatibility of one vs. another.

Yeah you're right. It's since been pointed out to me that ES uses
vanilla Lucene these days. I hope Solr can remain in the same boat, it
would be a shame to fork Lucene and lose the benefits of compatibility.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Lucene-Solr split (Solr promoted to TLP)

[Bram Van Dam]

On 10/05/2020 08:20, David Smiley wrote:
> An idea just occurred to me that may help make a split nicer for Solr
> than it is today.  Solr could use a branch of the Lucene project that's
> used for the Solr project.

Maybe I'm alone in this, but (better) Lucene compatibility is one of the
reasons why our company chose Solr over ElasticSearch.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] Lucene-Solr split (Solr promoted to TLP)

[Bram Van Dam]

> The big question is this: “Is this the right time to split Solr and
> Lucene into two independent projects?”.

Sounds like there are quite a few tasks to complete to get this done.
Splitting the build and codebase. Presumably a bunch of administration
within Apache/the PMC. Setting up infrastructure etc.

These are the costs, to be paid up front in the currency of someone's
time. The benefits are less clear. Faster build times and easier
maintenance sound attractive, but when will those benefits be visible?
Next month? Or in a year?

Whoever will be doing this work should probably ask themselves the
questions: is this the best use of their time?

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Fwd: [ANNOUNCE] Apache Lucene 7.7.3 released

[Bram Van Dam]

Thanks all, for making this release, much appreciated!

On 04/05/2020 15:16, Noble Paul wrote:
>
>
> -- Forwarded message -
> From: *Noble Paul* mailto:no...@apache.org>>
> Date: Tue, Apr 28, 2020, 4:51 PM
> Subject: [ANNOUNCE] Apache Lucene 7.7.3 released
> To: mailto:annou...@apache.org>>
>
>
>
> 28 April 2020, Apache Lucene™ 7.7.3 available
> 
> The Lucene PMC is pleased to announce the release of Apache Lucene 7.7.3.
> 
> Apache Lucene is a high-performance, full-featured text search engine
> library written entirely in Java. It is a technology suitable for nearly
> any application that requires full-text search, especially cross-platform.
> 
> This release contains 1 bugfix in Lucene. The release is available for
> immediate download at:
>   
> 
> Please read CHANGES.txt for a full list of changes:
>   
> 
> Note: The Apache Software Foundation uses an extensive mirroring network
> for distributing releases. It is possible that the mirror you are using
> may not have replicated the release yet. If that is the case, please try
> another mirror. This also applies to Maven access.
> 
> --Noble Paul
> 
> -
> Noble Paul
> www.lucidworks.com 

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [DISCUSS] SIP-4 Solr should own the bootstrap process

[Bram Van Dam]

On 24/03/2020 16:01, Jan Høydahl wrote:
> It has been proposed many times to make Solr a truly standalone app and that 
> we should start Jetty from within Solr and not the other way around :)

Do you see this running in a single JVM, i.e. you run solr.jar and it
then starts an embedded Jetty server? Or would solr.jar exec a new Java
process and take it from there?

If it's all in one JVM, then the startup scripts probably won't be all
that different. Memory settings, OOM killers etc will still have to be
set up before using startup shellscripts and whatnot. If it's just a
small bootstrap jar which execs another JVM, a lot of the complicated
shell foo could probably be simplified.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

SolrJ: ZooKeeper status path & admin requests

[Bram Van Dam]

Hey folks,

SolrJ's CommonParams.java contains a list of URL paths which are admin
requests (ADMIN_PATHS). Requests to these paths follow a different route
in the (Cloud)SolrClient, one where they don't require a "collection"
property. Which makes sense, after all System Info etc does not relate
to any individual collection.

The ZK_STATUS_PATH is conspicuously missing from this list, which means
it effectively can't queried using SolrJ. The actual path,
/admin/zookeeper/status, seems to suggest that it is, after all, an
admin request.

Is there a reason this is missing? Or shall I open a jira and attach a
patch? Right now I have to jump through hoops to obtain the ZK status
info, it'd be much more convenient if I could just use SolrJ.

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: New Dev

[Bram Van Dam]

> Will start with the docs as you suggested.

To add to that, I recently created a "top ten" list of files that seem
important and have no top level Javadoc. These might be a useful place
to start.


solr/core/src/java/org/apache/solr/core/CoreContainer.java
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
solr/core/src/java/org/apache/solr/handler/StreamHandler.java
solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java
solr/core/src/java/org/apache/solr/search/JoinQParserPlugin.java
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
solr/core/src/java/org/apache/solr/handler/SolrConfigHandler.java

I've been meaning to add an ant target so checkstyle can detect these
automatically, but I haven't gotten around to figuring out how to add a
dependency to the build.

 - Bram

Re: Do we want to pursue an LTS designation?

[Bram Van Dam]

On 15/11/2019 23:07, Erick Erickson wrote:
> Andrzej Bailecki and I worked on something similar, it might be good to 
> compare notes. I gave a talk last Activate that might be useful for you to 
> look at for ideas.
> https://www.youtube.com/watch?v=eaQBH_H3d3g

Thanks, that sounds super useful! I'll look into it.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Do we want to pursue an LTS designation?

[Bram Van Dam]

On 15/11/2019 19:14, Jörn Franke wrote:
> Well sometimes even reindexing might not always be avoidable when upgrading. 
> However, there should be a more user friendly way. Not every Solr deployment 
> that one might inherit has foreseen this. Eg in case Solr is used as a NoSQL 
> database where the application puts data in Solr, but not outside Solr for 
> the case of reindexing. In those case it would be useful that Solr can 
> (configurable) write the original SolrInputDocument somewhere and this 
> original document can then be user for reindexing in case of major updates.

Writing the SolrInputDocument to disk would get really expensive really
quickly.

When all fields are stored it's relatively easy to build a new index
from an old one. But when you have a any non-stored fields, things
become a bit less convenient. In some cases you can uninvert the field
and recreate the content based on the terms used. Still tricky.

A way to perform in-place updates in Lucene would help in this case. But
that seems hard to implement, and it would only help with this
particular upgrade problem.

When I have more time I'll keep hacking away on an upgrade tool that
works for our use-case. If it turns out to be usable for anyone else,
I'll give it back to the community.

Also; not sure why the MS mail servers are messing up some of my mails.
Apologies.

 - Bram

Re: Do we want to pursue an LTS designation?

[Bram Van Dam]

On 13/11/2019 20:36, Erick Erickson wrote:
> I have zero interest in having an LTS “policy” that requires any further 
> commitment of my personal time to random people who ask for it but aren’t 
> willing to pay. I consider my non-paid work to Solr a _volunteer_ activity, 
> not something anyone has a right to demand as “support”, long-term or 
> otherwise.

I hope no one is expecting you to put in free support hours based on
some random LTS designation. And if you're ever in my neck of the woods,
I'll gladly buy you a couple of beers to thank you for the effort you do
put in.

It *is* really annoying that there is no better upgrade path than
"reindex everything and hope for the best". Making it easier to perform
in place upgrades would probably go a long way in making users happier,
and would certainly reduce the calls for an LTS version.

I had some time off this week and I spent it poking around at the
possibility of making index upgrades easier, but I'm afraid I didn't get
anywhere with it. Too much low level Lucene stuff is involved that I'm
not familiar with (yet).

 - Bram

Missing top level javadocs

[Bram Van Dam]

David Smiley mentioned this in the "SolrCloud is sick" thread. Instead
of hijacking that, I figured I'd start another thread.

On 03/11/2019 05:32, David Smiley wrote:
>  requiring javadocs on all top level classes.  I think more javadocs and
> code comments would be very helpful -- especially for the major
> classes.

This sounds like something that's actionable.

I'm not sure if there are any guidelines regarding documentation on the
Solr project, but on my team there's a rule that says all classes must
have a top-level javadoc that explains the "why" of the class. "Why does
it exist/what's it for?"

Excluding contrib, solrj and tests, there are some 400 source files with
classes with missing top level Javadoc. This includes some files with
undocumented nested "public static" classes -- couldn't find an obvious
way to exclude those using checkstyle.

Here's a "top ten most frequently modified files with missing Javadoc"
below. This is an arbitrary metric, the "most referenced classes" might
be more useful, but that was harder to hack together with shell foo.

solr/core/src/java/org/apache/solr/core/CoreContainer.java
solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
solr/core/src/java/org/apache/solr/handler/StreamHandler.java
solr/core/src/java/org/apache/solr/cloud/ElectionContext.java
solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java
solr/core/src/java/org/apache/solr/search/JoinQParserPlugin.java
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
solr/core/src/java/org/apache/solr/handler/SolrConfigHandler.java

If there's any interest in this, I could write a patch to include
something like this in the build (ant or gradle, whatever).

 - Bram

Following checkstyle configuration detects classes with missing Javadoc:

check.xml:
==

https://checkstyle.org/dtds/configuration_1_3.dtd";>







Bit of shell foo to list offending files:
=

java -jar checkstyle-8.26-all.jar -c config.xml solr/ | cut -d ' ' -f 2
| sed "s:.*/lucene-solr/::g" | cut -d ':' -f 1 | sort | uniq


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SolrCloud is sick.

[Bram Van Dam]

> SolrCloud is sick right now. The way low level Zookeeper is handeled

On an unrelated project, I've stopped using "raw" ZK client access and
have switched to Curator. The API is a fair bit easier to work with, and
it results in less ugly code. I realize that this won't go very far in
resolving more fundamental issues, but it might be something that can
help improve the shape of the code.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Renaming SolrCloud

[Bram Van Dam]

On 20/10/2019 01:32, Noble Paul wrote:
> A standalone mode should be nothing but a SolrCloud setup with an
> embedded ZK and a single node.

From an operations point of view, I disagree. ZK is another process to
monitor, another process which can fail, another process which exposes
various security concerns. Additionally, ZK's Administrator Guide still
insists that its transaction log should be on a separate disk ("A
dedicated partition is not enough"), which is quite frankly a pain in
the arse for smaller setups. I'm not sure whether it's all that
important in standalone mode, but it's certainly something to consider.

 - Bram


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: cwiki write access

[Bram Van Dam]

Looks like I forgot to include that crucial bit of information 😉 My cwiki 
username is bvd

Thanks,

 - Bram



From: David Smiley 
Sent: 13 September 2019 20:05
To: Solr/Lucene Dev 
Subject: Re: cwiki write access

Thanks for offering!
I will give you access but I believe you need to create an account first.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Fri, Sep 13, 2019 at 1:15 PM Bram Van Dam 
mailto:bram.van...@intix.eu>> wrote:
Hey folks,

I'd like to request write access to the Solr cwiki. Mostly so I can
correct the odd typo as I see them.

Thanks,

 - Bram

cwiki write access

[Bram Van Dam]

Hey folks,

I'd like to request write access to the Solr cwiki. Mostly so I can
correct the odd typo as I see them.

Thanks,

 - Bram

Re: Can we change forceMerge to not need as much disk space?

[Bram Van Dam]

On 02/09/2019 17:19, Erick Erickson wrote:
> 4> Don’t quite know what to do if maxSegments is 1 (or other very low number).

Having maxSegments set to > 5 (or whatever) seems like an acceptable
constraint if it enables optimize without 200% disk usage.

> Something like this would also pave the way for “background optimizing”. 
> Instead of a monolithic forceMerge, I can envision a process whereby we 
> created a low-level task that merged one max-sized segment at a time, came up 
> for air and reopened searchers then went back in and merged the next one. 
> With its own problems about coordinating ongoing updates, but that’s another 
> discussion ;).
> 
> There’s lots of details to work out, throwing this out for discussion. I can 
> raise a JIRA if people think the idea has legs.

Without having looked at the code, and going only on your assumptions
and my own observations: it sounds like a good idea. The idea of a
background optimizing process is particularly tantalizing.

AFAICT there hasn't been any other feedback re this? :-/

 - Bram

Re: [ANNOUNCE] Apache Solr 7.7.2 released

[Bram Van Dam]

On 05/06/2019 16:26, Jan Høydahl wrote:
> 4 June 2019, Apache Solr™ 7.7.2 available

Thanks for making this happen, Jan, much appreciated!

 - Bram

Re: [VOTE] Release Lucene/Solr 7.7.2 RC1

[Bram Van Dam]

I don't get a vote, but smoke tests pass for me.

On 29/05/2019 01:54, Jan Høydahl wrote:
> Please vote for release candidate 1 for Lucene/Solr 7.7.2
> 
> The artifacts can be downloaded from:
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-7.7.2-RC1-revd4c30fc2856154f2c1fefc589eb7cd070a415b94
> 
> You can run the smoke tester directly with this command:
> 
> python3 -u dev-tools/scripts/smokeTestRelease.py \
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-7.7.2-RC1-revd4c30fc2856154f2c1fefc589eb7cd070a415b94
> 
> The vote will be open for at least 3 working days, i.e. until 2019-06-04 
> 08:00 UTC.
> 
> [ ] +1  approve
> [ ] +0  no opinion
> [ ] -1  disapprove (and reason why)
> 
> Here is my +1
> 
> SUCCESS! [1:06:20.144701]
> 
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 

Re: 7.7.2?

[Bram Van Dam]

On 29/04/2019 23:33, Jan Høydahl wrote:
> I'll vounteer as RM for 7.7.2 and aim at first RC on Tuesday May 7th

Thank you!

7.7.2?

[Bram Van Dam]

Hey folks,

Was anyone planning on releasing 7.7.2? It seems like there are quite a
few bug fixes, including the one with the high CPU usage.

Full list of fixes so far: SOLR-13414, SOLR-13409, SOLR-13408,
SOLR-13392, SOLR-13355, SOLR-13349, SOLR-13344, SOLR-13285, SOLR-13281,
SOLR-13234, SOLR-13126, SOLR-12860, SOLR-12708, SOLR-12371, SOLR-11876

Feel free to let me know if there's anything I can do to help out with
the process, though I'm not a Solr committer.

Thanks,

 - Bram

Re: We have a cause for high CPU usage on Solr 7.7

[Bram Van Dam]

On 25/03/2019 17:04, Erick Erickson wrote:
> Short form: Apparently there’s a java 8 bug that’s the root cause, not fixed 
> until Java 9: https://bugs.openjdk.java.net/browse/JDK-8129861

Thanks for this, Erick & Adam. This explains a few things I've been
seeing (in my own code as well as sporadically in Solr 7.7.0). Looks
like a couple of other users on the -user list reported similar issues.

It looks like there are already a couple of bugfixes scheduled for 7.7.2
anyway, so I guess someone was already planning a new release?

Anything I can do to help out?

 - Bram

[jira] [Created] (SOLR-13247) Incorrect shard placement during Collection creation

[Bram Van Dam (JIRA)]

Bram Van Dam created SOLR-13247:
---

 Summary: Incorrect shard placement during Collection creation
 Key: SOLR-13247
 URL: https://issues.apache.org/jira/browse/SOLR-13247
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: SolrCloud
Affects Versions: 7.7, 7.6
Reporter: Bram Van Dam


TL;DR; createNodeSet & shards combination is not being respected.

I'm attempting to create a collection with multiple shards, but
apparently the value of createNodeSet is not being respected and shards
are being assigned to nodes seemingly at random.

createNodeSet.shuffle is set to false, so that's not the cause.
Furthermore, sometimes not all nodes in the request are used.

Here's the create request, cleaned up for legibility. Note that the node names
are IP addresses but I've removed the first 3 octets for legibility.


admin/collections
?action=CREATE
&name=collectionName
&router.name=implicit
&shards=collectionName1,collectionName2,collectionName3,collectionName4,collectionName5,collectionName6
&maxShardsPerNode=1024
&collection.configName=some_config
&createNodeSet=171:8180_solr,172:8180_solr,173:8180_solr,177:8180_solr,179:8180_solr,179:8180_solr
&createNodeSet.shuffle=false
&waitForFinalState=true

Note that I'm creating a collection with 6 shards across 5 nodes.

Requested:
collectionName1: 171:8180_solr
collectionName2: 172:8180_solr
collectionName3: 173:8180_solr
collectionName4: 177:8180_solr
collectionName5: 179:8180_solr
collectionName6: 179:8180_solr

Actual:
collectionName1: 177:8180_solr
collectionName2: 172:8180_solr
collectionName3: 179:8180_solr
collectionName4: 173:8180_solr
collectionName5: 171:8180_solr
collectionName6: 171:8180_solr

Not a single shard ends up on the requested node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13136) Queries fail during shard creation [testcase included]

[Bram Van Dam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bram Van Dam updated SOLR-13136:

Issue Type: Bug  (was: Improvement)

> Queries fail during shard creation [testcase included]
> --
>
> Key: SOLR-13136
> URL: https://issues.apache.org/jira/browse/SOLR-13136
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 7.6
>    Reporter: Bram Van Dam
>Priority: Major
> Attachments: TestSearchDuringShardCreation.java
>
>
> Shard creation takes a certain amount of time. Queries launched between the 
> start of shard creation and the full "upness" of the shard will fail: "Error 
> from server at foo: no servers hosting shard: bar".
> Ideally, shards should not be used in queries before their creation is fully 
> completed. A new shard is empty anyway, so its upness couldn't even influence 
> the query results in any way.
> I'm attaching a testcase which reproduces the problem. Tested on two 
> machines, but your mileage may vary assuming this is a concurrency issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13136) Queries fail during shard creation [testcase included]

[Bram Van Dam (JIRA)]

Bram Van Dam created SOLR-13136:
---

 Summary: Queries fail during shard creation [testcase included]
 Key: SOLR-13136
 URL: https://issues.apache.org/jira/browse/SOLR-13136
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: SolrCloud
Affects Versions: 7.6
Reporter: Bram Van Dam
 Attachments: TestSearchDuringShardCreation.java

Shard creation takes a certain amount of time. Queries launched between the 
start of shard creation and the full "upness" of the shard will fail: "Error 
from server at foo: no servers hosting shard: bar".

Ideally, shards should not be used in queries before their creation is fully 
completed. A new shard is empty anyway, so its upness couldn't even influence 
the query results in any way.

I'm attaching a testcase which reproduces the problem. Tested on two machines, 
but your mileage may vary assuming this is a concurrency issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12953) Support for TLS/SSL key alias configuration

[Bram Van Dam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bram Van Dam updated SOLR-12953:

Fix Version/s: 7.6

> Support for TLS/SSL key alias configuration
> ---
>
> Key: SOLR-12953
> URL: https://issues.apache.org/jira/browse/SOLR-12953
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.5
>    Reporter: Bram Van Dam
>Priority: Major
>  Labels: patch
> Fix For: 7.5.1, 7.6
>
> Attachments: SOLR-12953.patch, SOLR-12953.patch
>
>
> As discussed on the mailing list:
> *Context:*
> There's a jetty-ssl.xml config file which configures Jetty's 
> SslContextFactory using properties set in solr.in.sh, but it's incomplete for 
> some purposes.
> *Problem:*
> I've noticed that no "certAlias" property is present. This means that when 
> Jetty starts, it will pick an arbitrary (based on some internal order, 
> apparently the newest?) key from the keystore to use. This is fine when 
> you're only using your keystore for Solr and it only contains one key, but it 
> makes life a lot more complicated in environments where keystores are managed 
> and distributed to servers automagically.
> When you add a key to the keystore, you can assign an alias. Jetty can then 
> use the key with that alias by means of its certAlias config property.
> The Solr documentation [1] confusingly assigns the alias "solr-ssl" to the 
> key, but as far as I can tell this alias isn't actually used or referenced 
> anywhere else. 
> *Solution:*
> I'm currently dealing with a slightly more complicated TLS setup, so I'm 
> attaching a patch which adds an extra config property in order to 
> (optionally) specify the key alias. When the option is omitted, the old 
> behaviour remains unchanged. Patch modifies the configuration and includes 
> updates to the enabling-ssl documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 7.6

[Bram Van Dam]

On 02/11/2018 15:41, Nicholas Knize wrote:
> If needed I can hold off on cutting the 7.6 branch and feature freezing
> until Friday of next week. That would still give at least two weeks of
> jenkins testing & bug fixing before a target release the last week of
> November.

If you're cutting 7.6 soon, could you be so kind as to have a look at
including SOLR-12953?

Thanks!

 - Bram

[jira] [Updated] (SOLR-12953) Support for TLS/SSL key alias configuration

[Bram Van Dam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bram Van Dam updated SOLR-12953:

Attachment: SOLR-12953.patch

> Support for TLS/SSL key alias configuration
> ---
>
> Key: SOLR-12953
> URL: https://issues.apache.org/jira/browse/SOLR-12953
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.5
>    Reporter: Bram Van Dam
>Priority: Major
>  Labels: patch
> Fix For: 7.5.1
>
> Attachments: SOLR-12953.patch, SOLR-12953.patch
>
>
> As discussed on the mailing list:
> *Context:*
> There's a jetty-ssl.xml config file which configures Jetty's 
> SslContextFactory using properties set in solr.in.sh, but it's incomplete for 
> some purposes.
> *Problem:*
> I've noticed that no "certAlias" property is present. This means that when 
> Jetty starts, it will pick an arbitrary (based on some internal order, 
> apparently the newest?) key from the keystore to use. This is fine when 
> you're only using your keystore for Solr and it only contains one key, but it 
> makes life a lot more complicated in environments where keystores are managed 
> and distributed to servers automagically.
> When you add a key to the keystore, you can assign an alias. Jetty can then 
> use the key with that alias by means of its certAlias config property.
> The Solr documentation [1] confusingly assigns the alias "solr-ssl" to the 
> key, but as far as I can tell this alias isn't actually used or referenced 
> anywhere else. 
> *Solution:*
> I'm currently dealing with a slightly more complicated TLS setup, so I'm 
> attaching a patch which adds an extra config property in order to 
> (optionally) specify the key alias. When the option is omitted, the old 
> behaviour remains unchanged. Patch modifies the configuration and includes 
> updates to the enabling-ssl documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr TLS/SSL key alias configuration (with patch to come)

[Bram Van Dam]

On 31/10/2018 15:51, Erick Erickson wrote:
> There's no reason I can imagine not to open a JIRA,
> basically anyone willing to create a patch has my vote!

Done: SOLR-12953 including a patch. Decided to do this on the 7_5
branch. Merges cleanly to master as well, but we're stickign to 7.5 for
the time being.

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12953) Support for TLS/SSL key alias configuration

[Bram Van Dam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bram Van Dam updated SOLR-12953:

Description: 
As discussed on the mailing list:

*Context:*
There's a jetty-ssl.xml config file which configures Jetty's SslContextFactory 
using properties set in solr.in.sh, but it's incomplete for some purposes.

*Problem:*
I've noticed that no "certAlias" property is present. This means that when 
Jetty starts, it will pick an arbitrary (based on some internal order, 
apparently the newest?) key from the keystore to use. This is fine when you're 
only using your keystore for Solr and it only contains one key, but it makes 
life a lot more complicated in environments where keystores are managed and 
distributed to servers automagically.

When you add a key to the keystore, you can assign an alias. Jetty can then use 
the key with that alias by means of its certAlias config property.

The Solr documentation [1] confusingly assigns the alias "solr-ssl" to the key, 
but as far as I can tell this alias isn't actually used or referenced anywhere 
else. 

*Solution:*
I'm currently dealing with a slightly more complicated TLS setup, so I'm 
attaching a patch which adds an extra config property in order to (optionally) 
specify the key alias. When the option is omitted, the old behaviour remains 
unchanged. Patch modifies the configuration and includes updates to the 
enabling-ssl documentation.


  was:
As discussed on the mailing list:

*Context:*
There's a jetty-ssl.xml config file which configures Jetty's SslContextFactory 
using properties set in solr.in.sh, but it's incomplete for some purposes.

*Problem:*
I've noticed that no "certAlias" property is present. This means that when 
Jetty starts, it will pick an arbitrary (based on some internal order, 
apparently the newest?) key from the keystore to use. This is fine when you're 
only using your keystore for Solr and it only contains one key, but it makes 
life a lot more complicated in environments where keystores are managed and 
distributed to servers automagically.

When you add a key to the keystore, you can assign an alias. Jetty can then use 
the key with that alias by means of its certAlias config property.

The Solr documentation [1] confusingly assigns the alias "solr-ssl" to the key, 
but as far as I can tell this alias isn't actually used or referenced anywhere 
else. 

*Solution:*
I'm currently dealing with a slightly more complicated TLS setup, so I'm 
attaching a patch which adds an extra config property in order to (optionally) 
specify the key alias. When the option is omitted, the old behaviour remains 
unchanged.



> Support for TLS/SSL key alias configuration
> ---
>
> Key: SOLR-12953
> URL: https://issues.apache.org/jira/browse/SOLR-12953
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.5
>Reporter: Bram Van Dam
>Priority: Major
>  Labels: patch
> Fix For: 7.5.1
>
> Attachments: SOLR-12953.patch
>
>
> As discussed on the mailing list:
> *Context:*
> There's a jetty-ssl.xml config file which configures Jetty's 
> SslContextFactory using properties set in solr.in.sh, but it's incomplete for 
> some purposes.
> *Problem:*
> I've noticed that no "certAlias" property is present. This means that when 
> Jetty starts, it will pick an arbitrary (based on some internal order, 
> apparently the newest?) key from the keystore to use. This is fine when 
> you're only using your keystore for Solr and it only contains one key, but it 
> makes life a lot more complicated in environments where keystores are managed 
> and distributed to servers automagically.
> When you add a key to the keystore, you can assign an alias. Jetty can then 
> use the key with that alias by means of its certAlias config property.
> The Solr documentation [1] confusingly assigns the alias "solr-ssl" to the 
> key, but as far as I can tell this alias isn't actually used or referenced 
> anywhere else. 
> *Solution:*
> I'm currently dealing with a slightly more complicated TLS setup, so I'm 
> attaching a patch which adds an extra config property in order to 
> (optionally) specify the key alias. When the option is omitted, the old 
> behaviour remains unchanged. Patch modifies the configuration and includes 
> updates to the enabling-ssl documentation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12953) Support for TLS/SSL key alias configuration

[Bram Van Dam (JIRA)]

Bram Van Dam created SOLR-12953:
---

 Summary: Support for TLS/SSL key alias configuration
 Key: SOLR-12953
 URL: https://issues.apache.org/jira/browse/SOLR-12953
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 7.5
Reporter: Bram Van Dam
 Fix For: 7.5.1


As discussed on the mailing list:

*Context:*
There's a jetty-ssl.xml config file which configures Jetty's SslContextFactory 
using properties set in solr.in.sh, but it's incomplete for some purposes.

*Problem:*
I've noticed that no "certAlias" property is present. This means that when 
Jetty starts, it will pick an arbitrary (based on some internal order, 
apparently the newest?) key from the keystore to use. This is fine when you're 
only using your keystore for Solr and it only contains one key, but it makes 
life a lot more complicated in environments where keystores are managed and 
distributed to servers automagically.

When you add a key to the keystore, you can assign an alias. Jetty can then use 
the key with that alias by means of its certAlias config property.

The Solr documentation [1] confusingly assigns the alias "solr-ssl" to the key, 
but as far as I can tell this alias isn't actually used or referenced anywhere 
else. 

*Solution:*
I'm currently dealing with a slightly more complicated TLS setup, so I'm 
attaching a patch which adds an extra config property in order to (optionally) 
specify the key alias. When the option is omitted, the old behaviour remains 
unchanged.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr TLS/SSL key alias configuration (with patch to come)

[Bram Van Dam]

On 31/10/2018 15:51, Erick Erickson wrote:
> Probably just add a note to the upgrade section of CHANGES.txt,
> unless others disagree. I confess knowing very little about the
> mechanics here.

Thanks for the quick feedback. Will try to do this tomorrow.

> BTW, don't know if you're familiar with asciidoc but in case not I
> wanted to mention that there's an IntelliJ (and, I assume Eclipse)
> plugin showing you the rendering, and you can also use Atom.

vim has wonderful asciidoc support as well :-)

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr TLS/SSL key alias configuration (with patch to come)

[Bram Van Dam]

Hey folks,

Context:
There's a jetty-ssl.xml config file which configures Jetty's
SslContextFactory using properties set in solr.in.sh, but it's
incomplete for some purposes.

Problem:
I've noticed that no "certAlias" property is present. This means that
when Jetty starts, it will pick an arbitrary (based on some internal
order, I guess?) key from the keystore to use. This is fine when you're
only using your keystore for Solr and it only contains one key, but it
makes life a lot more complicated in environments where keystores are
managed and distributed to servers automagically.

When you add a key to the keystore, you can assign an alias. Jetty can
then use the key with that alias by means of its certAlias config property.

The Solr documentation [1] confusingly assigns the alias "solr-ssl" to
the key, but as far as I can tell this alias isn't actually used or
referenced anywhere else.

Solution:
I'm currently dealing with a slightly more complicated TLS setup, so I
propose I patch jetty-ssl.xml, solr.in.sh|cmd and enabling-ssl.adoc to
(optionally) use the alias? Unless someone can think of a reason why I
shouldn't do this?

I'm a bit worried that adding certAlias to jetty-ssl.xml might break
existing setups which don't use an alias, but I'm guessing that only
keystores with more than one key will be affected?

 - Bram

[1] https://lucene.apache.org/solr/guide/7_5/enabling-ssl.html

Re: [RESULT][VOTE] Release Lucene/Solr 5.5.5 RC2

[Bram Van Dam]

On 23/10/17 17:43, Steve Rowe wrote:
> I’ll go do the release now.

Thanks! I missed the announcement, but I just noticed 5.5.5 appear in
the changelogs. Much appreciated.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr JSON Facet API & aggregation functions documentation

[Bram Van Dam]

On 19/06/17 17:38, Erick Erickson wrote:
> We've recently gone to AsciiDoc which is much easier to edit IMO, and
> also allows anyone (hint hint) to attach a patch to a JIRA for the
> documentation that makes it into the next version of reference guide
> as well as the current online version.

Hint taken, I'll see what I can do. I have to document some of the
functions (hll and others) for use in our application anyway, so I might
as well see if I can take it a bit further.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr JSON Facet API & aggregation functions documentation

[Bram Van Dam]

Hey folks,

Yonik's blog [1] contains a list of aggregation functions supported by
the JSON Facet API (the json.facet parameter) butI can't seem to find
documentation for these on the cwiki.

There's an old TODO list on the cwiki which mentions that this still
needs to happen. [2]

Is anyone working on this? Do I simply suck at searching the cwiki? Or
can we kindly borrow Yonik's blog post and use its contents for
documentation?

[1] http://yonik.com/solr-facet-functions/
[2] https://cwiki.apache.org/confluence/display/solr/Internal+-+TODO+List

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Stop using SHA-1 and MD5 hashes?

[Bram Van Dam]

> From what I can see, hashes and signatures are both missing on the
> download mirrors for Lucene and Solr.  That's probably prudent for
> hashes, but should signatures be there?

I vaguely remembering raising this issue before -- though it might have
been regarding a different Apache project. From what I remember, the ASF
signature guidelines don't require software signing keys to be signed by
anyone in particular. So unless the signature file is on the (https)
Apache download site, it's probably effectively useless.

After all there's nothing stopping me from setting up a rogue mirror,
creating a "Shawn Heisey " GPG key and signing my
fake release with it.

Including signatures on mirrors would only lead to sloppy verification
by whoever is downloading the software.

That is, unless there's some kind of web of trust in the release
signature, but that currently doesn't seem to be the case.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [ANNOUNCE] Apache Lucene 5.5.3 released

[Bram Van Dam]

> 09 September 2016, Apache Lucene™ 5.5.3 available

Thanks for creating the release, Anshum!

 - Bram


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9035) New cwiki page: IndexUpgrader

[Bram Van Dam (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15336032#comment-15336032
 ] 

Bram Van Dam commented on SOLR-9035:


Thanks for creating it.

What else is missing to create a dedicated upgrade section?

> New cwiki page: IndexUpgrader
> -
>
> Key: SOLR-9035
> URL: https://issues.apache.org/jira/browse/SOLR-9035
> Project: Solr
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 6.0
>Reporter: Bram Van Dam
>Assignee: Cassandra Targett
>  Labels: documentation
> Attachments: indexupgrader.html
>
>
> The cwiki does not contain any IndexUpgrader documentation, but it is 
> mentioned in passing in the "Major Changes"-pages.
> I'm attaching a file containing some basic usage instructions and adminitions 
> found in the IndexUpgrader javadoc. 
> Once the page is created, it would ideally be linked to from the Major 
> Changes page as well as the Upgrading Solr page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [ANNOUNCE] Apache Lucene 5.5.1 released

[Bram Van Dam]

On 06/05/16 08:46, Anshum Gupta wrote:
> 5 May 2016, Apache Lucene™ 5.5.1 available

Thanks for the hard work, Anshum!

- Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Schedule merges during off-peak hours?

[Bram Van Dam]

On 02/05/16 16:04, Ryan Josal wrote:
> I don't have a comment on whether or not it makes sense to have schedule
> based merges, but I will say that search speed is impacted by segment
> count, and if you're indexing constantly all day long, it's possible
> that saving a merge until off peak hours may put more stress on your
> servers than doing the merge.  Also, are you merging down to one
> segment?  I found it to be a happy medium to merge down to 2 or 3 segments.

You're probably right about the stress. Might be worth merging down to 3
segments during peak hours and then down to 1 during off peak hours or
something like that.

Thanks for the feedback!


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Can someone create this wiki page?

[Bram Van Dam]

Could someone take a look at SOLR-9035 and either create the cwiki page
or give me some feedback on the contents of the page?

I'd do it myself but I don't have commit access :-)

Thanks a bunch,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Schedule merges during off-peak hours?

[Bram Van Dam]

On 02/05/16 06:45, Erick Erickson wrote:
> If "commit intervals of around one minute" mean that you're taking 60
> seconds for the commit to finish, then perhaps it's the opposite and
> you're autowarming too much. I usually start my autowarm counts
> around 16 or so for filterCache and queryResultCache...

Hey Erick,

I meant that a commit occurs every minute. Autowarming is disabled
entirely, because with very frequent commits, searchers sometimes aren't
used at all before another commit happens.

 - Bram


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Schedule merges during off-peak hours?

[Bram Van Dam]

Hey folks,

Disclaimer: I haven't given this too much thought, so feel free to shoot
me down if this is a stupid idea.

We have a lot of Solr instances with a lot of different customers, and
most of them fit into a patterb of 9-5 searching and slow but steady
addition of documents 24/7, at a rate of about 10docs/second.

This means there's hardly any searching going on at night. This also
means that merges during the day can be a problem when indexes grow
large, especially with commit intervals of around one minute. This seems
wasteful, because the index doesn't grow *that* much larger during a
single working day.

I was wondering if it would be a good idea to create a MergePolicy that
only merges segments during off-peak hours (or maybe according to a
cron-style pattern or something).

Any thoughts?

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9051) Read Solr 4.10 indexes from Solr 6.x

[Bram Van Dam (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265289#comment-15265289
 ] 

Bram Van Dam commented on SOLR-9051:


[~ysee...@gmail.com] Are you considering rewriting the index? Or are you merely 
considering read-only?

> Read Solr 4.10 indexes from Solr 6.x
> 
>
> Key: SOLR-9051
> URL: https://issues.apache.org/jira/browse/SOLR-9051
> Project: Solr
>  Issue Type: Improvement
>Reporter: Yonik Seeley
>
> This is something I need to look into as part of my $day_job anyway,
> and I've heard that others are interested in this.  There are a lot of people 
> on 4.x (esp 4.10), and providing them an index upgrade path that doesn't 
> involve going through 5x would be nice.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9046) solr.cmd wrongly assumes Jetty will always listen on 0.0.0.0

[Bram Van Dam (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15261703#comment-15261703
 ] 

Bram Van Dam commented on SOLR-9046:


Thanks for these fixes and the commit. Just tested your changes on our setup 
and everything seems to work. 

Windows's netstat doesn't have an option to only display listening processes, 
which is why I added the 'find ":0 "'. Without that extra constraint, the stop 
process would try to shoot down internet explorer when I was using the admin UI 
:-) Whoops!

> solr.cmd wrongly assumes Jetty will always listen on 0.0.0.0
> 
>
> Key: SOLR-9046
> URL: https://issues.apache.org/jira/browse/SOLR-9046
> Project: Solr
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 5.5
> Environment: Windows
>Reporter: Bram Van Dam
>Assignee: Uwe Schindler
> Fix For: master, 5.5.1, 6.1
>
> Attachments: SOLR-9045.patch, SOLR-9046.patch, SOLR-9046.patch, 
> SOLR-9046.patch, SOLR-9046.patch
>
>
> The Windows solr.cmd script makes the (incorrect) assumption that Solr will 
> always be listening on 0.0.0.0 (all interfaces). When you change the interface
> address, say to 127.0.0.1, then the status and stop commands will fail.
> This patch adds a property in solr.in.cmd, which is passed to SOLR_OPTS as 
> -Djetty.host, and replaces the instances of 0.0.0.0 in solr.cmd.
> The patch includes some changes in the netstat logic used in solr.cmd to find 
> the correct Solr process(es). 
> Tested on Solr 5.5 on Windows 7 and 10. 
> Note: Untested on Solr 6. Currently using Solr 5.5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 5.5.1

[Bram Van Dam]

Hey Anshum,

Can you have a look at SOLR-9046 as well? It's something that's been
bugging me in 5.5. It's fine if it doesn't make it to 5.5.1, but it
would be really nice if it did!

Thx,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9046) solr.cmd wrongly assumes Jetty will always listen on 0.0.0.0

[Bram Van Dam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bram Van Dam updated SOLR-9046:
---
Attachment: SOLR-9045.patch

> solr.cmd wrongly assumes Jetty will always listen on 0.0.0.0
> 
>
> Key: SOLR-9046
> URL: https://issues.apache.org/jira/browse/SOLR-9046
> Project: Solr
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 5.5
> Environment: Windows
>    Reporter: Bram Van Dam
> Fix For: 5.5.1
>
> Attachments: SOLR-9045.patch
>
>
> The Windows solr.cmd script makes the (incorrect) assumption that Solr will 
> always be listening on 0.0.0.0 (all interfaces). When you change the interface
> address, say to 127.0.0.1, then the status and stop commands will fail.
> This patch adds a property in solr.in.cmd, which is passed to SOLR_OPTS as 
> -Djetty.host, and replaces the instances of 0.0.0.0 in solr.cmd.
> The patch includes some changes in the netstat logic used in solr.cmd to find 
> the correct Solr process(es). 
> Tested on Solr 5.5 on Windows 7 and 10. 
> Note: Untested on Solr 6. Currently using Solr 5.5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9046) solr.cmd wrongly assumes Jetty will always listen on 0.0.0.0

[Bram Van Dam (JIRA)]

Bram Van Dam created SOLR-9046:
--

 Summary: solr.cmd wrongly assumes Jetty will always listen on 
0.0.0.0
 Key: SOLR-9046
 URL: https://issues.apache.org/jira/browse/SOLR-9046
 Project: Solr
  Issue Type: Bug
  Components: Server
Affects Versions: 5.5
 Environment: Windows
Reporter: Bram Van Dam
 Fix For: 5.5.1


The Windows solr.cmd script makes the (incorrect) assumption that Solr will 
always be listening on 0.0.0.0 (all interfaces). When you change the interface
address, say to 127.0.0.1, then the status and stop commands will fail.

This patch adds a property in solr.in.cmd, which is passed to SOLR_OPTS as 
-Djetty.host, and replaces the instances of 0.0.0.0 in solr.cmd.

The patch includes some changes in the netstat logic used in solr.cmd to find 
the correct Solr process(es). 

Tested on Solr 5.5 on Windows 7 and 10. 

Note: Untested on Solr 6. Currently using Solr 5.5



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr - Windows solr.cmd & network interfaces: weird behaviour

[Bram Van Dam]

Hi folks,

Just noticed some weird behaviour on Windows. The Windows solr.cmd
script makes the (incorrect) assumption that Solr will always be
listening on 0.0.0.0 (all interfaces). When you change the interface
address, say to 127.0.0.1, then the status and stop commands will fail.

I changed the interface address by updating solr.in.cmd:

set SOLR_OPTS=%SOLR_OPTS% -Djetty.host=127.0.0.1

I'm thinking that 0.0.0.0 should probably be a parameter defined in
solr.in.cmd, instead of a hard coded magic number in solr.cmd.

Would that be an acceptable approach?

If so, I'm willing to write up a patch, but I think there are/were plans
of rewriting solr.cmd entirely and I wouldn't want to get in the way of
that?

 - Bram



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Upgrading Solr - Wiki

[Bram Van Dam]

On 21/04/16 14:05, Steve Rowe wrote:
> Docs welcome!

I've created a JIRA issue with the suggested content for an
IndexUpgrader page. If someone can create the page, I'll leave comments
on the other relevant pages so they can point to the new page.

https://issues.apache.org/jira/browse/SOLR-9035

Thanks,

 - Bram


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9035) New cwiki page: IndexUpgrader

[Bram Van Dam (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bram Van Dam updated SOLR-9035:
---
Attachment: indexupgrader.html

> New cwiki page: IndexUpgrader
> -
>
> Key: SOLR-9035
> URL: https://issues.apache.org/jira/browse/SOLR-9035
> Project: Solr
>  Issue Type: Improvement
>  Components: documentation
>Affects Versions: 6.0
>    Reporter: Bram Van Dam
>  Labels: documentation
> Attachments: indexupgrader.html
>
>
> The cwiki does not contain any IndexUpgrader documentation, but it is 
> mentioned in passing in the "Major Changes"-pages.
> I'm attaching a file containing some basic usage instructions and adminitions 
> found in the IndexUpgrader javadoc. 
> Once the page is created, it would ideally be linked to from the Major 
> Changes page as well as the Upgrading Solr page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9035) New cwiki page: IndexUpgrader

[Bram Van Dam (JIRA)]

Bram Van Dam created SOLR-9035:
--

 Summary: New cwiki page: IndexUpgrader
 Key: SOLR-9035
 URL: https://issues.apache.org/jira/browse/SOLR-9035
 Project: Solr
  Issue Type: Improvement
  Components: documentation
Affects Versions: 6.0
Reporter: Bram Van Dam


The cwiki does not contain any IndexUpgrader documentation, but it is mentioned 
in passing in the "Major Changes"-pages.

I'm attaching a file containing some basic usage instructions and adminitions 
found in the IndexUpgrader javadoc. 

Once the page is created, it would ideally be linked to from the Major Changes 
page as well as the Upgrading Solr page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Anybody seeing Jira issue: unable to add comments

[Bram Van Dam]

On 22/04/16 11:19, Uwe Schindler wrote:
> as a temporary workaround I added you to the issues.apache.org "contributors" 
> role in Lucene and Solr projects.

Could someone grant me the create issue permission as well, please?
Username is bvd. Was trying to create a new issue with some new
documentation.

Thanks a bunch,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Upgrading Solr - Wiki

[Bram Van Dam]

On 21/04/16 14:05, Steve Rowe wrote:
> I found mention of IndexUpgrader only on 
> ,
>  with no actual instructions anywhere else on the ref guide.  It’s not clear 
> to me exactly which page would host such info, maybe 
>  or 
> ?

I was considering creating an IndexUpgrader page with basic usage
instructions and maybe some information about how long it takes to
upgrade indexes of various sizes. The Upgrading Solr page and Major
Changes pages could then link to it.

> Please include your suggested documentation in a comment on one of the 
> above-linked upgrade-related ref guide pages.

Will do as soon as I'm done crunching numbers and making sure my
instructions actually work :-)

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Upgrading Solr - Wiki

[Bram Van Dam]

> However, it appears I don't have write access to the wiki. How do I go
> about getting write access? Or should I just type up the documentation
> and send to this mailing list?

I should probably clarify that I'm talking about the Confluence Wiki and
that my username is bvd.

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Upgrading Solr - Wiki

[Bram Van Dam]

Hey folks,

I wanted to update the Upgrading Solr wiki page to include a section
about the usage of the IndexUpgrader tool, as its usage wasn't
immediately obvious to me.

However, it appears I don't have write access to the wiki. How do I go
about getting write access? Or should I just type up the documentation
and send to this mailing list?

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 5.5.1

[Bram Van Dam]

On 13/04/16 19:26, Anshum Gupta wrote:
> Can you please test it with 5.5.1 so someone could back port it. Thanks.

I've finished testing SOLR-8145 on branch_5_5. Patch applies cleanly and
the bugfix works fine. I don't have commit access, so someone else will
have to take care of that part.

git cherry-pick 80801a2 does the trick.

Thanks,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 5.5.1

[Bram Van Dam]

On 11/04/16 19:04, Ishan Chattopadhyaya wrote:
> I'd like someone to backport the following:
> SOLR-8838
> SOLR-8082
> SOLR-8865

Can we please add SOLR-8145 to that list? It fixes an incorrect
oom_killer argument in the startupt script and a Jetty warning on startup.

Please let me know if you'd like me to help test 5.5.1.

 - Bram



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Preferred Development Branch?

[Bram Van Dam]

Hi folks,

If I wanted to make some changes that I'd like to see in 5.x, should I branch 
off 5.x and work on that? Or is it preferable to branch trunk and then backport 
to 5.x?

Thanks!

 - Bram
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Prepare for Release of Lucene/Solr 4.8!

[Bram Van Dam]

On 04/09/2014 04:19 PM, Erick Erickson wrote:

I was hoping to get the analytics stuff in place for 4.8, but I think
it would be a Bad Idea (tm) to try to add this much new code (with the
dozen or so merges I need to reconcile) in at the last minute, even
though it doesn't touch much existing code.


I was hoping to see the new analytics stuff in 4.8 as well, that's a 
shame. Would also still like to get SOLR-5789 in. But then I guess 
there's always a next release :-)


 - Bram


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [GitHub] lucene-solr pull request: SOLR-5789 Add min/max modifiers to Atomi...

[Bram Van Dam]

Could someone be so kind as to have a look at this PR? It's a feature 
that would make my life substantially easier :-) Thx!


On 02/27/2014 05:25 PM, Bram Van Dam wrote:

GitHub user codematters opened a pull request:
 https://github.com/apache/lucene-solr/pull/39


Discussion and tips for improvements are welcome!



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [GitHub] lucene-solr pull request: SOLR-5789 Add min/max modifiers to Atomi...

[Bram Van Dam]

GitHub user codematters opened a pull request:
 https://github.com/apache/lucene-solr/pull/39


Discussion and tips for improvements are welcome!

Thanks




-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Patches in Jira or pull requests on github?

[Bram Van Dam]

Opening a pull request will send a notification to the mailing list, so
that should get noticed the same as opening a JIRA.


Excellent, thanks!

Will also have a closer look at the developer tips.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Patches in Jira or pull requests on github?

[Bram Van Dam]

Hi folks,

What's the preferred way of contributing a patch?

The easiest from my perspective is a pull request on github (and it 
looks like there's a couple of pull requests already on there).


Should I create a jira issue with an attachment or a link to the pull 
request? Anything else I should know about?


Thx,

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: The Old Git Discussion

[Bram Van Dam]

On 01/03/2014 12:13 AM, Shawn Heisey wrote:

I can't make any cogent arguments about whether git is superior to
subversion at the basic job of version control, whether it produces
better workflows, or any similar argument. Generally speaking, I do
think it's a good idea for projects like Apache to eat their own
dogfood, which currently we do because we use subversion.


Regardless of preference, experience, dogfooding or workflows, git makes 
merging (branches and patches) a lot easier than SVN. For that reason 
alone I switched all of my repos from CVS/SVN to git a long time ago.


All the other arguments (like workflow, data integrity, tool support 
with things like gitlab) are secondary for me, but do represent a nice 
cherry on top.


There are some things that take a bit of getting used to, if you're a 
die-hard SVN/CVS user, but it's a hurdle that's well worth taking imo.


As for the "crappy license" someone mentioned .. that's a philosophical 
discussion at best. One that's easy to get carried away in and is 
ultimately irrelevant unless you're modifying git for your own project 
(which I very much hope you're not).


Just my two cents.

 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR-5302, Analytics

[Bram Van Dam]

See, if people don't raise objections, I can spread the blame if things
go wrong :)


My opinion isn't particularly important, but I've been keeping an eye on 
this feature for a while. The devs seemed pretty confident about the 
status last I checked.


It's an isolated feature with little (no?) impact on the rest of the 
code, so I say go for it.


*walks away quietly*

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Pivot Faceting with distinct counts

[Bram Van Dam]

On 10/22/2013 02:25 PM, Andrew Psaltis wrote:

1. Getting distinct counts for a field other than the one being faceted on.
2. How would I get this field to surface in the response.


If I'm understanding you correctly, you're using pivots as SQL-speak 
GROUP BY and would like to add aggregate functions (SUM, COUNT, ..).


This is currently not possible. The way we accomplish this is by relying 
on the StatsComponent and its one-dimensional faceting (thankfully we 
only need sums ..). This is tedious and slow, as it requires many 
queries instead of (ideally) 1 query.


SOLR-5302 is probably your best bet. The people working on SOLR-3583 
might be able to have a few things to say about this as well. I will say 
this: the faceting code is pretty scary. Adding support for (what is 
essentially) multi-dimensional pivoting with aggregation is a pretty big 
challenge. My original plan was to start work on this myself, but time 
constraints do not permit that. We would probably be able and willing to 
sponsor someone to work on this.


 - Bram

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org