Significant terms expression giving error "length needs to be >= 1"

[ufuk yılmaz]

We have a SolrCloud cluster, version 8.4

At the customer’s site there’s a collection with very few documents, around 12. 
We usually have collections with hundreds of millions of documents, so that 
collection is a bit of an exception.

When I send a significantTerms streaming expression it immediately gets a 
“IllegalArgumentException("length needs to be >= 1")” from that collection’s 
shard. I took a look at it, but it doesn’t seem to have anything different in 
it from other collections. We also don’t get that exception in our own cluster, 
which is very similar to customer’s.

I found the exception log in 
“lucene-solr\lucene\core\src\java\org\apache\lucene\util\SparseFixedBitSet.java”
 but I don’t have enough knowlegde on the inner workings of the streaming 
expression to interpret it.

What may cause this?

--ufuk

Sent from Mail for Windows 10

Your ZK connection string (3 hosts) is different

[Gell-Holleron, Daniel]

Hello there,

I've installed Solr 8.7.0 on 3 servers that are using 3 ZooKeeper servers, 
which are running on version 3.6.2. These are all running on a Windows OS.

After running upconfigs to the ZooKeeper servers and starting Solr up, it all 
seems to be running ok.

However I see an issue when I go to the ZK Status page and see the message 
"Errors: Your ZK connection string (3 hosts) is different from the dynamic 
ensemble config (3 hosts). Solr does not currently support dynamic 
configuration and will only be able to connect to the zk hosts in your 
connection string.

I saw on JIRA, the bug 
SOLR-14371. This was fixed in 
version 8.6.0 by the looks of it.

Is there a reason why I still see this message as I'm on a version where this 
has been fixed?

In my solr.in.cmd file, I have the line 'SET 
ZK_HOST=zookeeper1:2181,zookeeper2:2181,zookeeper3:2181'

In my zoo.cfg I have the following:

tickTime=2000
initLimit=5
syncLimit=2
dataDir=s:\\apps\zookeeper\\data\\1
clientPort=2181
server.1=zookeeper1:2888:3888
server.2=zookeeper2:2888:3888
server.3=zookeeper3:2888:3888
4lw.commands.whitelist=mntr,conf,ruok
admin.serverPort=8080

Any help would be greatly appreciated.

Thanks,

Daniel

Re: SolrJ: SolrInputDocument.addField()

[Steven White]

Hi Shawn,

Yes, I have managed schema enabled like so:

  
true
cp-schema.xml
  

The reason why I enabled it is so that I can dynamically customize the
schema based on what's in the DB.  So that I can add fields to the schema
dynamically.

I didn't know about the field "guessing" part.  Now that I know I see this
in my solrconfig.xml file:

  



  

If I remove this block, what will happen?

I guess a better question, to meet my need, is this: how do I tell Solr, in
schema-less mode, to use *my* defined field-type whenever it needs to
create a new field?

I'm on Solr 8.6.1 and the link at
https://lucene.apache.org/solr/guide/8_6/schema-factory-definition-in-solrconfig.html#schema-factory-definition-in-solrconfig
doesn't offer much help.

Thanks

Steven


On Mon, Feb 15, 2021 at 11:09 AM Shawn Heisey  wrote:

> On 2/15/2021 6:52 AM, Steven White wrote:
> > It looks to me that SolrInputDocument.addField() is either missnamed or
> > isn't well implemented.
> >
> > When it is called on a field that doesn't exist in the schema, it will
> > create that field and give it a type based on the data.  Not only that,
> it
> > will set default values.  For example, this call
> >
> >  SolrInputDocument doc = new SolrInputDocument();
> >  doc.addField("Company", "ACM company");
> >
> > Will create the following:
> >
> >  
> >  
>
> That SolrJ code does not make those changes to your schema.  At least
> not in the way you're thinking.
>
> It sounds to me like your solrconfig.xml includes what we call
> "schemaless mode" -- an update processor that adds unknown fields when
> they are indexed.  You should disable it.  We strongly recommend never
> using it in production, because it can make the wrong guess about which
> fieldType is required.  The fieldType chosen has very little to do with
> the SolrJ code.  It is controlled by what's in solrconfig.xml.
>
> Thanks,
> Shawn
>

Re: SolrJ: SolrInputDocument.addField()

[Shawn Heisey]

On 2/15/2021 6:52 AM, Steven White wrote:

It looks to me that SolrInputDocument.addField() is either missnamed or
isn't well implemented.

When it is called on a field that doesn't exist in the schema, it will
create that field and give it a type based on the data.  Not only that, it
will set default values.  For example, this call

 SolrInputDocument doc = new SolrInputDocument();
 doc.addField("Company", "ACM company");

Will create the following:

 
 


That SolrJ code does not make those changes to your schema.  At least 
not in the way you're thinking.


It sounds to me like your solrconfig.xml includes what we call 
"schemaless mode" -- an update processor that adds unknown fields when 
they are indexed.  You should disable it.  We strongly recommend never 
using it in production, because it can make the wrong guess about which 
fieldType is required.  The fieldType chosen has very little to do with 
the SolrJ code.  It is controlled by what's in solrconfig.xml.


Thanks,
Shawn

Re: SolrJ: SolrInputDocument.addField()

[Steven White]

Thanks Shawn.

It looks to me that SolrInputDocument.addField() is either missnamed or
isn't well implemented.

When it is called on a field that doesn't exist in the schema, it will
create that field and give it a type based on the data.  Not only that, it
will set default values.  For example, this call

SolrInputDocument doc = new SolrInputDocument();
doc.addField("Company", "ACM company");

Will create the following:




Since this is happening without the caller knowing it (this is not
documented) it leads to search issues (the intended analyzer will not be
used to name one).

It looks to me that the only way I can fix this is to first create the
field type first and then call addField() passing it the field I
just created.  To that end, I cannot find a SolrJ API to create a field
type.  Is that the case?

Steven.


On Sun, Feb 14, 2021 at 7:17 PM Shawn Heisey  wrote:

> On 2/14/2021 9:00 AM, Steven White wrote:
> > It looks like I'm misusing SolrJ API  SolrInputDocument.addField() thus I
> > need clarification.
> >
> > Here is an example of what I have in my code:
> >
> >  SolrInputDocument doc = new SolrInputDocument();
> >  doc.addField("MyFieldOne", "some data");
> >  doc.addField("MyFieldTwo", 100);
> >
> > The above code is creating 2 fields for me (if they don't exist already)
> > and then indexing the data to those fields.  The data is "some data" and
> > the number 100  However, when the field is created, it is not using the
> > field type that I custom created in my schema.  My question is, how do I
> > tell addField() to use my custom field type?
>
> There is no way in SolrJ code to control which fieldType is used.  That
> is controlled solely by the server-side schema definition.
>
> How do you know that Solr is not using the correct fieldType?  If you
> are looking at the documents returned by a search and aren't seeing the
> transformations described in the schema, you're looking in the wrong place.
>
> Solr search results always returns what was originally sent in for
> indexing.  Only Update Processors (defined in solrconfig.xml, not the
> schema) can affect what gets returned in results, fieldType definitions
> NEVER affect data returned in search results.
>
> Thanks,
> Shawn
>

Re: Elevation in dataDir in Solr Cloud

[Mónica Marrero]

Of course, here is the full stack trace (collection 'techproducts' with
just one core to make it easier):

org.apache.solr.common.SolrException: Unable to reload core
[techproducts2_shard1_replica_n1]
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1472)
at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$2(CoreAdminOperation.java:131)
at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:396)
at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at 
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:502)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at 
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at 
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error initializing
QueryElevationComponent
at org.apache.solr.core.SolrCore.(SolrCore.java:1048)
at org.apache.solr.core.SolrCore.reload(SolrCore.java:666)
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1448)
... 44 more
Caused by: org.apache.solr.common.SolrException: Error initializing
QueryElevationComponent
at 
org.apache.solr.handler.component.QueryElevationComponent.handleInitializationException(QueryElevationComponent.java:277)
at 
org.apache.solr.handler.component.QueryElevationComponent.inform(QueryElevationComponent.java:165)