[jira] [Commented] (SOLR-5513) automate SOLR core creation when installation is done...
[ https://issues.apache.org/jira/browse/SOLR-5513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834627#comment-13834627 ] Noble Paul commented on SOLR-5513: -- Please give more details as what exactly is the use case. automate SOLR core creation when installation is done... Key: SOLR-5513 URL: https://issues.apache.org/jira/browse/SOLR-5513 Project: Solr Issue Type: New Feature Components: clients - C# Environment: Windows,Ubuntu Reporter: Deep Kumar Lakharia Labels: build Fix For: 4.3, 4.4 Original Estimate: 24h Remaining Estimate: 24h Automate SOLR core creation when installation is done...as of now we are creating the core manually -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5216) Document updates to SolrCloud can cause a distributed deadlock.
[ https://issues.apache.org/jira/browse/SOLR-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834628#comment-13834628 ] Ricardo Merizalde commented on SOLR-5216: - Mark, can this issue affect SolrCloud deployments with a single shard? We've been running SolrCloud since April and we experienced an odd outage today we've never seen before. We are currently running Solr 4.5.1 with 4 slaves and we use CloudSolrServer to send updates. The number of threads went from under 100 to almost 400 in each of the instances in less than one minute. The heap filled up quickly as well until they ran out of memory. It filled about 2GB worth of heap in a couple minutes. Of course, all four JVM started doing major collections one after another but couldn't free any heap memory. Unfortunately, we forgot to take thread dumps in the rush for recovering our site. All we have are the heap dumps. Also, we do auto hard commits every 5 minutes or every 10k documents. We'll be trying 4.6 soon, however I want to check if we are headed in the right direction. Document updates to SolrCloud can cause a distributed deadlock. --- Key: SOLR-5216 URL: https://issues.apache.org/jira/browse/SOLR-5216 Project: Solr Issue Type: Bug Components: SolrCloud Reporter: Mark Miller Assignee: Mark Miller Priority: Critical Fix For: 4.6, 5.0 Attachments: SOLR-5216.patch -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-5350) Add Context Aware Suggester
Areek Zillur created LUCENE-5350: Summary: Add Context Aware Suggester Key: LUCENE-5350 URL: https://issues.apache.org/jira/browse/LUCENE-5350 Project: Lucene - Core Issue Type: New Feature Components: core/search Reporter: Areek Zillur Fix For: 5.0, 4.7 It would be nice to have a Context Aware Suggester (i.e. a suggester that could return suggestions depending on some specified context(s)). Use-cases: - location-based suggestions: - returns suggestions which 'match' the context of a particular area - suggest restaurants names which are in Palo Alto (context - Palo Alto) - category-based suggestions: - returns suggestions for items that are only in certain categories/genres (contexts) - suggest movies that are of the genre sci-fi and adventure (context - [sci-fi, adventure]) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5350) Add Context Aware Suggester
[ https://issues.apache.org/jira/browse/LUCENE-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Areek Zillur updated LUCENE-5350: - Description: It would be nice to have a Context Aware Suggester (i.e. a suggester that could return suggestions depending on some specified context(s)). Use-cases: - location-based suggestions: -- returns suggestions which 'match' the context of a particular area --- suggest restaurants names which are in Palo Alto (context - Palo Alto) - category-based suggestions: -- returns suggestions for items that are only in certain categories/genres (contexts) --- suggest movies that are of the genre sci-fi and adventure (context - [sci-fi, adventure]) was: It would be nice to have a Context Aware Suggester (i.e. a suggester that could return suggestions depending on some specified context(s)). Use-cases: - location-based suggestions: - returns suggestions which 'match' the context of a particular area - suggest restaurants names which are in Palo Alto (context - Palo Alto) - category-based suggestions: - returns suggestions for items that are only in certain categories/genres (contexts) - suggest movies that are of the genre sci-fi and adventure (context - [sci-fi, adventure]) Add Context Aware Suggester --- Key: LUCENE-5350 URL: https://issues.apache.org/jira/browse/LUCENE-5350 Project: Lucene - Core Issue Type: New Feature Components: core/search Reporter: Areek Zillur Fix For: 5.0, 4.7 It would be nice to have a Context Aware Suggester (i.e. a suggester that could return suggestions depending on some specified context(s)). Use-cases: - location-based suggestions: -- returns suggestions which 'match' the context of a particular area --- suggest restaurants names which are in Palo Alto (context - Palo Alto) - category-based suggestions: -- returns suggestions for items that are only in certain categories/genres (contexts) --- suggest movies that are of the genre sci-fi and adventure (context - [sci-fi, adventure]) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5510) genericCoreNodeNames=${genericCoreNodeNames:false} and old style solr.xml fails to create collection
[
https://issues.apache.org/jira/browse/SOLR-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-5510:
-
Attachment: SOLR-5510.patch
genericCoreNodeNames=${genericCoreNodeNames:false} and old style solr.xml
fails to create collection
--
Key: SOLR-5510
URL: https://issues.apache.org/jira/browse/SOLR-5510
Project: Solr
Issue Type: Bug
Affects Versions: 4.6
Reporter: Noble Paul
Assignee: Noble Paul
Priority: Minor
Attachments: SOLR-5510.patch
See this for some more details
https://gist.github.com/serba/1fe113e78ae7e01a4f58
This is a regression caused by SOLR-5311
There are two reasons why a core does not have a reference in clusterstate
# It is starting up for the first time (core creation)
# Somebody invoked a DELETEREPLICA when the node itself was down
we neded to differentiate these two because for 1) the registration should
succeed and for #2 the registration should fail
The only way to do that was to check for the presence of the attribute
coreNodeName in the core.properties. In case #1 it would be absent and in
case#2 it would be present
but when genericCoreNodeNames=${genericCoreNodeNames:false}
ZkController#getCoreNodeName(getCoreNodeName) behaves similarly for both the
cases and hence the failure
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-5350) Add Context Aware Suggester
[ https://issues.apache.org/jira/browse/LUCENE-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Areek Zillur updated LUCENE-5350: - Attachment: LUCENE-5350.patch Initial (rough) Patch: - Add contexts and hasContexts to InputIterator - Add support to Analyzing suggester to handle contexts - Add new ContextAwareSuggester (proxies Analyzing Suggester) - Add tests for ContextAwareSuggester TODO: - The patch breaks the Lookup API (I think its better to have LookupOptions that encapsulates the query-time input to suggesters) - Add context and hasContexts support to all impl of InputIterator - General refactoring - Add FuzzySuggester support - Add docs This patch demonstrates the idea; If the approach makes sense, the appropriate changes to the API will be the next task. Feedback, thoughts welcome! It would also be nice to figure out a way so that we dont have to subclass AnalyzingSuggester to 'use' it. Add Context Aware Suggester --- Key: LUCENE-5350 URL: https://issues.apache.org/jira/browse/LUCENE-5350 Project: Lucene - Core Issue Type: New Feature Components: core/search Reporter: Areek Zillur Fix For: 5.0, 4.7 Attachments: LUCENE-5350.patch It would be nice to have a Context Aware Suggester (i.e. a suggester that could return suggestions depending on some specified context(s)). Use-cases: - location-based suggestions: -- returns suggestions which 'match' the context of a particular area --- suggest restaurants names which are in Palo Alto (context - Palo Alto) - category-based suggestions: -- returns suggestions for items that are only in certain categories/genres (contexts) --- suggest movies that are of the genre sci-fi and adventure (context - [sci-fi, adventure]) -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5510) genericCoreNodeNames=${genericCoreNodeNames:false} and old style solr.xml fails to create collection
[
https://issues.apache.org/jira/browse/SOLR-5510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834660#comment-13834660
]
Noble Paul commented on SOLR-5510:
--
Ideally would like people to use collection admin commands to achieve this.
# call DELETEREPLICA and remove a node that died
# do a core create with appropriate collection/shard params or the upcoming
addNode api
However , we should not break an existing functionality. My patch should fix
this
genericCoreNodeNames=${genericCoreNodeNames:false} and old style solr.xml
fails to create collection
--
Key: SOLR-5510
URL: https://issues.apache.org/jira/browse/SOLR-5510
Project: Solr
Issue Type: Bug
Affects Versions: 4.6
Reporter: Noble Paul
Assignee: Noble Paul
Priority: Minor
Attachments: SOLR-5510.patch
See this for some more details
https://gist.github.com/serba/1fe113e78ae7e01a4f58
This is a regression caused by SOLR-5311
There are two reasons why a core does not have a reference in clusterstate
# It is starting up for the first time (core creation)
# Somebody invoked a DELETEREPLICA when the node itself was down
we neded to differentiate these two because for 1) the registration should
succeed and for #2 the registration should fail
The only way to do that was to check for the presence of the attribute
coreNodeName in the core.properties. In case #1 it would be absent and in
case#2 it would be present
but when genericCoreNodeNames=${genericCoreNodeNames:false}
ZkController#getCoreNodeName(getCoreNodeName) behaves similarly for both the
cases and hence the failure
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4260) Inconsistent numDocs between leader and replica
[ https://issues.apache.org/jira/browse/SOLR-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834760#comment-13834760 ] Markus Jelsma commented on SOLR-4260: - I've got some bad news, it happened again on one of our clusters using a build of november 19th.Three replica's went out of sync. Inconsistent numDocs between leader and replica --- Key: SOLR-4260 URL: https://issues.apache.org/jira/browse/SOLR-4260 Project: Solr Issue Type: Bug Components: SolrCloud Environment: 5.0.0.2013.01.04.15.31.51 Reporter: Markus Jelsma Assignee: Mark Miller Priority: Critical Fix For: 5.0, 4.7 Attachments: 192.168.20.102-replica1.png, 192.168.20.104-replica2.png, clusterstate.png After wiping all cores and reindexing some 3.3 million docs from Nutch using CloudSolrServer we see inconsistencies between the leader and replica for some shards. Each core hold about 3.3k documents. For some reason 5 out of 10 shards have a small deviation in then number of documents. The leader and slave deviate for roughly 10-20 documents, not more. Results hopping ranks in the result set for identical queries got my attention, there were small IDF differences for exactly the same record causing a record to shift positions in the result set. During those tests no records were indexed. Consecutive catch all queries also return different number of numDocs. We're running a 10 node test cluster with 10 shards and a replication factor of two and frequently reindex using a fresh build from trunk. I've not seen this issue for quite some time until a few days ago. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5514) atomic update throws exception if the schema contains uuid fields: Invalid UUID String: 'java.util.UUID:e26c4d56-e98d-41de-9b7f-f63192089670'
Dirk Reuss created SOLR-5514:
-
Summary: atomic update throws exception if the schema contains
uuid fields: Invalid UUID String:
'java.util.UUID:e26c4d56-e98d-41de-9b7f-f63192089670'
Key: SOLR-5514
URL: https://issues.apache.org/jira/browse/SOLR-5514
Project: Solr
Issue Type: Bug
Affects Versions: 4.5.1
Environment: unix and windows
Reporter: Dirk Reuss
I am updating an exiting document with the statement
adddocfield name='name' update='set'newvalue/field
All fields are stored and I have several UUID fields. About 10-20% of the
update commands will fail with the message: (example)
Invalid UUID String: 'java.util.UUID:532c9353-d391-4a04-8618-dc2fa1ef8b35'
the point is that java.util.UUID seems to be prepended to the original uuid
stored in the field and when the value is written this error occours.
I tried to check if this specific uuid field was the problem and
added the uuid field in the update xml with(field name='id1'
update='set'...). But the error simply moved to an other uuid field.
here is the original exception:
lst name=responseHeaderint name=status500/intint
name=QTime34/int/lstlst name=errorstr name=msgError while
creating field
'MyUUIDField{type=uuid,properties=indexed,stored,omitTermFreqAndPositions,required,
required=true}' from value
'java.util.UUID:e26c4d56-e98d-41de-9b7f-f63192089670'/strstr
name=traceorg.apache.solr.common.SolrException: Error while creating field
'MyUUIDField{type=uuid,properties=indexed,stored,omitTermFreqAndPositions,required,
required=true}' from value
'java.util.UUID:e26c4d56-e98d-41de-9b7f-f63192089670'
at org.apache.solr.schema.FieldType.createField(FieldType.java:259)
at org.apache.solr.schema.StrField.createFields(StrField.java:56)
at
org.apache.solr.update.DocumentBuilder.addField(DocumentBuilder.java:47)
at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:118)
at
org.apache.solr.update.AddUpdateCommand.getLuceneDocument(AddUpdateCommand.java:77)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:215)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:556)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:692)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:435)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:703)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:406)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:781)
Caused by: org.apache.solr.common.SolrException: Invalid UUID String:
'java.util.UUID:e26c4d56-e98d-41de-9b7f-f63192089670'
at
[jira] [Created] (LUCENE-5351) DirectoryReader#close can throw AlreadyClosedException if it's and NRT reader
Simon Willnauer created LUCENE-5351:
---
Summary: DirectoryReader#close can throw AlreadyClosedException if
it's and NRT reader
Key: LUCENE-5351
URL: https://issues.apache.org/jira/browse/LUCENE-5351
Project: Lucene - Core
Issue Type: Bug
Components: core/index
Affects Versions: 4.6
Reporter: Simon Willnauer
Assignee: Simon Willnauer
Fix For: 5.0, 4.7
in StandartDirectoryReader#doClose we do this:
{noformat}
if (writer != null) {
// Since we just closed, writer may now be able to
// delete unused files:
writer.deletePendingFiles();
}
{noformat}
which can throw AlreadyClosedException from the directory if the Direcotory has
already closed. To me this looks like a bug and we should catch this exception
from the directory.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5480) Make MoreLikeThisHandler distributable
[ https://issues.apache.org/jira/browse/SOLR-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834853#comment-13834853 ] Markus Jelsma commented on SOLR-5480: - Hi Steve - do you have a patch for trunk? Some things changed, IndexReader no longer returns Document but StoredDocument, which cannot be passed to IndexSchema.getUniqueKeyField(), that still needs Document. Make MoreLikeThisHandler distributable -- Key: SOLR-5480 URL: https://issues.apache.org/jira/browse/SOLR-5480 Project: Solr Issue Type: Improvement Reporter: Steve Molloy Attachments: SOLR-5480.patch, SOLR-5480.patch The MoreLikeThis component, when used in the standard search handler supports distributed searches. But the MoreLikeThisHandler itself doesn't, which prevents from say, passing in text to perform the query. I'll start looking into adapting the SearchHandler logic to the MoreLikeThisHandler. If anyone has some work done already and want to share, or want to contribute, any help will be welcomed. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5216) Document updates to SolrCloud can cause a distributed deadlock.
[ https://issues.apache.org/jira/browse/SOLR-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834886#comment-13834886 ] Erick Erickson commented on SOLR-5216: -- This shouldn't affect single-shard setups. The deadlock, as I remember, showed up when lots of nodes split up incoming batches of documents to forward to lots of leaders. Since a single shard won't split up the documents, I doubt this is the root of what you're seeing. But yeah, a stack trace would tell us for certain. And Mark committed SOLR-5232 which uses a different mechanism anyway. To recap: I doubt this issue is a problem in single-shard setups. Document updates to SolrCloud can cause a distributed deadlock. --- Key: SOLR-5216 URL: https://issues.apache.org/jira/browse/SOLR-5216 Project: Solr Issue Type: Bug Components: SolrCloud Reporter: Mark Miller Assignee: Mark Miller Priority: Critical Fix For: 4.6, 5.0 Attachments: SOLR-5216.patch -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-5364) SolrCloud stops accepting updates
[ https://issues.apache.org/jira/browse/SOLR-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson resolved SOLR-5364. -- Resolution: Fixed Fix Version/s: 4.6 Since the two SOLR tickets above pretty much change how all this is done (especially SOLR-5232 when using DIH), I'll mark this closed. Of course if anyone sees this kind of behavior again we'll re-open. SolrCloud stops accepting updates - Key: SOLR-5364 URL: https://issues.apache.org/jira/browse/SOLR-5364 Project: Solr Issue Type: Bug Components: SolrCloud Affects Versions: 4.4, 4.5, 4.6 Reporter: Chris Priority: Blocker Fix For: 4.6 Attachments: threaddump.201311180812.zip I'm attempting to import data into a SolrCloud cluster. After a certain amount of time, the cluster stops accepting updates. I have tried numerous suggestions in IRC from Elyorag and others without resolve. I have had this issue with 4.4, and understood there was a deadlock issue fixed in 4.5, which hasn't resolved the issue, neither have the 4.6 snapshots. I've tried with Tomcat, various tomcat configuration changes to threading, and with Jetty. Tried with various index merging configurations as I initially thought there was a deadlock with concurrent merg scheduler, however same issue with SerialMergeScheduler. The cluster stops accepting updates after some amount of time, this seems to vary and is inconsistent. Sometimes I manage to index 400k docs, other times ~1million . Querying the cluster continues to work. I can reproduce the issue consistently, and is currently blocking our transition to Solr. I can provide stack traces, thread dumps, jstack dumps as required. Here are two jstacks thus far: http://pastebin.com/1ktjBYbf http://pastebin.com/8JiQc3rb I have got these jstacks from the latest 4.6 snapshot, also running solrj snapshot. The issue is also consistently reproducable with BinaryRequest writer. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5488) Fix up test failures for Analytics Component
[ https://issues.apache.org/jira/browse/SOLR-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834900#comment-13834900 ] Erick Erickson commented on SOLR-5488: -- OK, I've _finally_ managed to put in the dumps that tell us something. It took me long enough See: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8492/ Here's the crux: Looking for: /response/lst[@name='stats']/lst[@name='ar']/double[@name='sum'] in the response. Here's the ar section. Obviously no sum value, and the parser barfs. So, is this ever expected? In which it's just a test fix, don't try to parse an empty number. If it's not expected, we need to fix the underlying analytics code. Over to [~sbower] and [~houstonputman] for a reading on whether this is an underlying code problem or a test issue. Let me know if having empty values is ever expected, and if so I'll fix the tests. If not, I'll look for a patch from someone else that fixes the underlying code. lst name=ar long name=count100/long null name=mcm/ null name=mean/ null name=median/ null name=su/ null name=sum/ null name=unique/ /lst Fix up test failures for Analytics Component Key: SOLR-5488 URL: https://issues.apache.org/jira/browse/SOLR-5488 Project: Solr Issue Type: Bug Affects Versions: 5.0, 4.7 Reporter: Erick Erickson Assignee: Erick Erickson Attachments: SOLR-5488.patch, SOLR-5488.patch, SOLR-5488.patch The analytics component has a few test failures, perhaps environment-dependent. This is just to collect the test fixes in one place for convenience when we merge back into 4.x -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1043 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1043/ Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC All tests passed Build Log: [...truncated 10035 lines...] [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20131128_151524_833.syserr [junit4] JVM J0: stderr (verbatim) [junit4] java(521,0x1376b2000) malloc: *** error for object 0x1376a04f0: pointer being freed was not allocated [junit4] *** set a breakpoint in malloc_error_break to debug [junit4] JVM J0: EOF [...truncated 1 lines...] [junit4] ERROR: JVM J0 ended with an exception, command line: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/bin/java -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/heapdumps -Dtests.prefix=tests -Dtests.seed=3F6DA9AB2D4B3A96 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=4.7 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.slow=true -Dtests.asserts.gracious=false -Dtests.multiplier=1 -DtempDir=. -Djava.io.tmpdir=. -Djunit4.tempDir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/temp -Dclover.db.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/clover/db -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Djava.security.policy=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/tools/junit4/tests.policy -Dlucene.version=4.7-SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.disableHdfs=true -Dfile.encoding=UTF-8 -classpath
[jira] [Commented] (SOLR-5488) Fix up test failures for Analytics Component
[ https://issues.apache.org/jira/browse/SOLR-5488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834920#comment-13834920 ] Erick Erickson commented on SOLR-5488: -- Actually, when I _look_ at it, the structure is just wrong with the null name=sum/ bits. So a better question is why is the response being assembled like this? Fix up test failures for Analytics Component Key: SOLR-5488 URL: https://issues.apache.org/jira/browse/SOLR-5488 Project: Solr Issue Type: Bug Affects Versions: 5.0, 4.7 Reporter: Erick Erickson Assignee: Erick Erickson Attachments: SOLR-5488.patch, SOLR-5488.patch, SOLR-5488.patch The analytics component has a few test failures, perhaps environment-dependent. This is just to collect the test fixes in one place for convenience when we merge back into 4.x -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5480) Make MoreLikeThisHandler distributable
[ https://issues.apache.org/jira/browse/SOLR-5480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13834933#comment-13834933 ] Steve Molloy commented on SOLR-5480: Not yet, will try to find some time to see how it can be adjusted for trunk. Make MoreLikeThisHandler distributable -- Key: SOLR-5480 URL: https://issues.apache.org/jira/browse/SOLR-5480 Project: Solr Issue Type: Improvement Reporter: Steve Molloy Attachments: SOLR-5480.patch, SOLR-5480.patch The MoreLikeThis component, when used in the standard search handler supports distributed searches. But the MoreLikeThisHandler itself doesn't, which prevents from say, passing in text to perform the query. I'll start looking into adapting the SearchHandler logic to the MoreLikeThisHandler. If anyone has some work done already and want to share, or want to contribute, any help will be welcomed. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5515) NPE when getting stats on date field with empty result on solrcloud
Alexander Sagen created SOLR-5515:
-
Summary: NPE when getting stats on date field with empty result on
solrcloud
Key: SOLR-5515
URL: https://issues.apache.org/jira/browse/SOLR-5515
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 4.5.1, 4.6
Environment: ubuntu 13.04, java 1.7.0_45-b18
Reporter: Alexander Sagen
Priority: Critical
Steps to reproduce:
1. Download solr 4.6.0, unzip twice in different directories.
2. Start a two-shard cluster based on default example
{quote}
dir1/example java -Dbootstrap_confdir=./solr/collection1/conf
-Dcollection.configName=myconf -DzkRun -DnumShards=2 -jar start.jar
dir2/example java -Djetty.port=7574 -DzkHost=localhost:9983 -jar start.jar
{quote}
3. Visit
http://localhost:8983/solr/query?q=author:astats=truestats.field=last_modified
This causes a nullpointer (given that the index is empty or the query returns 0
rows)
Stacktrace:
{noformat}
190 [qtp290025410-11] INFO org.apache.solr.core.SolrCore – [collection1]
webapp=/solr path=/query
params={stats.field=last_modifiedstats=trueq=author:a} hits=0 status=500
QTime=8
191 [qtp290025410-11] ERROR org.apache.solr.servlet.SolrDispatchFilter –
null:java.lang.NullPointerException
at
org.apache.solr.handler.component.DateStatsValues.updateTypeSpecificStats(StatsValuesFactory.java:409)
at
org.apache.solr.handler.component.AbstractStatsValues.accumulate(StatsValuesFactory.java:109)
at
org.apache.solr.handler.component.StatsComponent.handleResponses(StatsComponent.java:113)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:710)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: VOTE: RC0 Release apache-solr-ref-guide-4.6.pdf
Hi, On page 293 : rm -r shard*/solr/zoo_data should be rm -r node*/solr/zoo_data On page 297 : ... shard, an d forwards ... should be ... shard, and forwards ... Thanks, Ahmet On Wednesday, November 27, 2013 2:47 PM, Cassandra Targett casstarg...@gmail.com wrote: I noticed a couple of small typos and inconsistencies that I've fixed, but I don't think they warrant a respin. They're more for appearance than for any factual problems. +1 Sorry for the delay from me - I've been traveling for holidays. On Tue, Nov 26, 2013 at 4:22 AM, Jan Høydahl jan@cominvent.com wrote: * Page 5: Screenshots with 4.0.0-beta texts * Page 165: Links to 4.0.0 version of JavaDoc (now fixed in Confluence) * Page 204: Table - group.func - Supported only in Sol4r 4.0. (should be Supported since Solr 4.0.) (now fixed in Confluence) * Page 308: Strange xml code box layout, why all the whitespace? But these are minors, so here's my +1 -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com 25. nov. 2013 kl. 19:34 skrev Chris Hostetter hossman_luc...@fucit.org: Please VOTE to release the following as apache-solr-ref-guide-4.6.pdf ... https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-4.6-RC0/ $ cat apache-solr-ref-guide-4.6.pdf.sha1 7ad494c5a3cdc085e01a54d507ae33a75cc319e6 apache-solr-ref-guide-4.6.pdf -Hoss - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5354) Distributed sort is broken with CUSTOM FieldType
[
https://issues.apache.org/jira/browse/SOLR-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835077#comment-13835077
]
ASF subversion and git services commented on SOLR-5354:
---
Commit 1546457 from [~steve_rowe] in branch 'dev/trunk'
[ https://svn.apache.org/r1546457 ]
SOLR-5354: Distributed sort is broken with CUSTOM FieldType
Distributed sort is broken with CUSTOM FieldType
Key: SOLR-5354
URL: https://issues.apache.org/jira/browse/SOLR-5354
Project: Solr
Issue Type: Bug
Components: SearchComponents - other
Affects Versions: 4.4, 4.5, 5.0
Reporter: Jessica Cheng
Assignee: Steve Rowe
Labels: custom, query, sort
Attachments: SOLR-5354.patch, SOLR-5354.patch, SOLR-5354.patch,
SOLR-5354.patch
We added a custom field type to allow an indexed binary field type that
supports search (exact match), prefix search, and sort as unsigned bytes
lexicographical compare. For sort, BytesRef's UTF8SortedAsUnicodeComparator
accomplishes what we want, and even though the name of the comparator
mentions UTF8, it doesn't actually assume so and just does byte-level
operation, so it's good. However, when we do this across different nodes, we
run into an issue where in QueryComponent.doFieldSortValues:
// Must do the same conversion when sorting by a
// String field in Lucene, which returns the terms
// data as BytesRef:
if (val instanceof BytesRef) {
UnicodeUtil.UTF8toUTF16((BytesRef)val, spare);
field.setStringValue(spare.toString());
val = ft.toObject(field);
}
UnicodeUtil.UTF8toUTF16 is called on our byte array,which isn't actually
UTF8. I did a hack where I specified our own field comparator to be
ByteBuffer based to get around that instanceof check, but then the field
value gets transformed into BYTEARR in JavaBinCodec, and when it's
unmarshalled, it gets turned into byte[]. Then, in QueryComponent.mergeIds, a
ShardFieldSortedHitQueue is constructed with ShardDoc.getCachedComparator,
which decides to give me comparatorNatural in the else of the TODO for
CUSTOM, which barfs because byte[] are not Comparable...
From Chris Hostetter:
I'm not very familiar with the distributed sorting code, but based on your
comments, and a quick skim of the functions you pointed to, it definitely
seems like there are two problems here for people trying to implement
custom sorting in custom FieldTypes...
1) QueryComponent.doFieldSortValues - this definitely seems like it should
be based on the FieldType, not an instanceof BytesRef check (oddly: the
comment event suggestsion that it should be using the FieldType's
indexedToReadable() method -- but it doesn't do that. If it did, then
this part of hte logic should work for you as long as your custom
FieldType implemented indexedToReadable in a sane way.
2) QueryComponent.mergeIds - that TODO definitely looks like a gap that
needs filled. I'm guessing the sanest thing to do in the CUSTOM case
would be to ask the FieldComparatorSource (which should be coming from the
SortField that the custom FieldType produced) to create a FieldComparator
(via newComparator - the numHits sortPos could be anything) and then
wrap that up in a Comparator facade that delegates to
FieldComparator.compareValues
That way a custom FieldType could be in complete control of the sort
comparisons (even when merging ids).
...But as i said: i may be missing something, i'm not super familia with
that code. Please try it out and let us know if thta works -- either way
please open a Jira pointing out the problems trying to implement
distributed sorting in a custom FieldType.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5354) Distributed sort is broken with CUSTOM FieldType
[
https://issues.apache.org/jira/browse/SOLR-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Rowe updated SOLR-5354:
-
Attachment: SOLR-5354.patch
I added a Trie long field to the sort missing first/last tests, and moved them
from the existing {{TestDistributedSearch}} class to a new class
{{TestDistributedMissingSort}} with its own dedicated minimal schema file.
All solr tests pass, and {{ant precommit}} passes.
Committing shortly.
I'll make a separate issue for testing distributed collation.
Distributed sort is broken with CUSTOM FieldType
Key: SOLR-5354
URL: https://issues.apache.org/jira/browse/SOLR-5354
Project: Solr
Issue Type: Bug
Components: SearchComponents - other
Affects Versions: 4.4, 4.5, 5.0
Reporter: Jessica Cheng
Assignee: Steve Rowe
Labels: custom, query, sort
Attachments: SOLR-5354.patch, SOLR-5354.patch, SOLR-5354.patch,
SOLR-5354.patch
We added a custom field type to allow an indexed binary field type that
supports search (exact match), prefix search, and sort as unsigned bytes
lexicographical compare. For sort, BytesRef's UTF8SortedAsUnicodeComparator
accomplishes what we want, and even though the name of the comparator
mentions UTF8, it doesn't actually assume so and just does byte-level
operation, so it's good. However, when we do this across different nodes, we
run into an issue where in QueryComponent.doFieldSortValues:
// Must do the same conversion when sorting by a
// String field in Lucene, which returns the terms
// data as BytesRef:
if (val instanceof BytesRef) {
UnicodeUtil.UTF8toUTF16((BytesRef)val, spare);
field.setStringValue(spare.toString());
val = ft.toObject(field);
}
UnicodeUtil.UTF8toUTF16 is called on our byte array,which isn't actually
UTF8. I did a hack where I specified our own field comparator to be
ByteBuffer based to get around that instanceof check, but then the field
value gets transformed into BYTEARR in JavaBinCodec, and when it's
unmarshalled, it gets turned into byte[]. Then, in QueryComponent.mergeIds, a
ShardFieldSortedHitQueue is constructed with ShardDoc.getCachedComparator,
which decides to give me comparatorNatural in the else of the TODO for
CUSTOM, which barfs because byte[] are not Comparable...
From Chris Hostetter:
I'm not very familiar with the distributed sorting code, but based on your
comments, and a quick skim of the functions you pointed to, it definitely
seems like there are two problems here for people trying to implement
custom sorting in custom FieldTypes...
1) QueryComponent.doFieldSortValues - this definitely seems like it should
be based on the FieldType, not an instanceof BytesRef check (oddly: the
comment event suggestsion that it should be using the FieldType's
indexedToReadable() method -- but it doesn't do that. If it did, then
this part of hte logic should work for you as long as your custom
FieldType implemented indexedToReadable in a sane way.
2) QueryComponent.mergeIds - that TODO definitely looks like a gap that
needs filled. I'm guessing the sanest thing to do in the CUSTOM case
would be to ask the FieldComparatorSource (which should be coming from the
SortField that the custom FieldType produced) to create a FieldComparator
(via newComparator - the numHits sortPos could be anything) and then
wrap that up in a Comparator facade that delegates to
FieldComparator.compareValues
That way a custom FieldType could be in complete control of the sort
comparisons (even when merging ids).
...But as i said: i may be missing something, i'm not super familia with
that code. Please try it out and let us know if thta works -- either way
please open a Jira pointing out the problems trying to implement
distributed sorting in a custom FieldType.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5354) Distributed sort is broken with CUSTOM FieldType
[
https://issues.apache.org/jira/browse/SOLR-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13835081#comment-13835081
]
ASF subversion and git services commented on SOLR-5354:
---
Commit 1546461 from [~steve_rowe] in branch 'dev/branches/branch_4x'
[ https://svn.apache.org/r1546461 ]
SOLR-5354: Distributed sort is broken with CUSTOM FieldType (merged trunk
r1546457)
Distributed sort is broken with CUSTOM FieldType
Key: SOLR-5354
URL: https://issues.apache.org/jira/browse/SOLR-5354
Project: Solr
Issue Type: Bug
Components: SearchComponents - other
Affects Versions: 4.4, 4.5, 5.0
Reporter: Jessica Cheng
Assignee: Steve Rowe
Labels: custom, query, sort
Fix For: 5.0, 4.7
Attachments: SOLR-5354.patch, SOLR-5354.patch, SOLR-5354.patch,
SOLR-5354.patch
We added a custom field type to allow an indexed binary field type that
supports search (exact match), prefix search, and sort as unsigned bytes
lexicographical compare. For sort, BytesRef's UTF8SortedAsUnicodeComparator
accomplishes what we want, and even though the name of the comparator
mentions UTF8, it doesn't actually assume so and just does byte-level
operation, so it's good. However, when we do this across different nodes, we
run into an issue where in QueryComponent.doFieldSortValues:
// Must do the same conversion when sorting by a
// String field in Lucene, which returns the terms
// data as BytesRef:
if (val instanceof BytesRef) {
UnicodeUtil.UTF8toUTF16((BytesRef)val, spare);
field.setStringValue(spare.toString());
val = ft.toObject(field);
}
UnicodeUtil.UTF8toUTF16 is called on our byte array,which isn't actually
UTF8. I did a hack where I specified our own field comparator to be
ByteBuffer based to get around that instanceof check, but then the field
value gets transformed into BYTEARR in JavaBinCodec, and when it's
unmarshalled, it gets turned into byte[]. Then, in QueryComponent.mergeIds, a
ShardFieldSortedHitQueue is constructed with ShardDoc.getCachedComparator,
which decides to give me comparatorNatural in the else of the TODO for
CUSTOM, which barfs because byte[] are not Comparable...
From Chris Hostetter:
I'm not very familiar with the distributed sorting code, but based on your
comments, and a quick skim of the functions you pointed to, it definitely
seems like there are two problems here for people trying to implement
custom sorting in custom FieldTypes...
1) QueryComponent.doFieldSortValues - this definitely seems like it should
be based on the FieldType, not an instanceof BytesRef check (oddly: the
comment event suggestsion that it should be using the FieldType's
indexedToReadable() method -- but it doesn't do that. If it did, then
this part of hte logic should work for you as long as your custom
FieldType implemented indexedToReadable in a sane way.
2) QueryComponent.mergeIds - that TODO definitely looks like a gap that
needs filled. I'm guessing the sanest thing to do in the CUSTOM case
would be to ask the FieldComparatorSource (which should be coming from the
SortField that the custom FieldType produced) to create a FieldComparator
(via newComparator - the numHits sortPos could be anything) and then
wrap that up in a Comparator facade that delegates to
FieldComparator.compareValues
That way a custom FieldType could be in complete control of the sort
comparisons (even when merging ids).
...But as i said: i may be missing something, i'm not super familia with
that code. Please try it out and let us know if thta works -- either way
please open a Jira pointing out the problems trying to implement
distributed sorting in a custom FieldType.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-5354) Distributed sort is broken with CUSTOM FieldType
[
https://issues.apache.org/jira/browse/SOLR-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Rowe resolved SOLR-5354.
--
Resolution: Fixed
Fix Version/s: 4.7
5.0
Committed to trunk and branch_4x.
Distributed sort is broken with CUSTOM FieldType
Key: SOLR-5354
URL: https://issues.apache.org/jira/browse/SOLR-5354
Project: Solr
Issue Type: Bug
Components: SearchComponents - other
Affects Versions: 4.4, 4.5, 5.0
Reporter: Jessica Cheng
Assignee: Steve Rowe
Labels: custom, query, sort
Fix For: 5.0, 4.7
Attachments: SOLR-5354.patch, SOLR-5354.patch, SOLR-5354.patch,
SOLR-5354.patch
We added a custom field type to allow an indexed binary field type that
supports search (exact match), prefix search, and sort as unsigned bytes
lexicographical compare. For sort, BytesRef's UTF8SortedAsUnicodeComparator
accomplishes what we want, and even though the name of the comparator
mentions UTF8, it doesn't actually assume so and just does byte-level
operation, so it's good. However, when we do this across different nodes, we
run into an issue where in QueryComponent.doFieldSortValues:
// Must do the same conversion when sorting by a
// String field in Lucene, which returns the terms
// data as BytesRef:
if (val instanceof BytesRef) {
UnicodeUtil.UTF8toUTF16((BytesRef)val, spare);
field.setStringValue(spare.toString());
val = ft.toObject(field);
}
UnicodeUtil.UTF8toUTF16 is called on our byte array,which isn't actually
UTF8. I did a hack where I specified our own field comparator to be
ByteBuffer based to get around that instanceof check, but then the field
value gets transformed into BYTEARR in JavaBinCodec, and when it's
unmarshalled, it gets turned into byte[]. Then, in QueryComponent.mergeIds, a
ShardFieldSortedHitQueue is constructed with ShardDoc.getCachedComparator,
which decides to give me comparatorNatural in the else of the TODO for
CUSTOM, which barfs because byte[] are not Comparable...
From Chris Hostetter:
I'm not very familiar with the distributed sorting code, but based on your
comments, and a quick skim of the functions you pointed to, it definitely
seems like there are two problems here for people trying to implement
custom sorting in custom FieldTypes...
1) QueryComponent.doFieldSortValues - this definitely seems like it should
be based on the FieldType, not an instanceof BytesRef check (oddly: the
comment event suggestsion that it should be using the FieldType's
indexedToReadable() method -- but it doesn't do that. If it did, then
this part of hte logic should work for you as long as your custom
FieldType implemented indexedToReadable in a sane way.
2) QueryComponent.mergeIds - that TODO definitely looks like a gap that
needs filled. I'm guessing the sanest thing to do in the CUSTOM case
would be to ask the FieldComparatorSource (which should be coming from the
SortField that the custom FieldType produced) to create a FieldComparator
(via newComparator - the numHits sortPos could be anything) and then
wrap that up in a Comparator facade that delegates to
FieldComparator.compareValues
That way a custom FieldType could be in complete control of the sort
comparisons (even when merging ids).
...But as i said: i may be missing something, i'm not super familia with
that code. Please try it out and let us know if thta works -- either way
please open a Jira pointing out the problems trying to implement
distributed sorting in a custom FieldType.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
RE: Collections API
Thanks, I already had the genericCoreNodeNames=true in solrcloud section of
solr.xml, new format. But I had a str entry instead of a bool, which
apparently is simply treated as false. Anyhow, in my case the fix works if I
move the bit setting the coreNodeName after the publish, not before. If it's
before, I get a timeout error while it waits for a replica that is never set in
waitForShardId.
I'll both apply the modified patch and switch from str to bool. :)
Thanks for the help,
Steve
From: Alexey Serba [ase...@gmail.com]
Sent: November 28, 2013 2:10 AM
To: dev@lucene.apache.org
Subject: Re: Collections API
https://issues.apache.org/jira/browse/SOLR-5510
I don't really understand all the details why is that happening, but the
workaround is to add genericCoreNodeNames=${genericCoreNodeNames:true}
attribute to cores element in your solr.xml file.
On Tue, Nov 26, 2013 at 10:10 PM, Steve Molloy
smol...@opentext.com wrote:
I'm trying to reconcile our fork with 4.6 tag and I'm getting weird behaviour
in Collections API, more specifically in ZkController's preRegister method
after calling the create method of the collections API. When it checks if a
slice has a replica for current
node name, there is never any because at this stage, the slice has no replica.
This is the new code that seems to be causing my issue, I can force the
autoCreated to be always true to avoid the issue, but would like a cleaner
way if there is one.
if(cd.getCloudDescriptor().getCollectionName() !=null
cd.getCloudDescriptor().getCoreNodeName() != null ) {
//we were already registered
if(zkStateReader.getClusterState().hasCollection(cd.getCloudDescriptor().getCollectionName())){
DocCollection coll =
zkStateReader.getClusterState().getCollection(cd.getCloudDescriptor().getCollectionName());
if(!true.equals(coll.getStr(autoCreated))){
Slice slice = coll.getSlice(cd.getCloudDescriptor().getShardId());
if(slice != null){
== if(slice.getReplica(cd.getCloudDescriptor().getCoreNodeName()) ==
null) {
log.info(core_removed This core is removed from ZK);
throw new SolrException(ErrorCode.NOT_FOUND,coreNodeName + is
removed);
}
}
}
}
}
Thanks.
Steve
-
To unsubscribe, e-mail:
dev-unsubscr...@lucene.apache.org
For additional commands, e-mail:
dev-h...@lucene.apache.org
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-5352) PackedInts.fastestFormatAndBits returns slower format for Direct
Robert Muir created LUCENE-5352: --- Summary: PackedInts.fastestFormatAndBits returns slower format for Direct Key: LUCENE-5352 URL: https://issues.apache.org/jira/browse/LUCENE-5352 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir from some simple testing, DirectPacked64SingleBlockReader seems slower than DirectPackedReader (at least in a few cases i tested, more testing needed, but look at the code). this is unintuitive, because by passing e.g. FASTER you are making it slower. -- This message was sent by Atlassian JIRA (v6.1#6144) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1045 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1045/ Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC All tests passed Build Log: [...truncated 10445 lines...] [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20131129_052331_303.syserr [junit4] JVM J0: stderr (verbatim) [junit4] java(458,0x1471bb000) malloc: *** error for object 0x1471aa0e2: pointer being freed was not allocated [junit4] *** set a breakpoint in malloc_error_break to debug [junit4] JVM J0: EOF [...truncated 1 lines...] [junit4] ERROR: JVM J0 ended with an exception, command line: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/jre/bin/java -XX:-UseCompressedOops -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/heapdumps -Dtests.prefix=tests -Dtests.seed=83809F20459D2411 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=4.7 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.slow=true -Dtests.asserts.gracious=false -Dtests.multiplier=1 -DtempDir=. -Djava.io.tmpdir=. -Djunit4.tempDir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/temp -Dclover.db.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/build/clover/db -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Djava.security.policy=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/tools/junit4/tests.policy -Dlucene.version=4.7-SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.disableHdfs=true -Dfile.encoding=ISO-8859-1 -classpath
