[jira] [Updated] (SOLR-5530) SolrJ NoOpResponseParser
[ https://issues.apache.org/jira/browse/SOLR-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitaliy Zhovtyuk updated SOLR-5530: --- Attachment: SOLR-5530.patch Added comments to org.apache.solr.client.solrj.impl.NoOpResponseParser Added Tests checking NoOpResponseParser Added test demonstrating how to use NoOpResponseParser in order to construct query SolrJ NoOpResponseParser Key: SOLR-5530 URL: https://issues.apache.org/jira/browse/SOLR-5530 Project: Solr Issue Type: New Feature Components: clients - java Reporter: Upayavira Priority: Minor Attachments: PATCH-5530.txt, SOLR-5530.patch If you want the raw response string out of SolrJ, the advice seems to be to just use an HttpClient directly. However, sometimes you may have a lot of SolrJ infrastructure already in place to build out queries, etc, so it would be much simpler to just use SolrJ to do the work. This patch offers a NoOpResponseParser, which simply puts the entire response into an entry in a NamedList. Because the response isn't parsed into a QueryResponse, usage is slightly different: HttpSolrServer server = new HttpSolrServer(http://localhost:8983/solr;); SolrQuery query = new SolrQuery(*:*); QueryRequest req = new QueryRequest(query); server.setParser(new NoOpResponseParser()); NamedListObject resp = server.request(req); String responseString = resp.get(response); -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5228) IndexWriter.addIndexes copies raw files but acquires no locks
[ https://issues.apache.org/jira/browse/LUCENE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882266#comment-13882266 ] Littlestar commented on LUCENE-5228: not fixed in 4.6.1? Thanks. I see stuff like: merge problem with lucene 3 and 4 indices (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory) may related to https://issues.apache.org/jira/browse/LUCENE-5377 IndexWriter.addIndexes copies raw files but acquires no locks - Key: LUCENE-5228 URL: https://issues.apache.org/jira/browse/LUCENE-5228 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir Assignee: Michael McCandless Fix For: 5.0, 4.7 Attachments: LUCENE-5228.patch I see stuff like: merge problem with lucene 3 and 4 indices (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory) it currently has in its javadocs: NOTE: the index in each Directory must not be changed (opened by a writer) while this method is running. This method does not acquire a write lock in each input Directory, so it is up to the caller to enforce this. This method should be acquiring locks: its copying *RAW FILES*. Otherwise we should remove it. If someone doesnt like that, or is mad because its 10ns slower, they can use NoLockFactory. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-5228) IndexWriter.addIndexes copies raw files but acquires no locks
[ https://issues.apache.org/jira/browse/LUCENE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882266#comment-13882266 ] Littlestar edited comment on LUCENE-5228 at 1/26/14 10:54 AM: -- not fixed in 4.6.1? Thanks. I see stuff like: merge problem with lucene 3 and 4 indices (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory) may related to LUCENE-5377 was (Author: cnstar9988): not fixed in 4.6.1? Thanks. I see stuff like: merge problem with lucene 3 and 4 indices (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory) may related to https://issues.apache.org/jira/browse/LUCENE-5377 IndexWriter.addIndexes copies raw files but acquires no locks - Key: LUCENE-5228 URL: https://issues.apache.org/jira/browse/LUCENE-5228 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir Assignee: Michael McCandless Fix For: 5.0, 4.7 Attachments: LUCENE-5228.patch I see stuff like: merge problem with lucene 3 and 4 indices (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory) it currently has in its javadocs: NOTE: the index in each Directory must not be changed (opened by a writer) while this method is running. This method does not acquire a write lock in each input Directory, so it is up to the caller to enforce this. This method should be acquiring locks: its copying *RAW FILES*. Otherwise we should remove it. If someone doesnt like that, or is mad because its 10ns slower, they can use NoLockFactory. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5228) IndexWriter.addIndexes copies raw files but acquires no locks
[ https://issues.apache.org/jira/browse/LUCENE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882270#comment-13882270 ] Littlestar commented on LUCENE-5228: IndexWriter.addIndexes(Directory[]) now acquires the IW write lock I think the write lock is too expensive for my app. a SnapshotDeletionPolicy is OK for some situation. is there a way equal to IndexWriter.addIndexes(IndexCommit)? IndexWriter.addIndexes copies raw files but acquires no locks - Key: LUCENE-5228 URL: https://issues.apache.org/jira/browse/LUCENE-5228 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir Assignee: Michael McCandless Fix For: 5.0, 4.7 Attachments: LUCENE-5228.patch I see stuff like: merge problem with lucene 3 and 4 indices (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory) it currently has in its javadocs: NOTE: the index in each Directory must not be changed (opened by a writer) while this method is running. This method does not acquire a write lock in each input Directory, so it is up to the caller to enforce this. This method should be acquiring locks: its copying *RAW FILES*. Otherwise we should remove it. If someone doesnt like that, or is mad because its 10ns slower, they can use NoLockFactory. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Benson Margulies as Lucene/Solr committer!
Good to have you aboard! Erick On Sat, Jan 25, 2014 at 10:52 PM, Mark Miller markrmil...@gmail.com wrote: Welcome! - Mark http://about.me/markrmiller On Jan 25, 2014, at 4:40 PM, Michael McCandless luc...@mikemccandless.com wrote: I'm pleased to announce that Benson Margulies has accepted to join our ranks as a committer. Benson has been involved in a number of Lucene/Solr issues over time (see http://jirasearch.mikemccandless.com/search.py?index=jirachg=ddsa1=allUsersa2=Benson+Margulies ), most recently on debugging tricky analysis issues. Benson, it is tradition that you introduce yourself with a brief bio. I know you're heavily involved in other Apache projects already... Once your account is set up, you should then be able to add yourself to the who we are page on the website as well. Congratulations and welcome! Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [JENKINS] Lucene-Solr-4.x-Windows (64bit/jdk1.7.0_51) - Build # 3636 - Still Failing!
thx robert On Sun, Jan 26, 2014 at 5:20 AM, Robert Muir rcm...@gmail.com wrote: I committed a fix. On Sat, Jan 25, 2014 at 9:18 PM, Policeman Jenkins Server jenk...@thetaphi.de wrote: Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Windows/3636/ Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseParallelGC 4 tests failed. FAILED: org.apache.lucene.search.suggest.DocumentValueSourceDictionaryTest.testWithoutPayload Error Message: this codec cannot write docvalues Stack Trace: java.lang.UnsupportedOperationException: this codec cannot write docvalues at __randomizedtesting.SeedInfo.seed([7DBDDBAB981B118D:C7275C3F6C51FF68]:0) at org.apache.lucene.codecs.lucene3x.Lucene3xCodec$1.fieldsConsumer(Lucene3xCodec.java:73) at org.apache.lucene.index.DocValuesProcessor.flush(DocValuesProcessor.java:77) at org.apache.lucene.index.TwoStoredFieldsConsumers.flush(TwoStoredFieldsConsumers.java:42) at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:80) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:465) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:506) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:616) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2864) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989) at org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:237) at org.apache.lucene.search.suggest.DocumentValueSourceDictionaryTest.testWithoutPayload(DocumentValueSourceDictionaryTest.java:138) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at
[jira] [Commented] (SOLR-5577) indexing delay due to zookeeper election
[
https://issues.apache.org/jira/browse/SOLR-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882316#comment-13882316
]
ASF subversion and git services commented on SOLR-5577:
---
Commit 1561499 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_4x'
[ https://svn.apache.org/r1561499 ]
SOLR-5577: Harden leaking Timer thread.
indexing delay due to zookeeper election
Key: SOLR-5577
URL: https://issues.apache.org/jira/browse/SOLR-5577
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Christine Poerschke
Assignee: Mark Miller
Fix For: 5.0, 4.7, 4.6.1
Attachments: SOLR-5577.patch
The behaviour we observed was that a zookeeper election took about 2s plus
1.5s for reading the zoo_data snapshot. During this time solr tried to
establish connections to any zookeeper in the ensemble but only succeeded
once a leader was elected *and* that leader was done reading the snapshot.
Solr document updates were slowed down during this time window.
Is this expected to happen during and shortly after elections, that is
zookeeper closing existing connections, rejecting new connections and thus
stalling solr updates?
Other than avoiding zookeeper elections, are there ways of reducing their
impact on solr?
+zookeeper log extract+
{noformat}
08:18:54,968 [QuorumCnxManager.java:762] Connection broken for id ...
08:18:56,916 [Leader.java:345] LEADING - LEADER ELECTION TOOK - 1941
08:18:56,918 [FileSnap.java:83] Reading snapshot ...
...
08:18:57,010 [NIOServerCnxnFactory.java:197] Accepted socket connection from
...
08:18:57,010 [NIOServerCnxn.java:354] Exception causing close of session 0x0
due to java.io.IOException: ZooKeeperServer not running
08:18:57,010 [NIOServerCnxn.java:1001] Closed socket connection for client
... (no session established for client)
...
08:18:58,496 [FileTxnSnapLog.java:240] Snapshotting: ... to ...
{noformat}
+solr log extract+
{noformat}
08:18:54,968 [ClientCnxn.java:1085] Unable to read additional data from
server sessionid ... likely server has closed socket, closing socket
connection and attempting reconnect
08:18:55,068 [ConnectionManager.java:72] Watcher
org.apache.solr.common.cloud.ConnectionManager@... name:ZooKeeperConnection
Watcher:host1:port1,host2:port2,host3:port3,... got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
08:18:55,068 [ConnectionManager.java:132] zkClient has disconnected
...
08:18:55,961 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:55,961 [ClientCnxn.java:849] Socket connection established to ...
08:18:55,962 [ClientCnxn.java:1085] Unable to read additional data from
server sessionid ... likely server has closed socket, closing socket
connection and attempting reconnect
...
08:18:56,714 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:56,715 [ClientCnxn.java:849] Socket connection established to ...
08:18:56,715 [ClientCnxn.java:1085] Unable to read additional data from ...
...
08:18:57,640 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:57,641 [ClientCnxn.java:849] Socket connection established to ...
08:18:57,641 [ClientCnxn.java:1085] Unable to read additional data from ...
...
08:18:58,352 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:58,353 [ClientCnxn.java:849] Socket connection established to ...
08:18:58,353 [ClientCnxn.java:1085] Unable to read additional data from ...
...
08:18:58,749 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:58,749 [ClientCnxn.java:849] Socket connection established to ...
08:18:58,751 [ClientCnxn.java:1207] Session establishment complete on server
... sessionid = ..., negotiated timeout = ...
08:18:58,751 ... [ConnectionManager.java:72] Watcher
org.apache.solr.common.cloud.ConnectionManager@... name:ZooKeeperConnection
Watcher:host1:port1,host2:port2,host3:port3,... got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5577) indexing delay due to zookeeper election
[
https://issues.apache.org/jira/browse/SOLR-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882315#comment-13882315
]
ASF subversion and git services commented on SOLR-5577:
---
Commit 1561497 from [~markrmil...@gmail.com] in branch 'dev/trunk'
[ https://svn.apache.org/r1561497 ]
SOLR-5577: Harden leaking Timer thread.
indexing delay due to zookeeper election
Key: SOLR-5577
URL: https://issues.apache.org/jira/browse/SOLR-5577
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Christine Poerschke
Assignee: Mark Miller
Fix For: 5.0, 4.7, 4.6.1
Attachments: SOLR-5577.patch
The behaviour we observed was that a zookeeper election took about 2s plus
1.5s for reading the zoo_data snapshot. During this time solr tried to
establish connections to any zookeeper in the ensemble but only succeeded
once a leader was elected *and* that leader was done reading the snapshot.
Solr document updates were slowed down during this time window.
Is this expected to happen during and shortly after elections, that is
zookeeper closing existing connections, rejecting new connections and thus
stalling solr updates?
Other than avoiding zookeeper elections, are there ways of reducing their
impact on solr?
+zookeeper log extract+
{noformat}
08:18:54,968 [QuorumCnxManager.java:762] Connection broken for id ...
08:18:56,916 [Leader.java:345] LEADING - LEADER ELECTION TOOK - 1941
08:18:56,918 [FileSnap.java:83] Reading snapshot ...
...
08:18:57,010 [NIOServerCnxnFactory.java:197] Accepted socket connection from
...
08:18:57,010 [NIOServerCnxn.java:354] Exception causing close of session 0x0
due to java.io.IOException: ZooKeeperServer not running
08:18:57,010 [NIOServerCnxn.java:1001] Closed socket connection for client
... (no session established for client)
...
08:18:58,496 [FileTxnSnapLog.java:240] Snapshotting: ... to ...
{noformat}
+solr log extract+
{noformat}
08:18:54,968 [ClientCnxn.java:1085] Unable to read additional data from
server sessionid ... likely server has closed socket, closing socket
connection and attempting reconnect
08:18:55,068 [ConnectionManager.java:72] Watcher
org.apache.solr.common.cloud.ConnectionManager@... name:ZooKeeperConnection
Watcher:host1:port1,host2:port2,host3:port3,... got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
08:18:55,068 [ConnectionManager.java:132] zkClient has disconnected
...
08:18:55,961 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:55,961 [ClientCnxn.java:849] Socket connection established to ...
08:18:55,962 [ClientCnxn.java:1085] Unable to read additional data from
server sessionid ... likely server has closed socket, closing socket
connection and attempting reconnect
...
08:18:56,714 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:56,715 [ClientCnxn.java:849] Socket connection established to ...
08:18:56,715 [ClientCnxn.java:1085] Unable to read additional data from ...
...
08:18:57,640 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:57,641 [ClientCnxn.java:849] Socket connection established to ...
08:18:57,641 [ClientCnxn.java:1085] Unable to read additional data from ...
...
08:18:58,352 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:58,353 [ClientCnxn.java:849] Socket connection established to ...
08:18:58,353 [ClientCnxn.java:1085] Unable to read additional data from ...
...
08:18:58,749 [ClientCnxn.java:966] Opening socket connection to server ...
08:18:58,749 [ClientCnxn.java:849] Socket connection established to ...
08:18:58,751 [ClientCnxn.java:1207] Session establishment complete on server
... sessionid = ..., negotiated timeout = ...
08:18:58,751 ... [ConnectionManager.java:72] Watcher
org.apache.solr.common.cloud.ConnectionManager@... name:ZooKeeperConnection
Watcher:host1:port1,host2:port2,host3:port3,... got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-5665) Would like a 'REBUILD' method in the collections api to reconstruct missing cores
[ https://issues.apache.org/jira/browse/SOLR-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nathan Neulinger closed SOLR-5665. -- Resolution: Won't Fix Response indicates it'll be addressed with future functionality in a different way. Attached script is available if anyone wants it or you want to stick it in contrib. Would like a 'REBUILD' method in the collections api to reconstruct missing cores - Key: SOLR-5665 URL: https://issues.apache.org/jira/browse/SOLR-5665 Project: Solr Issue Type: New Feature Components: SolrCloud Reporter: Nathan Neulinger Attachments: recreate-solr-cores.pl Test Scenario: Multinode solrCloud deployment Completely delete one of those nodes Bring it back online empty (no cores, no indexes at all) Node will come up with each of the replicas in recovery mode and they will stay there forever since the cores don't actually exist. I've written a small external script that goes and gets the collections/shards/replicas-core mapping from clusterstate.json, and then calls the core create url for each core that is in a down state. This was relatively easy to implement externally - and should be even more trivial to implement inside of the collections api. I envision this to apply this to all collections: http://localhost:8983/solr/admin/collections?action=REBUILD and this to apply the operation to a single collection: http://localhost:8983/solr/admin/collections?action=REBUILDname=collname Independently, I'd like to see this sort of server is blank, recreate missing automatically triggered automatically, but I can see where that might not be expected behavior. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5661) PriorityQueue has OOM (Requested array size exceeds VM limit) issue
[
https://issues.apache.org/jira/browse/SOLR-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882338#comment-13882338
]
Shalin Shekhar Mangar commented on SOLR-5661:
-
{quote}
If you have multiple shard for one collection, send the query url for max
integer rowid, it can easy replicate.
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at org.apache.lucene.util.PriorityQueue.init(PriorityQueue.java:64)
at org.apache.lucene.util.PriorityQueue.init(PriorityQueue.java:37)
at
org.apache.solr.handler.component.ShardFieldSortedHitQueue.init(ShardDoc.java:113)
at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:790)
{quote}
Raintung, please correct me if I am wrong but the max integer value is being
used in the constructor only because you have explicitly asked for that many
rows? Solr doesn't pass such a big value to the PriorityQueue constructor by
itself.
Also, I think the fix committed to Lucene takes care of the problem on Solr's
side as well. Even if we implement a check on Solr's side, the error message
can't be any different.
PriorityQueue has OOM (Requested array size exceeds VM limit) issue
---
Key: SOLR-5661
URL: https://issues.apache.org/jira/browse/SOLR-5661
Project: Solr
Issue Type: Bug
Components: contrib - Solr Cell (Tika extraction)
Affects Versions: 4.3.1, 4.4, 4.5, 4.5.1, 4.6
Environment: JDK 7
Reporter: Raintung Li
Assignee: Michael McCandless
Fix For: 5.0, 4.7
Attachments: patch-5661.txt
It look like JDK7 change the design for max_array_length logic, it isn't
max_jint, and it should be max_jint - header_size(type).
If you deliver the Integer.MaxValue to create the PriorityQueue and have
enough memory, you will find it is ok in JVM6 but not work in JVM7.
JVM7 will throw OOM error while do array rang checking.
It should the compatible issue between JVM6 and JVM7.
Maybe need protect in the code logic, throw OOM look like big issues for
customer.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Areek Zillur as Lucene/Solr committer!
On 1/21/2014 12:26 PM, Robert Muir wrote: I'm pleased to announce that Areek Zillur has accepted to join our ranks as a committer. Areek has been improving suggester support in Lucene and Solr, including a revamped Solr component slated for the 4.7 release. [1] Areek, it is tradition that you introduce yourself with a brief bio. Congratulations and Welcome to this ACORN! (a company of rowdy nuts) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Benson Margulies as Lucene/Solr committer!
On 1/25/2014 2:40 PM, Michael McCandless wrote: I'm pleased to announce that Benson Margulies has accepted to join our ranks as a committer. Benson has been involved in a number of Lucene/Solr issues over time (see http://jirasearch.mikemccandless.com/search.py?index=jirachg=ddsa1=allUsersa2=Benson+Margulies ), most recently on debugging tricky analysis issues. Congratulations and welcome! One more to try and keep me in line. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5376) Add a demo search server
[ https://issues.apache.org/jira/browse/LUCENE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882374#comment-13882374 ] ASF subversion and git services commented on LUCENE-5376: - Commit 1561524 from [~mikemccand] in branch 'dev/branches/lucene5376' [ https://svn.apache.org/r1561524 ] LUCENE-5376: add some nocommits; cutover another pretty-print from jackson to json-smart; small simplification to replication Add a demo search server Key: LUCENE-5376 URL: https://issues.apache.org/jira/browse/LUCENE-5376 Project: Lucene - Core Issue Type: Improvement Reporter: Michael McCandless Assignee: Michael McCandless Attachments: lucene-demo-server.tgz I think it'd be useful to have a demo search server for Lucene. Rather than being fully featured, like Solr, it would be minimal, just wrapping the existing Lucene modules to show how you can make use of these features in a server setting. The purpose is to demonstrate how one can build a minimal search server on top of APIs like SearchManager, SearcherLifetimeManager, etc. This is also useful for finding rough edges / issues in Lucene's APIs that make building a server unnecessarily hard. I don't think it should have back compatibility promises (except Lucene's index back compatibility), so it's free to improve as Lucene's APIs change. As a starting point, I'll post what I built for the eating your own dog food search app for Lucene's Solr's jira issues http://jirasearch.mikemccandless.com (blog: http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html ). It uses Netty to expose basic indexing searching APIs via JSON, but it's very rough (lots nocommits). -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Benson Margulies as Lucene/Solr committer!
Welcome Benson! Joel Bernstein Search Engineer at Heliosearch On Sun, Jan 26, 2014 at 12:43 PM, Shawn Heisey s...@elyograg.org wrote: On 1/25/2014 2:40 PM, Michael McCandless wrote: I'm pleased to announce that Benson Margulies has accepted to join our ranks as a committer. Benson has been involved in a number of Lucene/Solr issues over time (see http://jirasearch.mikemccandless.com/search.py?index=jirachg=ddsa1=allUsersa2=Benson+Margulies ), most recently on debugging tricky analysis issues. Congratulations and welcome! One more to try and keep me in line. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 502 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/502/ All tests passed Build Log: [...truncated 10560 lines...] [junit4] JVM J1: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20140126_172441_790.sysout [junit4] JVM J1: stdout (verbatim) [junit4] java.lang.OutOfMemoryError: Java heap space [junit4] Dumping heap to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps/java_pid99633.hprof ... [junit4] Heap dump file created [441108206 bytes in 2.759 secs] [junit4] JVM J1: EOF [junit4] JVM J1: stderr was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20140126_172441_790.syserr [junit4] JVM J1: stderr (verbatim) [junit4] WARN: Unhandled exception in event serialization. - java.lang.OutOfMemoryError: GC overhead limit exceeded [junit4] [junit4] WARN: Event serializer exception. - java.io.IOException: Serializer already closed. [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:41) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.RunListenerEmitter.testFailure(RunListenerEmitter.java:54) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.NoExceptionRunListenerDecorator.testFailure(NoExceptionRunListenerDecorator.java:55) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.BeforeAfterRunListenerDecorator.testFailure(BeforeAfterRunListenerDecorator.java:60) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$4.notifyListener(OrderedRunNotifier.java:129) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier$SafeNotifier.run(OrderedRunNotifier.java:63) [junit4] at com.carrotsearch.ant.tasks.junit4.slave.OrderedRunNotifier.fireTestFailure(OrderedRunNotifier.java:126) [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$SubNotifier.fireTestFailure(ThreadLeakControl.java:179) [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:664) [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:661) [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.fireTestFailure(RandomizedRunner.java:661) [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:752) [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) [junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) [junit4] at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) [junit4] at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4] at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4] at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [junit4] at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) [junit4] at java.lang.Thread.run(Thread.java:724) [junit4] [junit4] WARN: Event serializer exception. - java.io.IOException: Serializer already closed. [junit4] at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:41) [junit4] at
[jira] [Commented] (LUCENE-5376) Add a demo search server
[ https://issues.apache.org/jira/browse/LUCENE-5376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882382#comment-13882382 ] ASF subversion and git services commented on LUCENE-5376: - Commit 1561529 from [~mikemccand] in branch 'dev/branches/lucene5376' [ https://svn.apache.org/r1561529 ] LUCENE-5376: relax type DocsEnum - DocIdSetIterator; fix typos Add a demo search server Key: LUCENE-5376 URL: https://issues.apache.org/jira/browse/LUCENE-5376 Project: Lucene - Core Issue Type: Improvement Reporter: Michael McCandless Assignee: Michael McCandless Attachments: lucene-demo-server.tgz I think it'd be useful to have a demo search server for Lucene. Rather than being fully featured, like Solr, it would be minimal, just wrapping the existing Lucene modules to show how you can make use of these features in a server setting. The purpose is to demonstrate how one can build a minimal search server on top of APIs like SearchManager, SearcherLifetimeManager, etc. This is also useful for finding rough edges / issues in Lucene's APIs that make building a server unnecessarily hard. I don't think it should have back compatibility promises (except Lucene's index back compatibility), so it's free to improve as Lucene's APIs change. As a starting point, I'll post what I built for the eating your own dog food search app for Lucene's Solr's jira issues http://jirasearch.mikemccandless.com (blog: http://blog.mikemccandless.com/2013/05/eating-dog-food-with-lucene.html ). It uses Netty to expose basic indexing searching APIs via JSON, but it's very rough (lots nocommits). -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-4.6.1-Linux-Java6-64-test-only - Build # 1322 - Failure!
Build: builds.flonkings.com/job/Lucene-4.6.1-Linux-Java6-64-test-only/1322/ No tests ran. Build Log: [...truncated 14 lines...] ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_6 org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_4_6 failed at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387) at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:910) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:891) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:874) at hudson.FilePath.act(FilePath.java:914) at hudson.FilePath.act(FilePath.java:887) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:850) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:788) at hudson.model.AbstractProject.checkout(AbstractProject.java:1414) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561) at hudson.model.Run.execute(Run.java:1678) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_4_6 failed at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97) ... 38 more Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_4_6' svn: E175002: timed out waiting for server at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382) ... 37 more Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_4_6' at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775) ... 38 more Caused by: svn: E175002: timed out
[JENKINS] Lucene-4x-Linux-Java6-64-test-only - Build # 11226 - Failure!
Build: builds.flonkings.com/job/Lucene-4x-Linux-Java6-64-test-only/11226/ No tests ran. Build Log: [...truncated 13 lines...] ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/branch_4x org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_4x failed at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387) at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:910) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:891) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:874) at hudson.FilePath.act(FilePath.java:914) at hudson.FilePath.act(FilePath.java:887) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:850) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:788) at hudson.model.AbstractProject.checkout(AbstractProject.java:1414) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561) at hudson.model.Run.execute(Run.java:1678) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_4x failed at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97) ... 38 more Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_4x' svn: E175002: timed out waiting for server at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382) ... 37 more Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_4x' at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775) ... 38 more Caused by: svn: E175002: timed out waiting for server at
[JENKINS] Lucene-trunk-Linux-Java7-64-test-only - Build # 74903 - Failure!
Build: builds.flonkings.com/job/Lucene-trunk-Linux-Java7-64-test-only/74903/ No tests ran. Build Log: [...truncated 12 lines...] ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/trunk org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387) at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:910) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:891) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:874) at hudson.FilePath.act(FilePath.java:914) at hudson.FilePath.act(FilePath.java:887) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:850) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:788) at hudson.model.AbstractProject.checkout(AbstractProject.java:1414) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561) at hudson.model.Run.execute(Run.java:1678) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/trunk failed at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97) ... 38 more Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk' svn: E175002: timed out waiting for server at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382) ... 37 more Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/trunk' at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775) ... 38 more Caused by: svn: E175002: timed out waiting for server at
[JENKINS] Lucene-4.6.1-Linux-Java6-64-test-only - Build # 1323 - Still Failing!
Build: builds.flonkings.com/job/Lucene-4.6.1-Linux-Java6-64-test-only/1323/ No tests ran. Build Log: [...truncated 5 lines...] ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_6 org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_4_6 failed at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339) at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364) at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27) at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387) at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:910) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:891) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:874) at hudson.FilePath.act(FilePath.java:914) at hudson.FilePath.act(FilePath.java:887) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:850) at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:788) at hudson.model.AbstractProject.checkout(AbstractProject.java:1414) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561) at hudson.model.Run.execute(Run.java:1678) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:231) Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/lucene_solr_4_6 failed at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97) ... 38 more Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_4_6' svn: E175002: timed out waiting for server at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382) ... 37 more Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/lucene_solr_4_6' at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775) ... 38 more Caused by: svn: E175002: timed out
[jira] [Created] (SOLR-5666) Using the hdfs write cache can result in appearance of corrupted index.
Mark Miller created SOLR-5666: - Summary: Using the hdfs write cache can result in appearance of corrupted index. Key: SOLR-5666 URL: https://issues.apache.org/jira/browse/SOLR-5666 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Fix For: 5.0, 4.7 -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5666) Using the hdfs write cache can result in appearance of corrupted index.
[ https://issues.apache.org/jira/browse/SOLR-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882479#comment-13882479 ] Mark Miller commented on SOLR-5666: --- This looks like it is the write cache. Workaround is to disable the write cache in HdfsDirectoryFactory: solr.hdfs.blockcache.write.enabled=false Index is not actually corrupt on disk, you can restart on it fine. Using the hdfs write cache can result in appearance of corrupted index. --- Key: SOLR-5666 URL: https://issues.apache.org/jira/browse/SOLR-5666 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Fix For: 5.0, 4.7 -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5477) Async execution of OverseerCollectionProcessor tasks
[ https://issues.apache.org/jira/browse/SOLR-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anshum Gupta updated SOLR-5477: --- Attachment: SOLR-5477.patch Fixed an issue with task tracking. The patch has failing tests as the CoreAdminHandler.shutdown() is called at the wrong place/time. Working on correcting it. Async execution of OverseerCollectionProcessor tasks Key: SOLR-5477 URL: https://issues.apache.org/jira/browse/SOLR-5477 Project: Solr Issue Type: Sub-task Components: SolrCloud Reporter: Noble Paul Assignee: Anshum Gupta Attachments: SOLR-5477-CoreAdminStatus.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch Typical collection admin commands are long running and it is very common to have the requests get timed out. It is more of a problem if the cluster is very large.Add an option to run these commands asynchronously add an extra param async=true for all collection commands the task is written to ZK and the caller is returned a task id. as separate collection admin command will be added to poll the status of the task command=statusid=7657668909 if id is not passed all running async tasks should be listed A separate queue is created to store in-process tasks . After the tasks are completed the queue entry is removed. OverSeerColectionProcessor will perform these tasks in multiple threads -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5463) Provide cursor/token based searchAfter support that works with arbitrary sorting (ie: deep paging)
[
https://issues.apache.org/jira/browse/SOLR-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882481#comment-13882481
]
Yonik Seeley commented on SOLR-5463:
Nice work guys!
Some further thoughts:
We should consider allowing non-zero start parameters with cursorMark. The
primary use case is when someone is skipping pages (perhaps trying to get to a
different section of results, or trying to get much later in a time based
search, or just viewing the long tail).
For example, a user at page 50 clicks on page 60. It would be nice to support
this by just specifying start=90 (i.e. 600-510) assuming 10 docs per page,
along with the normal cursorMark (that would have started at page 51 / doc
510). Currently, the prohibition on non-zero start parameters would mean that
we would either have to abandon cursoring altogether, or we would have to
actually retrieve 100 documents to continue it.
The other thought is around how to do reverse paging efficiently. One way is
to save previous cursorMarks on the client side and just return to them if one
wants to page backwards. The other potential way is to reverse the sort
parameters and use the current cursorMark. The only pitfall to this approach
is that you don't get the current document you are on (because we
searchAfter).
Provide cursor/token based searchAfter support that works with arbitrary
sorting (ie: deep paging)
--
Key: SOLR-5463
URL: https://issues.apache.org/jira/browse/SOLR-5463
Project: Solr
Issue Type: New Feature
Reporter: Hoss Man
Assignee: Hoss Man
Fix For: 5.0, 4.7
Attachments: SOLR-5463-randomized-faceting-test.patch,
SOLR-5463.patch, SOLR-5463.patch, SOLR-5463.patch, SOLR-5463.patch,
SOLR-5463.patch, SOLR-5463.patch, SOLR-5463.patch,
SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch,
SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch,
SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch,
SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch,
SOLR-5463__straw_man.patch, SOLR-5463__straw_man.patch,
SOLR-5463__straw_man__MissingStringLastComparatorSource.patch
I'd like to revist a solution to the problem of deep paging in Solr,
leveraging an HTTP based API similar to how IndexSearcher.searchAfter works
at the lucene level: require the clients to provide back a token indicating
the sort values of the last document seen on the previous page. This is
similar to the cursor model I've seen in several other REST APIs that
support pagnation over a large sets of results (notable the twitter API and
it's since_id param) except that we'll want something that works with
arbitrary multi-level sort critera that can be either ascending or descending.
SOLR-1726 laid some initial ground work here and was commited quite a while
ago, but the key bit of argument parsing to leverage it was commented out due
to some problems (see comments in that issue). It's also somewhat out of
date at this point: at the time it was commited, IndexSearcher only supported
searchAfter for simple scores, not arbitrary field sorts; and the params
added in SOLR-1726 suffer from this limitation as well.
---
I think it would make sense to start fresh with a new issue with a focus on
ensuring that we have deep paging which:
* supports arbitrary field sorts in addition to sorting by score
* works in distributed mode
{panel:title=Basic Usage}
* send a request with {{sort=Xstart=0rows=NcursorMark=*}}
** sort can be anything, but must include the uniqueKey field (as a tie
breaker)
** N can be any number you want per page
** start must be 0
** \* denotes you want to use a cursor starting at the beginning mark
* parse the response body and extract the (String) {{nextCursorMark}} value
* Replace the \* value in your initial request params with the
{{nextCursorMark}} value from the response in the subsequent request
* repeat until the {{nextCursorMark}} value stops changing, or you have
collected as many docs as you need
{panel}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5477) Async execution of OverseerCollectionProcessor tasks
[ https://issues.apache.org/jira/browse/SOLR-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anshum Gupta updated SOLR-5477: --- Attachment: (was: SOLR-5477.patch) Async execution of OverseerCollectionProcessor tasks Key: SOLR-5477 URL: https://issues.apache.org/jira/browse/SOLR-5477 Project: Solr Issue Type: Sub-task Components: SolrCloud Reporter: Noble Paul Assignee: Anshum Gupta Attachments: SOLR-5477-CoreAdminStatus.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch Typical collection admin commands are long running and it is very common to have the requests get timed out. It is more of a problem if the cluster is very large.Add an option to run these commands asynchronously add an extra param async=true for all collection commands the task is written to ZK and the caller is returned a task id. as separate collection admin command will be added to poll the status of the task command=statusid=7657668909 if id is not passed all running async tasks should be listed A separate queue is created to store in-process tasks . After the tasks are completed the queue entry is removed. OverSeerColectionProcessor will perform these tasks in multiple threads -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5477) Async execution of OverseerCollectionProcessor tasks
[ https://issues.apache.org/jira/browse/SOLR-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anshum Gupta updated SOLR-5477: --- Attachment: SOLR-5477.patch Async execution of OverseerCollectionProcessor tasks Key: SOLR-5477 URL: https://issues.apache.org/jira/browse/SOLR-5477 Project: Solr Issue Type: Sub-task Components: SolrCloud Reporter: Noble Paul Assignee: Anshum Gupta Attachments: SOLR-5477-CoreAdminStatus.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch Typical collection admin commands are long running and it is very common to have the requests get timed out. It is more of a problem if the cluster is very large.Add an option to run these commands asynchronously add an extra param async=true for all collection commands the task is written to ZK and the caller is returned a task id. as separate collection admin command will be added to poll the status of the task command=statusid=7657668909 if id is not passed all running async tasks should be listed A separate queue is created to store in-process tasks . After the tasks are completed the queue entry is removed. OverSeerColectionProcessor will perform these tasks in multiple threads -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Benson Margulies as Lucene/Solr committer!
Welcome, Benson! koji -- http://soleami.com/blog/mahout-and-machine-learning-training-course-is-here.html (14/01/26 6:40), Michael McCandless wrote: I'm pleased to announce that Benson Margulies has accepted to join our ranks as a committer. Benson has been involved in a number of Lucene/Solr issues over time (see http://jirasearch.mikemccandless.com/search.py?index=jirachg=ddsa1=allUsersa2=Benson+Margulies ), most recently on debugging tricky analysis issues. Benson, it is tradition that you introduce yourself with a brief bio. I know you're heavily involved in other Apache projects already... Once your account is set up, you should then be able to add yourself to the who we are page on the website as well. Congratulations and welcome! Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1301) Add a Solr contrib that allows for building Solr indexes via Hadoop's Map-Reduce.
[ https://issues.apache.org/jira/browse/SOLR-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882522#comment-13882522 ] Mark Miller commented on SOLR-1301: --- Major performance issue that relates to this: SOLR-5667 Performance problem when not using hdfs block cache. Add a Solr contrib that allows for building Solr indexes via Hadoop's Map-Reduce. - Key: SOLR-1301 URL: https://issues.apache.org/jira/browse/SOLR-1301 Project: Solr Issue Type: New Feature Reporter: Andrzej Bialecki Assignee: Mark Miller Fix For: 5.0, 4.7 Attachments: README.txt, SOLR-1301-hadoop-0-20.patch, SOLR-1301-hadoop-0-20.patch, SOLR-1301-maven-intellij.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SOLR-1301.patch, SolrRecordWriter.java, commons-logging-1.0.4.jar, commons-logging-api-1.0.4.jar, hadoop-0.19.1-core.jar, hadoop-0.20.1-core.jar, hadoop-core-0.20.2-cdh3u3.jar, hadoop.patch, log4j-1.2.15.jar This patch contains a contrib module that provides distributed indexing (using Hadoop) to Solr EmbeddedSolrServer. The idea behind this module is twofold: * provide an API that is familiar to Hadoop developers, i.e. that of OutputFormat * avoid unnecessary export and (de)serialization of data maintained on HDFS. SolrOutputFormat consumes data produced by reduce tasks directly, without storing it in intermediate files. Furthermore, by using an EmbeddedSolrServer, the indexing task is split into as many parts as there are reducers, and the data to be indexed is not sent over the network. Design -- Key/value pairs produced by reduce tasks are passed to SolrOutputFormat, which in turn uses SolrRecordWriter to write this data. SolrRecordWriter instantiates an EmbeddedSolrServer, and it also instantiates an implementation of SolrDocumentConverter, which is responsible for turning Hadoop (key, value) into a SolrInputDocument. This data is then added to a batch, which is periodically submitted to EmbeddedSolrServer. When reduce task completes, and the OutputFormat is closed, SolrRecordWriter calls commit() and optimize() on the EmbeddedSolrServer. The API provides facilities to specify an arbitrary existing solr.home directory, from which the conf/ and lib/ files will be taken. This process results in the creation of as many partial Solr home directories as there were reduce tasks. The output shards are placed in the output directory on the default filesystem (e.g. HDFS). Such part-N directories can be used to run N shard servers. Additionally, users can specify the number of reduce tasks, in particular 1 reduce task, in which case the output will consist of a single shard. An example application is provided that processes large CSV files and uses this API. It uses a custom CSV processing to avoid (de)serialization overhead. This patch relies on hadoop-core-0.19.1.jar - I attached the jar to this issue, you should put it in contrib/hadoop/lib. Note: the development of this patch was sponsored by an anonymous contributor and approved for release under Apache License. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5667) Performance problem when not using hdfs block cache.
Mark Miller created SOLR-5667: - Summary: Performance problem when not using hdfs block cache. Key: SOLR-5667 URL: https://issues.apache.org/jira/browse/SOLR-5667 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Fix For: 5.0, 4.7 Reads are terribly slow due to what looks like a too small buffer in the buffered input. This has consequences for SOLR-1301, as we turn off the block cache when writing indexes with map-reduce. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Benson Margulies as Lucene/Solr committer!
Welcome, Benson! Glad to have you aboard. Erik On Jan 25, 2014, at 5:10 PM, Benson Margulies bimargul...@gmail.com wrote: Hello Lucene development community, it's a pleasure to be welcomed aboard. In my view, the significant aspect of my bio is that I've been implementing things that go into or around Lucene for many years now. During the 'day', I'm the CTO of a company that works in the area of text analytics. We build Tokenizers and TokenFilters to allow our users to integrate our components into Lucene, and we've used Lucene and Solr as components of NLP devices that search on a large scale. So I have an abiding interest in the analysis chain and in the intersection of NLP and search. Elsewhere in Apache, I'm an active Maven dev, a semi-retired CXF dev, and a sort of uncle of several other projects. So I'm prone to be helpful or annoying with issues of Maven and Web Services. Thanks again, benson p.s. I think Uwe has already added me to the necessary wiring; would some kind soul please point me to the explanation of how the web site is maintained so I can add myself? Is it just the ASF CMS? On Sat, Jan 25, 2014 at 4:40 PM, Michael McCandless luc...@mikemccandless.com wrote: I'm pleased to announce that Benson Margulies has accepted to join our ranks as a committer. Benson has been involved in a number of Lucene/Solr issues over time (see http://jirasearch.mikemccandless.com/search.py?index=jirachg=ddsa1=allUsersa2=Benson+Margulies ), most recently on debugging tricky analysis issues. Benson, it is tradition that you introduce yourself with a brief bio. I know you're heavily involved in other Apache projects already... Once your account is set up, you should then be able to add yourself to the who we are page on the website as well. Congratulations and welcome! Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5666) Using the hdfs write cache can result in appearance of corrupted index.
[ https://issues.apache.org/jira/browse/SOLR-5666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882528#comment-13882528 ] Mark Miller commented on SOLR-5666: --- Grabbing the updated BufferStore stuff from https://issues.apache.org/jira/browse/BLUR-294 seem to fix this issue. Using the hdfs write cache can result in appearance of corrupted index. --- Key: SOLR-5666 URL: https://issues.apache.org/jira/browse/SOLR-5666 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Fix For: 5.0, 4.7 -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5661) PriorityQueue has OOM (Requested array size exceeds VM limit) issue
[ https://issues.apache.org/jira/browse/SOLR-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882535#comment-13882535 ] Raintung Li commented on SOLR-5661: --- Yes, I ask the many rows that it is max integer. The root node will collect the results from different node(shards) to combin and meger the results. For this case, will directly create the queue . Fix the Lucene side, it is enough. For solr issue, should be how to handle the biggest rowid, for biggest rowid should have the different logic. Can't direct create the biggest queue. PriorityQueue has OOM (Requested array size exceeds VM limit) issue --- Key: SOLR-5661 URL: https://issues.apache.org/jira/browse/SOLR-5661 Project: Solr Issue Type: Bug Components: contrib - Solr Cell (Tika extraction) Affects Versions: 4.3.1, 4.4, 4.5, 4.5.1, 4.6 Environment: JDK 7 Reporter: Raintung Li Assignee: Michael McCandless Fix For: 5.0, 4.7 Attachments: patch-5661.txt It look like JDK7 change the design for max_array_length logic, it isn't max_jint, and it should be max_jint - header_size(type). If you deliver the Integer.MaxValue to create the PriorityQueue and have enough memory, you will find it is ok in JVM6 but not work in JVM7. JVM7 will throw OOM error while do array rang checking. It should the compatible issue between JVM6 and JVM7. Maybe need protect in the code logic, throw OOM look like big issues for customer. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-5668) Solr map-reduce should try and turn on hard auto commit by default to deal with the transaction log.
Mark Miller created SOLR-5668: - Summary: Solr map-reduce should try and turn on hard auto commit by default to deal with the transaction log. Key: SOLR-5668 URL: https://issues.apache.org/jira/browse/SOLR-5668 Project: Solr Issue Type: Bug Reporter: Mark Miller Assignee: Mark Miller Fix For: 5.0, 4.7 We still count on the transaction log to generate versions for us. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b124) - Build # 9150 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9150/ Java: 32bit/jdk1.8.0-ea-b124 -client -XX:+UseSerialGC 1 tests failed. FAILED: org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch Error Message: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:55187 within 3 ms Stack Trace: java.lang.RuntimeException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:55187 within 3 ms at __randomizedtesting.SeedInfo.seed([88DA8DDADFFEF3BE:93C03C2A8A19382]:0) at org.apache.solr.common.cloud.SolrZkClient.init(SolrZkClient.java:147) at org.apache.solr.common.cloud.SolrZkClient.init(SolrZkClient.java:98) at org.apache.solr.common.cloud.SolrZkClient.init(SolrZkClient.java:93) at org.apache.solr.common.cloud.SolrZkClient.init(SolrZkClient.java:84) at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:89) at org.apache.solr.cloud.AbstractZkTestCase.buildZooKeeper(AbstractZkTestCase.java:83) at org.apache.solr.cloud.AbstractDistribZkTestBase.setUp(AbstractDistribZkTestBase.java:70) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.setUp(AbstractFullDistribZkTestBase.java:198) at org.apache.solr.client.solrj.impl.CloudSolrServerTest.setUp(CloudSolrServerTest.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:771) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at
[jira] [Updated] (SOLR-5477) Async execution of OverseerCollectionProcessor tasks
[ https://issues.apache.org/jira/browse/SOLR-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anshum Gupta updated SOLR-5477: --- Attachment: SOLR-5477.patch Patch without the ThreadPool. Fixed other issues - Exceptions in OCP and CoreAdminHandler. Response for CoreAdmin status request now contains: * responseHeader if successful. * Exception message if failure. Async execution of OverseerCollectionProcessor tasks Key: SOLR-5477 URL: https://issues.apache.org/jira/browse/SOLR-5477 Project: Solr Issue Type: Sub-task Components: SolrCloud Reporter: Noble Paul Assignee: Anshum Gupta Attachments: SOLR-5477-CoreAdminStatus.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch, SOLR-5477.patch Typical collection admin commands are long running and it is very common to have the requests get timed out. It is more of a problem if the cluster is very large.Add an option to run these commands asynchronously add an extra param async=true for all collection commands the task is written to ZK and the caller is returned a task id. as separate collection admin command will be added to poll the status of the task command=statusid=7657668909 if id is not passed all running async tasks should be listed A separate queue is created to store in-process tasks . After the tasks are completed the queue entry is removed. OverSeerColectionProcessor will perform these tasks in multiple threads -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS-MAVEN] Lucene-Solr-Maven-4.x #572: POMs out of sync
Build: https://builds.apache.org/job/Lucene-Solr-Maven-4.x/572/ 2 tests failed. REGRESSION: org.apache.solr.morphlines.cell.SolrCellMorphlineTest.testSolrCellDocumentTypes Error Message: key:ignored_creation_date expected:[2011-09-02T10:11:00Z] but was:[๒๐๑๑-๐๙-๐๒T๑๐:๑๑:๐๐Z] Stack Trace: java.lang.AssertionError: key:ignored_creation_date expected:[2011-09-02T10:11:00Z] but was:[๒๐๑๑-๐๙-๐๒T๑๐:๑๑:๐๐Z] at __randomizedtesting.SeedInfo.seed([F55D1C67AEBC0AA:959672581F309E7F]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.apache.solr.morphlines.solr.AbstractSolrMorphlineTestBase.testDocumentTypesInternal(AbstractSolrMorphlineTestBase.java:165) at org.apache.solr.morphlines.cell.SolrCellMorphlineTest.testSolrCellDocumentTypes(SolrCellMorphlineTest.java:193) REGRESSION: org.apache.solr.morphlines.cell.SolrCellMorphlineTest.testSolrCellDocumentTypes2 Error Message: key:ignored_creation_date expected:[2007-10-01T16:13:56Z] but was:[๑๔๖๔-๑๐-๐๑T๑๖:๑๓:๕๖Z] Stack Trace: java.lang.AssertionError: key:ignored_creation_date expected:[2007-10-01T16:13:56Z] but was:[๑๔๖๔-๑๐-๐๑T๑๖:๑๓:๕๖Z] at __randomizedtesting.SeedInfo.seed([F55D1C67AEBC0AA:84D56A267D1D5354]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.apache.solr.morphlines.solr.AbstractSolrMorphlineTestBase.testDocumentTypesInternal(AbstractSolrMorphlineTestBase.java:165) at org.apache.solr.morphlines.cell.SolrCellMorphlineTest.testSolrCellDocumentTypes2(SolrCellMorphlineTest.java:255) Build Log: [...truncated 52184 lines...] BUILD FAILED /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-4.x/build.xml:482: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-4.x/build.xml:176: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Maven-4.x/extra-targets.xml:77: Java returned: 1 Total time: 107 minutes 41 seconds Build step 'Invoke Ant' marked build as failure Recording test results Email was triggered for: Failure Sending email for trigger: Failure - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-5404) Add support to get number of entries a Suggester Lookup was built with and minor refactorings
[ https://issues.apache.org/jira/browse/LUCENE-5404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13882575#comment-13882575 ] Robert Muir commented on LUCENE-5404: - I guess mainly I wonder if we should just add it to the Lookup api directly (rather than a return value from build()). This would probably remove the necessity of having it in the iterator API (the lookup could just count, and then write the count into its data file if need be). Otherwise a lookup would only be able to provide this information when build() was called (rather than when it was loaded from e.g. a stored FST). Add support to get number of entries a Suggester Lookup was built with and minor refactorings - Key: LUCENE-5404 URL: https://issues.apache.org/jira/browse/LUCENE-5404 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Areek Zillur Fix For: 5.0, 4.7 Attachments: LUCENE-5404.patch, LUCENE-5404.patch It would be nice to be able to tell the number of entries a suggester lookup was built with. This would let components using lookups to keep some stats regarding how many entries were used to build a lookup. Additionally, Dictionary could use InputIterator rather than the BytesRefIteratator, as most of the implmentations now use it. -- This message was sent by Atlassian JIRA (v6.1.5#6160) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
