[jira] [Commented] (LUCENE-5168) ByteSliceReader assert trips with 32-bit oracle 1.7.0_25 + G1GC

[Dawid Weiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344694#comment-14344694
 ] 

Dawid Weiss commented on LUCENE-5168:
-

G1GC causes intermittent crashes in Lucene/ Solr tests. We don't know why. The 
crashes are hard or impossible to reproduce (and if they are reproducible, 
they're not easy to fix at hotspot level). I wouldn't use it in production.

> ByteSliceReader assert trips with 32-bit oracle 1.7.0_25 + G1GC
> ---
>
> Key: LUCENE-5168
> URL: https://issues.apache.org/jira/browse/LUCENE-5168
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Attachments: java8-windows-4x-3075-console.txt, log.0025, log.0042, 
> log.0078, log.0086, log.0100
>
>
> This assertion trips (sometimes from different tests), if you run the 
> highlighting tests on branch_4x with r1512807.
> It reproduces about half the time, always only with 32bit + G1GC (other 
> combinations do not seem to trip it, i didnt try looping or anything really 
> though).
> {noformat}
> rmuir@beast:~/workspace/branch_4x$ svn up -r 1512807
> rmuir@beast:~/workspace/branch_4x$ ant clean
> rmuir@beast:~/workspace/branch_4x$ rm -rf .caches #this is important,
> otherwise master seed does not work!
> rmuir@beast:~/workspace/branch_4x/lucene/highlighter$ ant test
> -Dtests.jvms=2 -Dtests.seed=EBBFA6F4E80A7365 -Dargs="-server
> -XX:+UseG1GC"
> {noformat}
> Originally showed up like this:
> {noformat}
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6874/
> Java: 32bit/jdk1.7.0_25 -server -XX:+UseG1GC
> 1 tests failed.
> REGRESSION:  
> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testUserFailedToIndexOffsets
> Error Message:
> Stack Trace:
> java.lang.AssertionError
> at 
> __randomizedtesting.SeedInfo.seed([EBBFA6F4E80A7365:1FBF811885F2D611]:0)
> at 
> org.apache.lucene.index.ByteSliceReader.readByte(ByteSliceReader.java:73)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108)
> at 
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:453)
> at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
> at org.apache.lucene.index.TermsHash.flush(TermsHash.java:116)
> at org.apache.lucene.index.DocInverter.flush(DocInverter.java:53)
> at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
> at 
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:501)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6322) IndexSearcher.doc(int docID, SetfieldsToLoad) is slower in Lucene 4.9 when compared to Lucene 2.9

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344673#comment-14344673
 ] 

Mikhail Khludnev commented on LUCENE-6322:
--

Sekhar,
Can you revise your storage scheme and move to binary docvalues, or so?  

> IndexSearcher.doc(int docID, SetfieldsToLoad)  is slower in Lucene 4.9 when 
> compared to Lucene 2.9
> --
>
> Key: LUCENE-6322
> URL: https://issues.apache.org/jira/browse/LUCENE-6322
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 4.9
> Environment: Windows, JDK 7/8
>Reporter: Sekhar
> Fix For: 4.10.x
>
>
> We use IndexSearcher.doc(int docID, SetfieldsToLoad) method to get the 
> document with selected stored fields. If we did not mention few stored fields 
> which have data more than 500KB, this call is slower in Lucene 4.9 when 
> compared to Lucene 2.9.
> I debugged the above method with Lucene 4.9 and found that 
> CompressingStoredFieldsReader#visitDocument(int docID, StoredFieldVisitor 
> visitor) is spending more time while loading file content and decompressing 
> in chunks of 16kb, even to skip the fields. It is noticeable degrade if the 
> document's field size is more than 1MB, and we call this method in loop for 
> more than 1000 such documents.
> In case of Lucene 2.9, there was no compression, and if we want to skip the 
> field, it just does file seek to set the next pointer to read the stored 
> field. For example see Lucene3xStoredFieldsReader#skipField() method how it 
> works for skipping a field in Lucene 2.9 which is VERY faster compared to 
> Lucene 4.9.
> We should have something in CompressingStoredFieldsReader to know the field’s 
> compressed length in file and just do the file seek to set the next pointer 
> instead of loading content from file and decompress that in 16KB chunks to 
> just skip the field from the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Noble Paul]

Awesome ,

It was long overdue
welcome ramkumar

On Tue, Mar 3, 2015 at 12:39 AM, Tomás Fernández Löbbe <
tomasflo...@gmail.com> wrote:

> Welcome Ramkumar!
>
> On Mon, Mar 2, 2015 at 8:01 AM, Ryan Ernst  wrote:
>
>> Welcome!
>> On Mar 1, 2015 8:40 PM, "Shalin Shekhar Mangar" 
>> wrote:
>>
>>> I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's
>>> invitation to become a committer.
>>>
>>> Ramkumar, it's tradition that you introduce yourself with a brief bio.
>>>
>>> Your handle "andyetitmoves" has already added to the “lucene" LDAP
>>> group, so you now have commit privileges. Please test this by adding
>>> yourself to the committers section of the Who We Are page on the website: <
>>> http://lucene.apache.org/whoweare.html> (use the ASF CMS bookmarklet at
>>> the bottom of the page here:  - more
>>> info here ).
>>>
>>> The ASF dev page also has lots of useful links: <
>>> http://www.apache.org/dev/>.
>>>
>>> Congratulations and welcome!
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>
>


-- 
-
Noble Paul

[jira] [Updated] (SOLR-7126) signing a jar and secure dynamic loading

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul updated SOLR-7126:
-
Description: 
We need to ensure that the jars loaded into solr are trusted 

We shall use simple PKI to protect the jars/config loaded into the system
The following are the steps involved for doing that.
{noformat}
#Step 1:
# generate a 768-bit RSA private key. or whaterver strength you would need
$ openssl genrsa -out priv_key.pem 768
# store your private keys safely (with  a password if possible)

# output public key portion in DER format (so that Java can read it)
$ openssl rsa -in priv_key.pem -pubout -outform DER -out pub_key.der

#Step 2:
#Load the .DER files to ZK under /keys or
# copy the public keys (the .DER files) to all Solr nodes under SOLR_HOME/keys 
. or, 
# Please note that you can store multiple public keys in that directory and all 
are valid

Step3:
# start all your servers with -Denable.runtime.lib=true 

Step 4:
# sign the sha1 digest of your jar with one of your private keys and get the 
base64 string of that signature . 
$ openssl dgst -sha1 -sign priv_key.pem myjar.jar | openssl enc -base64 

#Step 5:
# load your jars into blob store . refer SOLR-6787

#Step 6:
# use the command to add your jar to classpath as follows
{noformat}
{code}
curl http://localhost:8983/solr/collection1/config -H 
'Content-type:application/json'  -d '{
"add-runtimelib" : {"name": "jarname" , "version":2 , 
"sig":"mW1Gwtz2QazjfVdrLFHfbGwcr8xzFYgUOLu68LHqWRDvLG0uLcy1McQ+AzVmeZFBf1yLPDEHBWJb5KXr8bdbHN/PYgUB1nsr9pk4EFyD9KfJ8TqeH/ijQ9waa/vjqyiKEI9U550EtSzruLVZ32wJ7smvV0fj2YYhrUaaPzOn9g0="
 }// output of step 4. concatenate the lines 

}' 
{code}

sig is the extra parameter that is nothing but the base64 encoded value of the 
jar's sha1 signature 

If no keys are present , the jar is loaded without any checking. 

Before loading a jar from blob store , each Solr node would check if there are 
keys present in the keys directory. If yes, each jar's signature will be 
verified with all the available public keys. If atleast one succeeds , the jar 
is loaded into memory. If nothing succeeds , it will be rejected 

  was:
We need to ensure that the jars loaded into solr are trusted 

We shall use simple PKI to protect the jars/config loaded into the system
The following are the steps involved for doing that.
{noformat}
#Step 1:
# generate a 768-bit RSA private key. or whaterver strength you would need
$ openssl genrsa -out priv_key.pem 768
# store your private keys safely (with  a password if possible)

# output public key portion in DER format (so that Java can read it)
$ openssl rsa -in priv_key.pem -pubout -outform DER -out pub_key.der

#Step 2:
#Load the .DER files to ZK under /keys or
# copy the public keys (the .DER files) to all Solr nodes under SOLR_HOME/keys 
. or, 
# start all your solr servers with -Dpublic.keys.dir=/location/of/keys (where 
keys are stored). 
# Please note that you can store multiple public keys in that directory and all 
are valid

Step3:
# start all your servers with -Denable.dynamic.loading=true 

Step 4:
# sign the sha1 digest of your jar with one of your private keys and get the 
base64 string of that signature . 
$ openssl dgst -sha1 -sign priv_key.pem myjar.jar | openssl enc -base64 

#Step 5:
# load your jars into blob store . refer SOLR-6787

#Step 6:
# use the command to add your jar to classpath as follows
{noformat}
{code}
curl http://localhost:8983/solr/collection1/config -H 
'Content-type:application/json'  -d '{
"add-runtimelib" : {"name": "jarname" , "version":2 , 
"sig":"mW1Gwtz2QazjfVdrLFHfbGwcr8xzFYgUOLu68LHqWRDvLG0uLcy1McQ+AzVmeZFBf1yLPDEHBWJb5KXr8bdbHN/PYgUB1nsr9pk4EFyD9KfJ8TqeH/ijQ9waa/vjqyiKEI9U550EtSzruLVZ32wJ7smvV0fj2YYhrUaaPzOn9g0="
 }// output of step 4. concatenate the lines 

}' 
{code}

sig is the extra parameter that is nothing but the base64 encoded value of the 
jar's sha1 signature 

If no keys are present , the jar is loaded without any checking. 

Before loading a jar from blob store , each Solr node would check if there are 
keys present in the keys directory. If yes, each jar's signature will be 
verified with all the available public keys. If atleast one succeeds , the jar 
is loaded into memory. If nothing succeeds , it will be rejected 


> signing a jar and secure dynamic loading
> 
>
> Key: SOLR-7126
> URL: https://issues.apache.org/jira/browse/SOLR-7126
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
>  Labels: security
> Attachments: SOLR-7126.patch
>
>
> We need to ensure that the jars loaded into solr are trusted 
> We shall use simple PKI to protect the jars/config loaded into the system
> The following are the steps involved for doing that.
> {noformat}
> #Step 1:
> # generate a 768-bit

[jira] [Commented] (SOLR-4787) Join Contrib

[Bill Bell (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344601#comment-14344601
 ] 

Bill Bell commented on SOLR-4787:
-

This seems like a no-brainer.  Can we commit this into 5.xxx ?

> Join Contrib
> 
>
> Key: SOLR-4787
> URL: https://issues.apache.org/jira/browse/SOLR-4787
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.2.1
>Reporter: Joel Bernstein
>Priority: Minor
> Fix For: Trunk
>
> Attachments: SOLR-4787-deadlock-fix.patch, 
> SOLR-4787-pjoin-long-keys.patch, SOLR-4787-with-testcase-fix.patch, 
> SOLR-4787.patch, SOLR-4787.patch, SOLR-4787.patch, SOLR-4787.patch, 
> SOLR-4787.patch, SOLR-4787.patch, SOLR-4787.patch, SOLR-4787.patch, 
> SOLR-4787.patch, SOLR-4787.patch, SOLR-4787.patch, SOLR-4787.patch, 
> SOLR-4787.patch, SOLR-4787.patch, 
> SOLR-4797-hjoin-multivaluekeys-nestedJoins.patch, 
> SOLR-4797-hjoin-multivaluekeys-trunk.patch
>
>
> This contrib provides a place where different join implementations can be 
> contributed to Solr. This contrib currently includes 3 join implementations. 
> The initial patch was generated from the Solr 4.3 tag. Because of changes in 
> the FieldCache API this patch will only build with Solr 4.2 or above.
> *HashSetJoinQParserPlugin aka hjoin*
> The hjoin provides a join implementation that filters results in one core 
> based on the results of a search in another core. This is similar in 
> functionality to the JoinQParserPlugin but the implementation differs in a 
> couple of important ways.
> The first way is that the hjoin is designed to work with int and long join 
> keys only. So, in order to use hjoin, int or long join keys must be included 
> in both the to and from core.
> The second difference is that the hjoin builds memory structures that are 
> used to quickly connect the join keys. So, the hjoin will need more memory 
> then the JoinQParserPlugin to perform the join.
> The main advantage of the hjoin is that it can scale to join millions of keys 
> between cores and provide sub-second response time. The hjoin should work 
> well with up to two million results from the fromIndex and tens of millions 
> of results from the main query.
> The hjoin supports the following features:
> 1) Both lucene query and PostFilter implementations. A *"cost"* > 99 will 
> turn on the PostFilter. The PostFilter will typically outperform the Lucene 
> query when the main query results have been narrowed down.
> 2) With the lucene query implementation there is an option to build the 
> filter with threads. This can greatly improve the performance of the query if 
> the main query index is very large. The "threads" parameter turns on 
> threading. For example *threads=6* will use 6 threads to build the filter. 
> This will setup a fixed threadpool with six threads to handle all hjoin 
> requests. Once the threadpool is created the hjoin will always use it to 
> build the filter. Threading does not come into play with the PostFilter.
> 3) The *size* local parameter can be used to set the initial size of the 
> hashset used to perform the join. If this is set above the number of results 
> from the fromIndex then the you can avoid hashset resizing which improves 
> performance.
> 4) Nested filter queries. The local parameter "fq" can be used to nest a 
> filter query within the join. The nested fq will filter the results of the 
> join query. This can point to another join to support nested joins.
> 5) Full caching support for the lucene query implementation. The filterCache 
> and queryResultCache should work properly even with deep nesting of joins. 
> Only the queryResultCache comes into play with the PostFilter implementation 
> because PostFilters are not cacheable in the filterCache.
> The syntax of the hjoin is similar to the JoinQParserPlugin except that the 
> plugin is referenced by the string "hjoin" rather then "join".
> fq=\{!hjoin fromIndex=collection2 from=id_i to=id_i threads=6 
> fq=$qq\}user:customer1&qq=group:5
> The example filter query above will search the fromIndex (collection2) for 
> "user:customer1" applying the local fq parameter to filter the results. The 
> lucene filter query will be built using 6 threads. This query will generate a 
> list of values from the "from" field that will be used to filter the main 
> query. Only records from the main query, where the "to" field is present in 
> the "from" list will be included in the results.
> The solrconfig.xml in the main query core must contain the reference to the 
> hjoin.
>  class="org.apache.solr.joins.HashSetJoinQParserPlugin"/>
> And the join contrib lib jars must be registed in the solrconfig.xml.
>  
> After issuing the "ant dist" command from inside the solr directory the joins 
> contrib jar will appear in th

[JENKINS] Lucene-Solr-5.x-Windows (64bit/jdk1.8.0_31) - Build # 4519 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Windows/4411/
Java: 64bit/jdk1.8.0_31 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ReplicationFactorTest.test

Error Message:
org.apache.http.NoHttpResponseException: The target server failed to respond

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.http.NoHttpResponseException: The target server failed to respond
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:899)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
at 
org.apache.solr.cloud.ReplicationFactorTest.testRf3(ReplicationFactorTest.java:284)
at 
org.apache.solr.cloud.ReplicationFactorTest.test(ReplicationFactorTest.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:950)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:925)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.Sta

Re: How come we reference luceneMatchVersion 6.0 in 5x?

[Erick Erickson]

Makes sense. I just committed the fix.


On Mon, Mar 2, 2015 at 9:11 PM, Shalin Shekhar Mangar
 wrote:
> Probably due to a bad back-port from trunk to 5x branch?
>
> On Tue, Mar 3, 2015 at 10:34 AM, Erick Erickson 
> wrote:
>>
>> I happened across these:
>>
>>
>> ./solr/solrj/src/test-files/solrj/solr/multicore/core0/conf/solrconfig.xml:
>>  6.0.0
>>
>>
>> ./solr/solrj/src/test-files/solrj/solr/multicore/core1/conf/solrconfig.xml:
>>  6.0.0
>>
>> Any clue or is this a mistake?
>>
>> Erick
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7121) Solr nodes should go down based on configurable thresholds and not rely on resource exhaustion

[Otis Gospodnetic (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344566#comment-14344566
 ] 

Otis Gospodnetic commented on SOLR-7121:


bq. I will surely add these metrics to JMX but can we handle that in a 
follow-up ticket to this one?

Sure, if that works for you.

> Solr nodes should go down based on configurable thresholds and not rely on 
> resource exhaustion
> --
>
> Key: SOLR-7121
> URL: https://issues.apache.org/jira/browse/SOLR-7121
> Project: Solr
>  Issue Type: New Feature
>Reporter: Sachin Goyal
> Attachments: SOLR-7121.patch, SOLR-7121.patch, SOLR-7121.patch, 
> SOLR-7121.patch, SOLR-7121.patch
>
>
> Currently, there is no way to control when a Solr node goes down.
> If the server is having high GC pauses or too many threads or is just getting 
> too many queries due to some bad load-balancer, the cores in the machine keep 
> on serving unless they exhaust the machine's resources and everything comes 
> to a stall.
> Such a slow-dying core can affect other cores as well by taking huge time to 
> serve their distributed queries.
> There should be a way to specify some threshold values beyond which the 
> targeted core can its ill-health and proactively go down to recover.
> When the load improves, the core should come up automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2706 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2706/

5 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: http://127.0.0.1:16390/jci/cm/c8n_1x2_shard1_replica2

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: http://127.0.0.1:16390/jci/cm/c8n_1x2_shard1_replica2
at 
__randomizedtesting.SeedInfo.seed([7233D99518A6DAC7:FA67E64FB65AB73F]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:597)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:918)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:809)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
at 
org.apache.solr.cloud.HttpPartitionTest.doSendDoc(HttpPartitionTest.java:484)
at 
org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:501)
at 
org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:193)
at 
org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:950)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:925)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.Sta

Re: How come we reference luceneMatchVersion 6.0 in 5x?

[Shalin Shekhar Mangar]

Probably due to a bad back-port from trunk to 5x branch?

On Tue, Mar 3, 2015 at 10:34 AM, Erick Erickson 
wrote:

> I happened across these:
>
> ./solr/solrj/src/test-files/solrj/solr/multicore/core0/conf/solrconfig.xml:
>  6.0.0
>
> ./solr/solrj/src/test-files/solrj/solr/multicore/core1/conf/solrconfig.xml:
>  6.0.0
>
> Any clue or is this a mistake?
>
> Erick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
Regards,
Shalin Shekhar Mangar.

How come we reference luceneMatchVersion 6.0 in 5x?

[Erick Erickson]

I happened across these:

./solr/solrj/src/test-files/solrj/solr/multicore/core0/conf/solrconfig.xml:
 6.0.0

./solr/solrj/src/test-files/solrj/solr/multicore/core1/conf/solrconfig.xml:
 6.0.0

Any clue or is this a mistake?

Erick

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: DocValues instead of stored values

[Shalin Shekhar Mangar]

Sorry Hoss, yes the individual items are documented but if you are ever
searching for the answer to the question that Toke was asking, you'd spend
a very long time before arriving at the answer. Also, the docs for "field"
function query say that it is for numerics but it works great with strings
too. It probably won't work with multi-valued (I haven't tried them).
Ideally, we should have this tip in the DocValues page of the reference
guide.

On Tue, Mar 3, 2015 at 1:56 AM, Chris Hostetter 
wrote:

>
> : Hmm, this is indeed not documented. I'll fix.
>
> It's definitely documented...
>
>
> https://cwiki.apache.org/confluence/display/solr/Transforming+Result+Documents
>
> https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-UsingFunctionQuery
>
> https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-AvailableFunctions
>
> ...the problem is it's specifically tied to the use of function queries,
> and won't work very nicely for non-numerics of multivalued fields.
>
> What would be nice is to have an explicit [field] transformer like what i
> proposed here, that included a localparam option to use DocValues instead
> of Stored Fields...
>
>
> https://issues.apache.org/jira/browse/SOLR-7070?focusedCommentId=14305965&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14305965
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
Regards,
Shalin Shekhar Mangar.

[jira] [Commented] (SOLR-6657) DocumentDictionaryFactory requires weightField to be mandatory, but it shouldn't

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344525#comment-14344525
 ] 

ASF subversion and git services commented on SOLR-6657:
---

Commit 1663527 from [~erickoerickson] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1663527 ]

SOLR-6657: DocumentDictionaryFactory requires weightField to be mandatory, but 
it shouldn't

>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> shouldn't
> -
>
> Key: SOLR-6657
> URL: https://issues.apache.org/jira/browse/SOLR-6657
> Project: Solr
>  Issue Type: Sub-task
>Affects Versions: 4.8, 4.8.1, 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Varun Thacker
>Assignee: Erick Erickson
>  Labels: suggester
> Fix For: Trunk, 5.1
>
>
>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> doesn't need to as DocumentDictionary allows it to be null
> So one has to define the weight field in the solrconfig.xml even if their 
> data doesn't contain any weights. We shouldn't make the weightField mandatory 
> in DocumentDictionaryFactory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-6657) DocumentDictionaryFactory requires weightField to be mandatory, but it shouldn't

[Erick Erickson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson closed SOLR-6657.

Resolution: Fixed

this would be super-easy to backport, but unless we fix the other problems with 
suggester there's no point.

>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> shouldn't
> -
>
> Key: SOLR-6657
> URL: https://issues.apache.org/jira/browse/SOLR-6657
> Project: Solr
>  Issue Type: Sub-task
>Affects Versions: 4.8, 4.8.1, 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Varun Thacker
>Assignee: Erick Erickson
>  Labels: suggester
> Fix For: Trunk, 5.1
>
>
>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> doesn't need to as DocumentDictionary allows it to be null
> So one has to define the weight field in the solrconfig.xml even if their 
> data doesn't contain any weights. We shouldn't make the weightField mandatory 
> in DocumentDictionaryFactory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6657) DocumentDictionaryFactory requires weightField to be mandatory, but it shouldn't

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344513#comment-14344513
 ] 

ASF subversion and git services commented on SOLR-6657:
---

Commit 1663525 from [~erickoerickson] in branch 'dev/trunk'
[ https://svn.apache.org/r1663525 ]

SOLR-6657: DocumentDictionaryFactory requires weightField to be mandatory, but 
it shouldn't

>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> shouldn't
> -
>
> Key: SOLR-6657
> URL: https://issues.apache.org/jira/browse/SOLR-6657
> Project: Solr
>  Issue Type: Sub-task
>Affects Versions: 4.8, 4.8.1, 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Varun Thacker
>Assignee: Erick Erickson
>  Labels: suggester
> Fix For: Trunk, 5.1
>
>
>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> doesn't need to as DocumentDictionary allows it to be null
> So one has to define the weight field in the solrconfig.xml even if their 
> data doesn't contain any weights. We shouldn't make the weightField mandatory 
> in DocumentDictionaryFactory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6657) DocumentDictionaryFactory requires weightField to be mandatory, but it shouldn't

[Erick Erickson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-6657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson updated SOLR-6657:
-
Fix Version/s: (was: 5.0)
   5.1

>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> shouldn't
> -
>
> Key: SOLR-6657
> URL: https://issues.apache.org/jira/browse/SOLR-6657
> Project: Solr
>  Issue Type: Sub-task
>Affects Versions: 4.8, 4.8.1, 4.9, 4.9.1, 4.10, 4.10.1
>Reporter: Varun Thacker
>Assignee: Erick Erickson
>  Labels: suggester
> Fix For: Trunk, 5.1
>
>
>  DocumentDictionaryFactory requires weightField to be mandatory, but it 
> doesn't need to as DocumentDictionary allows it to be null
> So one has to define the weight field in the solrconfig.xml even if their 
> data doesn't contain any weights. We shouldn't make the weightField mandatory 
> in DocumentDictionaryFactory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-4.10-Linux (32bit/jdk1.8.0_31) - Build # 11747 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.10-Linux/50/
Java: 32bit/jdk1.8.0_31 -client -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([E3E372A485BB9D2C]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([E3E372A485BB9D2C]:0)




Build Log:
[...truncated 12571 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: 
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-E3E372A485BB9D2C-001/init-core-data-001
   [junit4]   2> 1634839 T3373 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl 
(true) and clientAuth (true)
   [junit4]   2> 1634839 T3373 
oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system 
property: /e/h
   [junit4]   2> 1634845 T3373 oas.SolrTestCaseJ4.setUp ###Starting 
testDistribSearch
   [junit4]   2> 1634846 T3373 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1634847 T3374 oasc.ZkTestServer$ZKServerMain.runFromConfig 
Starting server
   [junit4]   2> 1634946 T3373 oasc.ZkTestServer.run start zk server on 
port:51954
   [junit4]   2> 1634947 T3373 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1634950 T3380 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@13f54cd name:ZooKeeperConnection 
Watcher:127.0.0.1:51954 got event WatchedEvent state:SyncConnected type:None 
path:null path:null type:None
   [junit4]   2> 1634951 T3373 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1634951 T3373 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1634955 T3373 oascc.ConnectionManager.waitForConnected Waiting 
for client to connect to ZooKeeper
   [junit4]   2> 1634962 T3382 oascc.ConnectionManager.process Watcher 
org.apache.solr.common.cloud.ConnectionManager@788094 name:ZooKeeperConnection 
Watcher:127.0.0.1:51954/solr got event WatchedEvent state:SyncConnected 
type:None path:null path:null type:None
   [junit4]   2> 1634963 T3373 oascc.ConnectionManager.waitForConnected Client 
is connected to ZooKeeper
   [junit4]   2> 1634963 T3373 oascc.SolrZkClient.makePath makePath: 
/collections/collection1
   [junit4]   2> 1634967 T3373 oascc.SolrZkClient.makePath makePath: 
/collections/collection1/shards
   [junit4]   2> 1634970 T3373 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection
   [junit4]   2> 1634972 T3373 oascc.SolrZkClient.makePath makePath: 
/collections/control_collection/shards
   [junit4]   2> 1634976 T3373 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
 to /configs/conf1/solrconfig.xml
   [junit4]   2> 1634977 T3373 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.xml
   [junit4]   2> 1634981 T3373 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
 to /configs/conf1/schema.xml
   [junit4]   2> 1634982 T3373 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/schema.xml
   [junit4]   2> 1634985 T3373 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
 to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1634986 T3373 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1634988 T3373 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
 to /configs/conf1/stopwords.txt
   [junit4]   2> 1634989 T3373 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/stopwords.txt
   [junit4]   2> 1634992 T3373 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
 to /configs/conf1/protwords.txt
   [junit4]   2> 1634992 T3373 oascc.SolrZkClient.makePath makePath: 
/configs/conf1/protwords.txt
   [junit4]   2> 1634995 T3373 oasc.AbstractZkTestCase.putConfig put 
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
 to /configs/conf1/currency.xml
   [junit4]   2> 1634996 T3373 oascc.So

[jira] [Commented] (LUCENE-5168) ByteSliceReader assert trips with 32-bit oracle 1.7.0_25 + G1GC

[Petro Semeniuk (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344469#comment-14344469
 ] 

Petro Semeniuk commented on LUCENE-5168:


[~mikemccand] hey! Could you please advise whenever you are running on 32bit or 
64bit?

We're on 1.8,0_20 64 bit and not sure whenever it's worth to use G1.

> ByteSliceReader assert trips with 32-bit oracle 1.7.0_25 + G1GC
> ---
>
> Key: LUCENE-5168
> URL: https://issues.apache.org/jira/browse/LUCENE-5168
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Attachments: java8-windows-4x-3075-console.txt, log.0025, log.0042, 
> log.0078, log.0086, log.0100
>
>
> This assertion trips (sometimes from different tests), if you run the 
> highlighting tests on branch_4x with r1512807.
> It reproduces about half the time, always only with 32bit + G1GC (other 
> combinations do not seem to trip it, i didnt try looping or anything really 
> though).
> {noformat}
> rmuir@beast:~/workspace/branch_4x$ svn up -r 1512807
> rmuir@beast:~/workspace/branch_4x$ ant clean
> rmuir@beast:~/workspace/branch_4x$ rm -rf .caches #this is important,
> otherwise master seed does not work!
> rmuir@beast:~/workspace/branch_4x/lucene/highlighter$ ant test
> -Dtests.jvms=2 -Dtests.seed=EBBFA6F4E80A7365 -Dargs="-server
> -XX:+UseG1GC"
> {noformat}
> Originally showed up like this:
> {noformat}
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6874/
> Java: 32bit/jdk1.7.0_25 -server -XX:+UseG1GC
> 1 tests failed.
> REGRESSION:  
> org.apache.lucene.search.postingshighlight.TestPostingsHighlighter.testUserFailedToIndexOffsets
> Error Message:
> Stack Trace:
> java.lang.AssertionError
> at 
> __randomizedtesting.SeedInfo.seed([EBBFA6F4E80A7365:1FBF811885F2D611]:0)
> at 
> org.apache.lucene.index.ByteSliceReader.readByte(ByteSliceReader.java:73)
> at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108)
> at 
> org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:453)
> at 
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
> at org.apache.lucene.index.TermsHash.flush(TermsHash.java:116)
> at org.apache.lucene.index.DocInverter.flush(DocInverter.java:53)
> at 
> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
> at 
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:501)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_31) - Build # 4519 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4519/
Java: 32bit/jdk1.8.0_31 -client -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestSolrConfigHandler

Error Message:
Could not remove the following files (in the order of attempts):
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf\params.json: 
java.nio.file.FileSystemException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf\params.json: The process 
cannot access the file because it is being used by another process. 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf: 
java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1: 
java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010: java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010: java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001: java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of 
attempts):
   
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf\params.json: 
java.nio.file.FileSystemException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf\params.json: The process 
cannot access the file because it is being used by another process.

   
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf: 
java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1\conf
   
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1: 
java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010\collection1
   
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010: java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010
   
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010: java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001\tempDir-010
   
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\build\solr-core\test\J0\temp\solr.core.TestSolrConfigHandler
 31AE8487A2FB247-001: java.nio.file.DirectoryNotEmptyException: 
C:\Users\JenkinsSlave\workspace\Lucene-Solr-trunk-Windows\solr\bu

[jira] [Commented] (SOLR-7180) MiniSolrCloudCluster should startup and shutdown its jetties in parallel

[Vamsee Yarlagadda (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344457#comment-14344457
 ] 

Vamsee Yarlagadda commented on SOLR-7180:
-

LGTM. One small concern.
{quote}
executor.invokeAll(startups);
executor.invokeAll(shutdowns);
{quote}
How do we know whether all the jetty runners came up/shutdown properly? 
invokeAll will return once the Future.isDone() on every task right. But it 
doesn't guarantee that there are no exceptions in the process.


> MiniSolrCloudCluster should startup and shutdown its jetties in parallel
> 
>
> Key: SOLR-7180
> URL: https://issues.apache.org/jira/browse/SOLR-7180
> Project: Solr
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Attachments: SOLR-7180.patch
>
>
> Followup to SOLR-7179.  Now JettySolrRunner doesn't use sysprops to pass 
> configuration, we can start up multiple runners in parallel.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-6327) ArrayIndexOutOfBoundsException in reading a lucene block

[Vamsee Yarlagadda (JIRA)]

Vamsee Yarlagadda created LUCENE-6327:
-

 Summary: ArrayIndexOutOfBoundsException in reading a lucene block
 Key: LUCENE-6327
 URL: https://issues.apache.org/jira/browse/LUCENE-6327
 Project: Lucene - Core
  Issue Type: Bug
Reporter: Vamsee Yarlagadda
Priority: Minor


I notice this error while trying to do heavy indexing in Solr. This error was 
seen in testing Cloudera Search (Solr 4.4 with lots of SolrCloud, other 
critical bug fixes)

{code}
2015-02-27 04:21:46,644 INFO org.apache.solr.core.SolrCore.Request: 
[crunch_sequence_collection_shard2_replica5] webapp=/solr path=/update 
params={distrib.from=http://search-15.vpc.cloudera.com:8983/solr/crunch_sequence_collection_shard2_replica2/&update.distrib=FROMLEADER&wt=javabin&version=2}
 status=0 QTime=246 
2015-02-27 04:21:46,773 ERROR org.apache.solr.core.SolrCore: 
java.lang.ArrayIndexOutOfBoundsException
at 
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.fillTerm(BlockTreeTermsReader.java:2934)
at 
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.scanToTermLeaf(BlockTreeTermsReader.java:2743)
at 
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.scanToTerm(BlockTreeTermsReader.java:2662)
at 
org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekExact(BlockTreeTermsReader.java:1695)
at 
org.apache.solr.search.SolrIndexSearcher.lookupId(SolrIndexSearcher.java:746)
at 
org.apache.solr.update.VersionInfo.getVersionFromIndex(VersionInfo.java:193)
at org.apache.solr.update.UpdateLog.lookupVersion(UpdateLog.java:739)
at 
org.apache.solr.update.VersionInfo.lookupVersion(VersionInfo.java:183)
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:716)
at 
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:459)
at 
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:247)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1953)
at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:766)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:397)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:186)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:272)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:277)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555)
at 
org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:277)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at 
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at 
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHan

[jira] [Updated] (SOLR-5332) Add "preserve original" setting to the EdgeNGramFilterFactory

[Alexander S. (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander S. updated SOLR-5332:
---
Fix Version/s: 5.1

> Add "preserve original" setting to the EdgeNGramFilterFactory
> -
>
> Key: SOLR-5332
> URL: https://issues.apache.org/jira/browse/SOLR-5332
> Project: Solr
>  Issue Type: Wish
>Affects Versions: 4.4, 4.5, 4.5.1, 4.6
>Reporter: Alexander S.
> Fix For: 5.1
>
>
> Hi, as described here: 
> http://lucene.472066.n3.nabble.com/Help-to-figure-out-why-query-does-not-match-td4086967.html
>  the problem is in that if you have these 2 strings to index:
> 1. facebook.com/someuser.1
> 2. facebook.com/someveryandverylongusername
> and the edge ngram filter factory with min and max gram size settings 2 and 
> 25, search requests for these urls will fail.
> But search requests for:
> 1. facebook.com/someuser
> 2. facebook.com/someveryandverylonguserna
> will work properly.
> It's because first url has "1" at the end, which is lover than the allowed 
> min gram size. In the second url the user name is longer than the max gram 
> size (27 characters).
> Would be good to have a "preserve original" option, that will add the 
> original string to the index if it does not fit the allowed gram size, so 
> that "1" and "someveryandverylongusername" tokens will also be added to the 
> index.
> Best,
> Alex



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5332) Add "preserve original" setting to the EdgeNGramFilterFactory

[Alexander S. (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander S. updated SOLR-5332:
---
Affects Version/s: (was: 5.1)

> Add "preserve original" setting to the EdgeNGramFilterFactory
> -
>
> Key: SOLR-5332
> URL: https://issues.apache.org/jira/browse/SOLR-5332
> Project: Solr
>  Issue Type: Wish
>Affects Versions: 4.4, 4.5, 4.5.1, 4.6
>Reporter: Alexander S.
> Fix For: 5.1
>
>
> Hi, as described here: 
> http://lucene.472066.n3.nabble.com/Help-to-figure-out-why-query-does-not-match-td4086967.html
>  the problem is in that if you have these 2 strings to index:
> 1. facebook.com/someuser.1
> 2. facebook.com/someveryandverylongusername
> and the edge ngram filter factory with min and max gram size settings 2 and 
> 25, search requests for these urls will fail.
> But search requests for:
> 1. facebook.com/someuser
> 2. facebook.com/someveryandverylonguserna
> will work properly.
> It's because first url has "1" at the end, which is lover than the allowed 
> min gram size. In the second url the user name is longer than the max gram 
> size (27 characters).
> Would be good to have a "preserve original" option, that will add the 
> original string to the index if it does not fit the allowed gram size, so 
> that "1" and "someveryandverylongusername" tokens will also be added to the 
> index.
> Best,
> Alex



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5332) Add "preserve original" setting to the EdgeNGramFilterFactory

[Alexander S. (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander S. updated SOLR-5332:
---
Affects Version/s: 5.1

> Add "preserve original" setting to the EdgeNGramFilterFactory
> -
>
> Key: SOLR-5332
> URL: https://issues.apache.org/jira/browse/SOLR-5332
> Project: Solr
>  Issue Type: Wish
>Affects Versions: 4.4, 4.5, 4.5.1, 4.6, 5.1
>Reporter: Alexander S.
>
> Hi, as described here: 
> http://lucene.472066.n3.nabble.com/Help-to-figure-out-why-query-does-not-match-td4086967.html
>  the problem is in that if you have these 2 strings to index:
> 1. facebook.com/someuser.1
> 2. facebook.com/someveryandverylongusername
> and the edge ngram filter factory with min and max gram size settings 2 and 
> 25, search requests for these urls will fail.
> But search requests for:
> 1. facebook.com/someuser
> 2. facebook.com/someveryandverylonguserna
> will work properly.
> It's because first url has "1" at the end, which is lover than the allowed 
> min gram size. In the second url the user name is longer than the max gram 
> size (27 characters).
> Would be good to have a "preserve original" option, that will add the 
> original string to the index if it does not fit the allowed gram size, so 
> that "1" and "someveryandverylongusername" tokens will also be added to the 
> index.
> Best,
> Alex



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7082) Streaming Aggregation for SolrCloud

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-7082:
-
Attachment: SOLR-7082.patch

Patch with all test passing.

> Streaming Aggregation for SolrCloud
> ---
>
> Key: SOLR-7082
> URL: https://issues.apache.org/jira/browse/SOLR-7082
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Joel Bernstein
> Fix For: Trunk, 5.1
>
> Attachments: SOLR-7082.patch, SOLR-7082.patch, SOLR-7082.patch
>
>
> This issue provides a general purpose streaming aggregation framework for 
> SolrCloud. An overview of how it works can be found at this link:
> http://heliosearch.org/streaming-aggregation-for-solrcloud/
> This functionality allows SolrCloud users to perform operations that we're 
> typically done using map/reduce or a parallel computing platform.
> Here is a brief explanation of how the framework works:
> There is a new Solrj *io* package found in: *org.apache.solr.client.solrj.io*
> Key classes:
> *Tuple*: Abstracts a document in a search result as a Map of key/value pairs.
> *TupleStream*: is the base class for all of the streams. Abstracts search 
> results as a stream of Tuples.
> *SolrStream*: connects to a single Solr instance. You call the read() method 
> to iterate over the Tuples.
> *CloudSolrStream*: connects to a SolrCloud collection and merges the results 
> based on the sort param. The merge takes place in CloudSolrStream itself.
> *Decorator Streams*: wrap other streams to gather *Metrics* on streams and 
> *transform* the streams. Some examples are the MetricStream, RollupStream, 
> GroupByStream, UniqueStream, MergeJoinStream, HashJoinStream, MergeStream, 
> FilterStream.
> *Going parallel with the ParallelStream and  "Worker Collections"*
> The io package also contains the *ParallelStream*, which wraps a TupleStream 
> and sends it to N worker nodes. The workers are chosen from a SolrCloud 
> collection. These "Worker Collections" don't have to hold any data, they can 
> just be used to execute TupleStreams.
> *The StreamHandler*
> The Worker nodes have a new RequestHandler called the *StreamHandler*. The 
> ParallelStream serializes a TupleStream, before it is opened, and sends it to 
> the StreamHandler on the Worker Nodes.
> The StreamHandler on each Worker node deserializes the TupleStream, opens the 
> stream, iterates the tuples and streams them back to the ParallelStream. The 
> ParallelStream performs the final merge of Metrics and can be wrapped by 
> other Streams to handled the final merged TupleStream.
> *Sorting and Partitioning search results (Shuffling)*
> Each Worker node is shuffled 1/N of the document results. There is a 
> "partitionKeys" parameter that can be included with each TupleStream to 
> ensure that Tuples with the same partitionKeys are shuffled to the same 
> Worker. The actual partitioning is done with a filter query using the 
> HashQParserPlugin. The DocSets from the HashQParserPlugin can be cached in 
> the filter cache which provides extremely high performance hash partitioning. 
> Many of the stream transformations rely on the sort order of the TupleStreams 
> (GroupByStream, MergeJoinStream, UniqueStream, FilterStream etc..). To 
> accommodate this the search results can be sorted by specific keys. The 
> "/export" handler can be used to sort entire result sets efficiently.
> By specifying the sort order of the results and the partition keys, documents 
> will be sorted and partitioned inside of the search engine. So when the 
> tuples hit the network they are already sorted, partitioned and headed 
> directly to correct worker node.
> *Extending The Framework*
> To extend the framework you create new TupleStream Decorators, that gather 
> custom metrics or perform custom stream transformations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-2562) Make Luke a Lucene/Solr Module

[Tomoko Uchida (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-2562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344292#comment-14344292
 ] 

Tomoko Uchida commented on LUCENE-2562:
---

Progress and status:

Honestly there are no progress in development, I'd like to suggest about merge 
of this Jira issue and the Luke project now on Github.
The (one of) Luke project's goal, "To port the thinlet UI to an ASL compliant 
license framework so that it can be contributed back to Apache Lucene.", is 
perfectly match the goal of this issue, I hope it would be desirable way.
https://github.com/DmitryKey/luke/tree/master

After a short discussion at solr mailing list, Dmitry Kan, the maintainer of 
Luke, has created "pivot-luke" branch in luke repository. 
And I have merged all the latest pivot's version  codes plugged in SVN to the 
branch.

The discussion is here: 
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201502.mbox/%3CCAHpHuj%3DZArPSbSLp2_d1X7OD_2V0NeBsDSRVizgTDTnBiAZPMQ%40mail.gmail.com%3E
The branch for pivot's version is here: 
https://github.com/DmitryKey/luke/tree/pivot-luke

It now can be built by maven, you can check out and test it.
We'll contribute back to Lucene it, I hope this is in not so distant future. Of 
course, developers are welcome. 



> I know it's homage to the original Luke default color, but we should really 
> change the default background color

I promise that, gray theme is also my favorite one! :)

I'll keep watching this ticket, any comments and suggestions are welcome.
Thanks.

> Make Luke a Lucene/Solr Module
> --
>
> Key: LUCENE-2562
> URL: https://issues.apache.org/jira/browse/LUCENE-2562
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Mark Miller
>  Labels: gsoc2014
> Attachments: LUCENE-2562-Ivy.patch, LUCENE-2562-Ivy.patch, 
> LUCENE-2562-Ivy.patch, LUCENE-2562-ivy.patch, LUCENE-2562.patch, 
> LUCENE-2562.patch, Luke-ALE-1.png, Luke-ALE-2.png, Luke-ALE-3.png, 
> Luke-ALE-4.png, Luke-ALE-5.png, luke1.jpg, luke2.jpg, luke3.jpg, 
> lukeALE-documents.png
>
>
> see
> "RE: Luke - in need of maintainer": 
> http://markmail.org/message/m4gsto7giltvrpuf
> "Web-based Luke": http://markmail.org/message/4xwps7p7ifltme5q
> I think it would be great if there was a version of Luke that always worked 
> with trunk - and it would also be great if it was easier to match Luke jars 
> with Lucene versions.
> While I'd like to get GWT Luke into the mix as well, I think the easiest 
> starting point is to straight port Luke to another UI toolkit before 
> abstracting out DTO objects that both GWT Luke and Pivot Luke could share.
> I've started slowly converting Luke's use of thinlet to Apache Pivot. I 
> haven't/don't have a lot of time for this at the moment, but I've plugged 
> away here and there over the past work or two. There is still a *lot* to do.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6320) speed up checkindex

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344275#comment-14344275
 ] 

ASF subversion and git services commented on LUCENE-6320:
-

Commit 1663510 from [~rcmuir] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1663510 ]

LUCENE-6320: speed up checkindex

> speed up checkindex
> ---
>
> Key: LUCENE-6320
> URL: https://issues.apache.org/jira/browse/LUCENE-6320
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Fix For: Trunk, 5.1
>
> Attachments: LUCENE-6320.patch, LUCENE-6320.patch
>
>
> This is fairly slow today, very ram intensive, and has some buggy stuff (e.g. 
> postingsenum reuse bugs). We can do better...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-6320) speed up checkindex

[Robert Muir (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir resolved LUCENE-6320.
-
   Resolution: Fixed
Fix Version/s: 5.1
   Trunk

> speed up checkindex
> ---
>
> Key: LUCENE-6320
> URL: https://issues.apache.org/jira/browse/LUCENE-6320
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Fix For: Trunk, 5.1
>
> Attachments: LUCENE-6320.patch, LUCENE-6320.patch
>
>
> This is fairly slow today, very ram intensive, and has some buggy stuff (e.g. 
> postingsenum reuse bugs). We can do better...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Anshum Gupta to the PMC

[Joel Bernstein]

Congratulations Anshum!

Joel Bernstein
Search Engineer at Heliosearch

On Mon, Mar 2, 2015 at 2:10 PM, Sanne Grinovero 
wrote:

> Congratulations Anshum!
>
> Regards,
> Sanne
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[JENKINS] Lucene-Solr-5.x-MacOSX (64bit/jdk1.7.0) - Build # 2025 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-MacOSX/1982/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  
org.apache.solr.search.function.TestOrdValues.testReverseOrdFieldExactScore

Error Message:
PermGen space

Stack Trace:
java.lang.OutOfMemoryError: PermGen space
at 
__randomizedtesting.SeedInfo.seed([C2476CBDD95741E2:A4C71EFD015F8871]:0)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at 
org.apache.lucene.util.LuceneTestCase.newSearcher(LuceneTestCase.java:1706)
at 
org.apache.lucene.util.LuceneTestCase.newSearcher(LuceneTestCase.java:1673)
at 
org.apache.lucene.util.LuceneTestCase.newSearcher(LuceneTestCase.java:1665)
at 
org.apache.solr.search.function.TestOrdValues.doTestExactScore(TestOrdValues.java:140)
at 
org.apache.solr.search.function.TestOrdValues.testReverseOrdFieldExactScore(TestOrdValues.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.function.TestOrdValues

Error Message:
MockDirectoryWrapper: cannot close: there are still open files: 
{_1_Lucene50_0.dvd=1, _1_Direct_0.dvdd=1, _1.fdt=1, 
_1_TestBloomFilteredLucenePostings_0.doc=1, _1.tvd=1, 
_1_TestBloomFilteredLucenePostings_0.pos=1, _1_Lucene50_1.dvd=1, 
_1_LuceneVarGapFixedInterval_0.pos=1, 
_1_TestBloomFilteredLucenePostings_0.tim=1, 
_1_LuceneVarGapFixedInterval_0.tib=1, _1_LuceneVarGapFixedInterval_0.doc=1}

Stack Trace:
java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 
open files: {_1_Lucene50_0.dvd=1, _1_Direct_0.dvdd=1, _1.fdt=1, 
_1_TestBloomFilteredLucenePostings_0.doc=1, _1.tvd=1, 
_1_TestBloomFilteredLucenePostings_0.pos=1, _1_Lucene50_1.dvd=1, 
_1_LuceneVarGapFixedInterval_0.pos=1, 
_1_TestBloomFilteredLucenePostings_0.tim=1, 
_1_LuceneVarGapFixedInterval_0.tib=1, _1_LuceneVarGapFixedInterval_0.doc=1}
at __randomizedtesting.SeedInfo.seed([C2476CBDD95741E2]:0)
at 
org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:747)
at 
org.apache.solr.search.function.TestOrdValues.afterClassFunctionTestSetup(TestOrdValues.java:213)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsea

[jira] [Commented] (LUCENE-6320) speed up checkindex

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344236#comment-14344236
 ] 

ASF subversion and git services commented on LUCENE-6320:
-

Commit 1663505 from [~rcmuir] in branch 'dev/trunk'
[ https://svn.apache.org/r1663505 ]

LUCENE-6320: speed up checkindex

> speed up checkindex
> ---
>
> Key: LUCENE-6320
> URL: https://issues.apache.org/jira/browse/LUCENE-6320
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Attachments: LUCENE-6320.patch, LUCENE-6320.patch
>
>
> This is fairly slow today, very ram intensive, and has some buggy stuff (e.g. 
> postingsenum reuse bugs). We can do better...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2705 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2705/

5 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: http://127.0.0.1:10511/l_out/c8n_1x2_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: http://127.0.0.1:10511/l_out/c8n_1x2_shard1_replica1
at 
__randomizedtesting.SeedInfo.seed([6DEB3E57C8B66EF1:E5BF018D664A0309]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:597)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:918)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:809)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
at 
org.apache.solr.cloud.HttpPartitionTest.doSendDoc(HttpPartitionTest.java:484)
at 
org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:501)
at 
org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:193)
at 
org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:950)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:925)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.State

[jira] [Commented] (SOLR-7115) UpdateLog can miss closing transaction log objects.

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344187#comment-14344187
 ] 

Yonik Seeley commented on SOLR-7115:


I've also run the test from SOLR-7113 a number of times after commenting out 
the suppression, and I can't get it to fail.

> UpdateLog can miss closing transaction log objects.
> ---
>
> Key: SOLR-7115
> URL: https://issues.apache.org/jira/browse/SOLR-7115
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>
> I've seen this happen on YourKit and in various tests - especially since 
> adding resource release tracking to the log objects. Now I've got a test that 
> catches it in SOLR-7113.
> It seems that in precommit, if prevTlog is not null, we need to close it 
> because we are going to overwrite prevTlog with a new log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: reuseAddress default in Solr jetty.xml

[Reitzel, Charles]

My bad.  Too long away from sockets since cleaning up those shutdown handlers.  
Your point is well taken, on the server side the risks of consuming a stray 
echo packet are fairly low (but non-zero, if you’ve ever spent any quality time 
with tcpdump/wireshark).

Still, in a production setting, SIGKILL (aka “kill -9”) should be a last resort 
after more reasonable methods (e.g. SIGINT, SIGTERM, SIGSTOP) have failed.

From: Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com]
Sent: Monday, March 02, 2015 7:00 PM
To: dev@lucene.apache.org
Subject: RE: reuseAddress default in Solr jetty.xml


No, reuseAddress doesn't allow you to have two processes, old and new, listen 
to the same port. There's no option which allows you to do that.

Tl;DR This can happen when you have a connection to a server which gets killed 
hard and comes back up immediately

So here's what happens.

When a server normally shuts down, it triggers an active close on all open TCP 
connections it has. That sends a three way msg exchange with the remote 
recipient (FIN, FIN+ACK, ACK) at the end of which the socket is closed and the 
kernel puts it in a TIME_WAIT state for a few minutes in the background 
(depends on the OS, maximum tends to be 4 mins). This is needed to allow for 
reordered older packets to reach the machine just in case. Now typically if the 
server restarts within that period and tries to bind again to the same port, 
the kernel is smart enough to not complain that there is an existing socket in 
TIME_WAIT, because it knows the last sequence number it used for the final 
message in the previous process, and since sequence numbers are always 
increasing, it can reject any messages before that sequence number as a new 
process has now taken the port.

Trouble is with abnormal shutdown. There's no time for a proper goodbye, so the 
kernel marks the socket to respond to remote packets with a rude RST (reset). 
Since there has been no goodbye with the remote end, it also doesn't know the 
last sequence number to delineate if a new process binds to the same port. 
Hence by default it denies binding to the new port for the TIME_WAIT period to 
avoid the off chance a stray packet gets picked up by the new process and 
utterly confuses it. By setting reuseAddress, you are essentially waiving off 
this protection. Note that this possibility of confusion is unbelievably 
miniscule in the first place (both the source and destination host:port should 
be the same and the client port is generally randomly allocated). If the port 
we are talking of is a local port, it's almost impossible -- you have bigger 
problems if a TCP packet is lost or delayed within the same machine!

As to Shawn's point, for Solr's stop port, you essentially need to be trying to 
actively shutdown the server using the stop port, or be within a few minutes of 
such an attempt while the server is killed. Just the server being killed 
without any active connection to it is not going to cause this issue.
Hi Ram,

It appears the problem is that the old solr/jetty process is actually still 
running when the new solr/jetty process is started.   That’s the problem that 
needs fixing.

This is not a rare problem in systems with worker threads dedicated to 
different tasks.   These threads need to wake up in response to the shutdown 
signal/command, as well the normal inputs.

It’s a bug I’ve created and fixed a couple times over the years … :-)I 
wouldn’t know where to start with Solr.  But, as I say, re-using the port is a 
band-aid.  I’ve yet to see a case where it is the best solution.

best,
Charlie

From: Ramkumar R. Aiyengar 
[mailto:andyetitmo...@gmail.com]
Sent: Saturday, February 28, 2015 8:15 PM
To: dev@lucene.apache.org
Subject: Re: reuseAddress default in Solr jetty.xml

Hey Charles, see my explanation above on why this is needed. If Solr has to be 
killed, it would generally be immediately restarted. This would normally not 
the case, except when things are potentially misconfigured or if there is a 
bug, but not doing so makes the impact worse..
In any case, turns out really that reuseAddress is true by default for the 
connectors we use, so that really isn't the issue. The issue more specifically 
is that the stop port doesn't do it, so the actual port by itself starts just 
fine on a restart, but the stop port fails to bind -- and there's no way 
currently in Jetty to configure that.
Based on my question in the jetty mailing list, I have now created an issue for 
them..

https://bugs.eclipse.org/bugs/show_bug.cgi?id=461133

On Fri, Feb 27, 2015 at 3:03 PM, Reitzel, Charles 
mailto:charles.reit...@tiaa-cref.org>> wrote:
Disclaimer: I’m not a Solr committer.  But, as a developer, I’ve never seen a 
good case for reusing the listening port.   Better to find and fix the root 
cause on the zombie state (or just slow shutdown, sometimes) and release the 
port.

From: Mark Miller [mailto:markrm

[jira] [Updated] (SOLR-7177) ConcurrentUpdateSolrClient should log connection information on http failures

[Vamsee Yarlagadda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vamsee Yarlagadda updated SOLR-7177:

Attachment: SOLR-7177v2.patch

Here is another revision of the patch.
We are now wrapping up the original exception in SolrServerException with 
additional information.

> ConcurrentUpdateSolrClient should log connection information on http failures 
> --
>
> Key: SOLR-7177
> URL: https://issues.apache.org/jira/browse/SOLR-7177
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 4.10.3, 5.0
>Reporter: Vamsee Yarlagadda
>Priority: Minor
> Attachments: SOLR-7177.patch, SOLR-7177v2.patch
>
>
> I notice when there is an http connection failure, we simply log the error 
> but not the connection information. It would be good to log this info to make 
> debugging easier.
> e.g:
> 1.
> {code}
> 2015-02-27 08:56:51,503 ERROR org.apache.solr.update.StreamingSolrServers: 
> error
> java.net.SocketException: Connection reset
>   at java.net.SocketInputStream.read(SocketInputStream.java:196)
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>   at 
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
>   at 
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
>   at 
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
>   at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
>   at 
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
>   at 
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
>   at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
>   at 
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
>   at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>   at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:235)
> {code}
>  
> 2.
> {code}
> 2015-02-27 10:26:12,363 ERROR org.apache.solr.update.StreamingSolrServers: 
> error
> org.apache.http.NoHttpResponseException: The target server failed to respond
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
>   at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
>   at 
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
>   at 
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
>   at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
>   at 
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
>   at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>   at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:235)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja

[jira] [Commented] (SOLR-7177) ConcurrentUpdateSolrClient should log connection information on http failures

[Vamsee Yarlagadda (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344152#comment-14344152
 ] 

Vamsee Yarlagadda commented on SOLR-7177:
-

bq.  I think it makes more sense to wrap the whole Exception in another 
IOException with a message including the URL?
+1. This looks much cleaner. I am planning to use SolrServerException (as this 
captures more of communication related errors).

bq. Why not just add this info where the exception is first thrown in solr cmd 
distributor? 
1. ConcurrentUpdateSolrClient is part of SolrJ right. Adding this info at this 
class will also benefit all the end users who are directly using solrj api to 
talk with Solr service. Adding this at SolrCmdDistributor will only expose this 
to internal solr server log. 
2. Moreover, ConcurrentUpdateSolrClient already does something similar to the 
above proposed idea. We are just trying to extend the same idea.
https://github.com/apache/lucene-solr/blob/trunk/solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.java#L242

> ConcurrentUpdateSolrClient should log connection information on http failures 
> --
>
> Key: SOLR-7177
> URL: https://issues.apache.org/jira/browse/SOLR-7177
> Project: Solr
>  Issue Type: Improvement
>Affects Versions: 4.10.3, 5.0
>Reporter: Vamsee Yarlagadda
>Priority: Minor
> Attachments: SOLR-7177.patch
>
>
> I notice when there is an http connection failure, we simply log the error 
> but not the connection information. It would be good to log this info to make 
> debugging easier.
> e.g:
> 1.
> {code}
> 2015-02-27 08:56:51,503 ERROR org.apache.solr.update.StreamingSolrServers: 
> error
> java.net.SocketException: Connection reset
>   at java.net.SocketInputStream.read(SocketInputStream.java:196)
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>   at 
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
>   at 
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
>   at 
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
>   at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
>   at 
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
>   at 
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
>   at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
>   at 
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
>   at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>   at 
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>   at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:235)
> {code}
>  
> 2.
> {code}
> 2015-02-27 10:26:12,363 ERROR org.apache.solr.update.StreamingSolrServers: 
> error
> org.apache.http.NoHttpResponseException: The target server failed to respond
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
>   at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
>   at 
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
>   at 
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
>   at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
>   at 
> org.apache.http.protocol.HttpRequestEx

[jira] [Updated] (SOLR-7121) Solr nodes should go down based on configurable thresholds and not rely on resource exhaustion

[Sachin Goyal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sachin Goyal updated SOLR-7121:
---
Attachment: SOLR-7121.patch

Added tests for long-running queries and 95th/5MinRateRequest statistics as 
well.
GC-time test will need to run for quite sometime before it can detect the same, 
hence not adding test for that.
But the remaining tests should provide a good testing of the patch.

> Solr nodes should go down based on configurable thresholds and not rely on 
> resource exhaustion
> --
>
> Key: SOLR-7121
> URL: https://issues.apache.org/jira/browse/SOLR-7121
> Project: Solr
>  Issue Type: New Feature
>Reporter: Sachin Goyal
> Attachments: SOLR-7121.patch, SOLR-7121.patch, SOLR-7121.patch, 
> SOLR-7121.patch, SOLR-7121.patch
>
>
> Currently, there is no way to control when a Solr node goes down.
> If the server is having high GC pauses or too many threads or is just getting 
> too many queries due to some bad load-balancer, the cores in the machine keep 
> on serving unless they exhaust the machine's resources and everything comes 
> to a stall.
> Such a slow-dying core can affect other cores as well by taking huge time to 
> serve their distributed queries.
> There should be a way to specify some threshold values beyond which the 
> targeted core can its ill-health and proactively go down to recover.
> When the load improves, the core should come up automatically.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release 4.10.4 RC0

[Michael McCandless]

Vote passes, I'll push the bits out!  Thanks for testing.

Mike McCandless

http://blog.mikemccandless.com


On Fri, Feb 27, 2015 at 6:27 PM, Michael McCandless
 wrote:
> Artifacts: 
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
>
> Smoke tester: python3 -u dev-tools/scripts/smokeTestRelease.py
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
> 1662817 4.10.4 /tmp/smoke4104 True
>
> SUCCESS! [0:39:34.527017]
>
> I also confirmed Elasticsearch 1.x tests pass after upgrading to this.
>
> Here's my +1
>
> Mike McCandless
>
> http://blog.mikemccandless.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: reuseAddress default in Solr jetty.xml

[Ramkumar R. Aiyengar]

No, reuseAddress doesn't allow you to have two processes, old and new,
listen to the same port. There's no option which allows you to do that.

Tl;DR This can happen when you have a connection to a server which gets
killed hard and comes back up immediately

So here's what happens.

When a server normally shuts down, it triggers an active close on all open
TCP connections it has. That sends a three way msg exchange with the remote
recipient (FIN, FIN+ACK, ACK) at the end of which the socket is closed and
the kernel puts it in a TIME_WAIT state for a few minutes in the background
(depends on the OS, maximum tends to be 4 mins). This is needed to allow
for reordered older packets to reach the machine just in case. Now
typically if the server restarts within that period and tries to bind again
to the same port, the kernel is smart enough to not complain that there is
an existing socket in TIME_WAIT, because it knows the last sequence number
it used for the final message in the previous process, and since sequence
numbers are always increasing, it can reject any messages before that
sequence number as a new process has now taken the port.

Trouble is with abnormal shutdown. There's no time for a proper goodbye, so
the kernel marks the socket to respond to remote packets with a rude RST
(reset). Since there has been no goodbye with the remote end, it also
doesn't know the last sequence number to delineate if a new process binds
to the same port. Hence by default it denies binding to the new port for
the TIME_WAIT period to avoid the off chance a stray packet gets picked up
by the new process and utterly confuses it. By setting reuseAddress, you
are essentially waiving off this protection. Note that this possibility of
confusion is unbelievably miniscule in the first place (both the source and
destination host:port should be the same and the client port is generally
randomly allocated). If the port we are talking of is a local port, it's
almost impossible -- you have bigger problems if a TCP packet is lost or
delayed within the same machine!

As to Shawn's point, for Solr's stop port, you essentially need to be
trying to actively shutdown the server using the stop port, or be within a
few minutes of such an attempt while the server is killed. Just the server
being killed without any active connection to it is not going to cause this
issue.

Hi Ram,



It appears the problem is that the old solr/jetty process is actually still
running when the new solr/jetty process is started.   That’s the problem
that needs fixing.



This is not a rare problem in systems with worker threads dedicated to
different tasks.   These threads need to wake up in response to the
shutdown signal/command, as well the normal inputs.



It’s a bug I’ve created and fixed a couple times over the years … :-)I
wouldn’t know where to start with Solr.  But, as I say, re-using the port
is a band-aid.  I’ve yet to see a case where it is the best solution.



best,

Charlie



*From:* Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com]
*Sent:* Saturday, February 28, 2015 8:15 PM
*To:* dev@lucene.apache.org
*Subject:* Re: reuseAddress default in Solr jetty.xml



Hey Charles, see my explanation above on why this is needed. If Solr has to
be killed, it would generally be immediately restarted. This would normally
not the case, except when things are potentially misconfigured or if there
is a bug, but not doing so makes the impact worse..

In any case, turns out really that reuseAddress is true by default for the
connectors we use, so that really isn't the issue. The issue more
specifically is that the stop port doesn't do it, so the actual port by
itself starts just fine on a restart, but the stop port fails to bind --
and there's no way currently in Jetty to configure that.

Based on my question in the jetty mailing list, I have now created an issue
for them..

https://bugs.eclipse.org/bugs/show_bug.cgi?id=461133



On Fri, Feb 27, 2015 at 3:03 PM, Reitzel, Charles <
charles.reit...@tiaa-cref.org> wrote:

Disclaimer: I’m not a Solr committer.  But, as a developer, I’ve never seen
a good case for reusing the listening port.   Better to find and fix the
root cause on the zombie state (or just slow shutdown, sometimes) and
release the port.



*From:* Mark Miller [mailto:markrmil...@gmail.com]
*Sent:* Thursday, February 26, 2015 5:28 PM
*To:* dev@lucene.apache.org
*Subject:* Re: reuseAddress default in Solr jetty.xml



+1

- Mark



On Thu, Feb 26, 2015 at 1:54 PM Ramkumar R. Aiyengar <
andyetitmo...@gmail.com> wrote:

The jetty.xml we currently ship by default doesn't set reuseAddress=true.
If you are having a bad GC day with things going OOM and resulting in Solr
not even being able to shutdown cleanly (or the oom_solr.sh script killing
it), whatever external service management mechanism you have is probably
going to try respawn it and fail with the default config because the ports
will be in TIME_WAIT. I guess there's the usual disclaimer wit

Re: DocValues instead of stored values

[Joel Bernstein]

The /export handler uses DocValues to output the field list. It's a pretty
big improvement in performance over stored fields. It supports single and
multivalue numerics and strings so you can test out the various performance
characteristics. Also as David mentioned DocValues has some nice options
for trading off memory for performance.

Joel Bernstein
Search Engineer at Heliosearch

On Mon, Mar 2, 2015 at 3:26 PM, Chris Hostetter 
wrote:

>
> : Hmm, this is indeed not documented. I'll fix.
>
> It's definitely documented...
>
>
> https://cwiki.apache.org/confluence/display/solr/Transforming+Result+Documents
>
> https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-UsingFunctionQuery
>
> https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-AvailableFunctions
>
> ...the problem is it's specifically tied to the use of function queries,
> and won't work very nicely for non-numerics of multivalued fields.
>
> What would be nice is to have an explicit [field] transformer like what i
> proposed here, that included a localparam option to use DocValues instead
> of Stored Fields...
>
>
> https://issues.apache.org/jira/browse/SOLR-7070?focusedCommentId=14305965&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14305965
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[jira] [Commented] (SOLR-7182) Make the Schema-API a first class citizen of SolrJ

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343841#comment-14343841
 ] 

Erick Erickson commented on SOLR-7182:
--

Sven:

There's really never a firm date in mind, it's more like whenever enough has 
accumulated that someone makes it a priority. It might be in the next several 
weeks but nothing's been planned yet.

Usually someone announces on dev list that they're thinking about cutting a new 
release giving people a few day's warning, no such announcement has been made.

> Make the Schema-API a first class citizen of SolrJ
> --
>
> Key: SOLR-7182
> URL: https://issues.apache.org/jira/browse/SOLR-7182
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Affects Versions: 5.0
> Environment: any
>Reporter: Sven Windisch
>Priority: Minor
>  Labels: api, schema, solrj
> Fix For: 5.1
>
>
> There are several Solr APIs that are handled as first class citizens in 
> SolrJ, esp. the Node API and the Collections API, i.e. they have their own 
> xxxAdminRequest Object extended from the SolrRequest Class. As someone who 
> programmatically changes Schemas a lot, I had hoped to see the Schema API 
> handled first class in release 5.0, too. As far as I dug into the code and 
> docs of SolrJ 5.0, that did not happen. If there is a reasonable point why 
> this won't happen at all, I would really like to hear it. If the only reason 
> is, that nobody had time for this, I would happily help out here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Backporting suggester JIRAS to 4.10.5

[Erick Erickson]

Jan:

The two JIRAs were SOLR-6845 SOLR-6864

I back-ported them to 4.10, but then infix-based suggesters were
rebuilding on startup afterwards even though they don't on 5.x. I
didn't track down whether there were other JIRAs that I needed to
back-port or whether I'd just messed up the back-porting of these two.
Either way it was getting to be more work than I'd bargained for,
other priorities and all that.

Then SOLR-6657 seemed like a good idea too, and then there's
LUCENE-5833 which seemed like another good idea. So it started to look
like I was going down a feature creep path that was more than a 4.10.5
called for so I stopped.

Erick

On Mon, Mar 2, 2015 at 2:14 AM, Jan Høydahl  wrote:
> Erick,
>
> LUCENE-5833 is already part of 5.0 and trunk, just repoened for 4.x. I have 
> applied and tested the patch on 4.10 branch locally.
>
> Which were the other suggester issues you failed to merge to 4.10.x?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 2. mar. 2015 kl. 07.49 skrev Erick Erickson :
>>
>> I decided against doing this for the time being unless someone really
>> clamors for it. It's looking more complicated than I thought (or I
>> didn't merge the JIRAs into 4.10 correctly or...).
>>
>> I'll look at  LUCENE-5833 and SOLR-6657 for 5.1 instead.
>>
>> Sorry for the noise.
>>
>> Erick
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7115) UpdateLog can miss closing transaction log objects.

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343836#comment-14343836
 ] 

Yonik Seeley commented on SOLR-7115:


OK... this is really strange.

UpdateLog.preCommit() / postCommit() pair are protected by the commit lock.
prevTlog is set in preCommit and cleared in postCommit.
Hmmm, is this happening in the presence of other errors (something that would 
prevent postCommit from being called?)
That would seem to be the only way to get into preCommit and have prevTlog 
still pointing at something.
I just verified by throwing an exception inside the "if (prevTlog != null) {" 
block in preCommit() and the entire test suite passes.

> UpdateLog can miss closing transaction log objects.
> ---
>
> Key: SOLR-7115
> URL: https://issues.apache.org/jira/browse/SOLR-7115
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>
> I've seen this happen on YourKit and in various tests - especially since 
> adding resource release tracking to the log objects. Now I've got a test that 
> catches it in SOLR-7113.
> It seems that in precommit, if prevTlog is not null, we need to close it 
> because we are going to overwrite prevTlog with a new log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2704 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2704/

5 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.test

Error Message:
org.apache.http.NoHttpResponseException: The target server failed to respond

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.http.NoHttpResponseException: The target server failed to respond
at 
__randomizedtesting.SeedInfo.seed([3279408195A8D70B:BA2D7F5B3B54BAF3]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:899)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
at 
org.apache.solr.cloud.HttpPartitionTest.doSendDoc(HttpPartitionTest.java:484)
at 
org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:501)
at 
org.apache.solr.cloud.HttpPartitionTest.testRf2(HttpPartitionTest.java:193)
at 
org.apache.solr.cloud.HttpPartitionTest.test(HttpPartitionTest.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:945)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:920)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.ut

[jira] [Commented] (LUCENE-6238) minimize tests.policy

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343763#comment-14343763
 ] 

ASF subversion and git services commented on LUCENE-6238:
-

Commit 1663432 from [~dawidweiss] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1663432 ]

LUCENE-6238: minimize tests.policy (upgrade of randomizedtesting, system 
properties are
read-only).

> minimize tests.policy
> -
>
> Key: LUCENE-6238
> URL: https://issues.apache.org/jira/browse/LUCENE-6238
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Fix For: Trunk, 5.1
>
> Attachments: LUCENE-6238-mmap.patch, LUCENE-6238.patch, 
> LUCENE-6238.patch, LUCENE-6238.patch, LUCENE-6238.patch, LUCENE-6238.patch
>
>
> This is overly permissive:
> {noformat}
>   // Basic permissions needed for Lucene to work:
>   permission java.util.PropertyPermission "*", "read,write";
>   permission java.lang.reflect.ReflectPermission "*";
>   permission java.lang.RuntimePermission "*";
> {noformat}
> Because of various BS like unsafe-hacks (only mmap seems to do it properly), 
> this means effectively you cannot use lucene with SM today, without allowing 
> SM itself to just be disabled with reflection. 
> This is easy to fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6238) minimize tests.policy

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343748#comment-14343748
 ] 

ASF subversion and git services commented on LUCENE-6238:
-

Commit 1663431 from [~dawidweiss] in branch 'dev/trunk'
[ https://svn.apache.org/r1663431 ]

LUCENE-6238: minimize tests.policy (upgrade of randomizedtesting, system 
properties are read-only).

> minimize tests.policy
> -
>
> Key: LUCENE-6238
> URL: https://issues.apache.org/jira/browse/LUCENE-6238
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Fix For: Trunk, 5.1
>
> Attachments: LUCENE-6238-mmap.patch, LUCENE-6238.patch, 
> LUCENE-6238.patch, LUCENE-6238.patch, LUCENE-6238.patch, LUCENE-6238.patch
>
>
> This is overly permissive:
> {noformat}
>   // Basic permissions needed for Lucene to work:
>   permission java.util.PropertyPermission "*", "read,write";
>   permission java.lang.reflect.ReflectPermission "*";
>   permission java.lang.RuntimePermission "*";
> {noformat}
> Because of various BS like unsafe-hacks (only mmap seems to do it properly), 
> this means effectively you cannot use lucene with SM today, without allowing 
> SM itself to just be disabled with reflection. 
> This is easy to fix.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release 4.10.4 RC0

[Tomás Fernández Löbbe]

+1
SUCCESS! [1:32:40.270547]

On Mon, Mar 2, 2015 at 11:56 AM, Yonik Seeley  wrote:

> +1
>
> -Yonik
>
>
> On Fri, Feb 27, 2015 at 3:27 PM, Michael McCandless
>  wrote:
> > Artifacts:
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
> >
> > Smoke tester: python3 -u dev-tools/scripts/smokeTestRelease.py
> >
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
> > 1662817 4.10.4 /tmp/smoke4104 True
> >
> > SUCCESS! [0:39:34.527017]
> >
> > I also confirmed Elasticsearch 1.x tests pass after upgrading to this.
> >
> > Here's my +1
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[jira] [Resolved] (SOLR-7171) BaseDistributedSearchTestCase.getSolrXml() not used consistently - multiple divergent ways a JettySolrRunner may be inited

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-7171.

   Resolution: Fixed
Fix Version/s: 5.1
   Trunk
 Assignee: Hoss Man

> BaseDistributedSearchTestCase.getSolrXml() not used consistently - multiple 
> divergent ways a JettySolrRunner may be inited
> --
>
> Key: SOLR-7171
> URL: https://issues.apache.org/jira/browse/SOLR-7171
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
>Assignee: Hoss Man
> Fix For: Trunk, 5.1
>
> Attachments: SOLR-7171.patch, SOLR-7171.patch
>
>
> As part of SOLR-6349, i wanted to take advantage of the new features in 
> SOLR-7147 to inspect shard requests in TestDistributedSearch, but even though 
> i added a proper override of getSolrXml...
> {code}
>   @Override
>   protected String getSolrXml() {
> return "solr-trackingshardhandler.xml"; 
>   }
> {code}
> ...that value was being ignored, and i was never getting an instance of 
> TrackingShardHandlerFactory.
> I poked around a bit and confirmed that getSolrXml() is used by 
> "setupJettySolrHome(File)" but that method is never called by 
> BaseDistributedSearchTestCase - it's only called in framework subclasses like 
> AbstractDistribZkTestBase and AbstractFullDistribZkTestBase.  Instead, for 
> simple subclasses of BaseDistributedSearchTestCase the jetty instances seem 
> to be coming from createServers(int)
> 
> I don't really understand why there are so many diff ways for a shard 
> instance to be inited, and presumably that should be refactored? .. but a 
> more immediate concern is that subclasses of BaseDistributedSearchTestCase 
> which attempt to override the solr.xml file used can't (unless they are 
> actually a subclass of AbstractDistribZkTestBase of 
> AbstractFullDistribZkTestBase)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7171) BaseDistributedSearchTestCase.getSolrXml() not used consistently - multiple divergent ways a JettySolrRunner may be inited

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343711#comment-14343711
 ] 

ASF subversion and git services commented on SOLR-7171:
---

Commit 1663421 from hoss...@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1663421 ]

SOLR-7171: BaseDistributedSearchTestCase now clones getSolrHome() for each 
subclass, and consistently uses getSolrXml() (merge r1663381)

> BaseDistributedSearchTestCase.getSolrXml() not used consistently - multiple 
> divergent ways a JettySolrRunner may be inited
> --
>
> Key: SOLR-7171
> URL: https://issues.apache.org/jira/browse/SOLR-7171
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
> Attachments: SOLR-7171.patch, SOLR-7171.patch
>
>
> As part of SOLR-6349, i wanted to take advantage of the new features in 
> SOLR-7147 to inspect shard requests in TestDistributedSearch, but even though 
> i added a proper override of getSolrXml...
> {code}
>   @Override
>   protected String getSolrXml() {
> return "solr-trackingshardhandler.xml"; 
>   }
> {code}
> ...that value was being ignored, and i was never getting an instance of 
> TrackingShardHandlerFactory.
> I poked around a bit and confirmed that getSolrXml() is used by 
> "setupJettySolrHome(File)" but that method is never called by 
> BaseDistributedSearchTestCase - it's only called in framework subclasses like 
> AbstractDistribZkTestBase and AbstractFullDistribZkTestBase.  Instead, for 
> simple subclasses of BaseDistributedSearchTestCase the jetty instances seem 
> to be coming from createServers(int)
> 
> I don't really understand why there are so many diff ways for a shard 
> instance to be inited, and presumably that should be refactored? .. but a 
> more immediate concern is that subclasses of BaseDistributedSearchTestCase 
> which attempt to override the solr.xml file used can't (unless they are 
> actually a subclass of AbstractDistribZkTestBase of 
> AbstractFullDistribZkTestBase)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: DocValues instead of stored values

[Chris Hostetter]

: Hmm, this is indeed not documented. I'll fix.

It's definitely documented...

https://cwiki.apache.org/confluence/display/solr/Transforming+Result+Documents
https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-UsingFunctionQuery
https://cwiki.apache.org/confluence/display/solr/Function+Queries#FunctionQueries-AvailableFunctions

...the problem is it's specifically tied to the use of function queries, 
and won't work very nicely for non-numerics of multivalued fields.

What would be nice is to have an explicit [field] transformer like what i 
proposed here, that included a localparam option to use DocValues instead 
of Stored Fields...

https://issues.apache.org/jira/browse/SOLR-7070?focusedCommentId=14305965&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14305965


-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.9.0-ea-b47) - Build # 11909 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11746/
Java: 32bit/jdk1.9.0-ea-b47 -client -XX:+UseG1GC

8 tests failed.
FAILED:  org.apache.solr.cloud.SaslZkACLProviderTest.testSaslZkACLProvider

Error Message:
Could not get the port for ZooKeeper server

Stack Trace:
java.lang.RuntimeException: Could not get the port for ZooKeeper server
at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:506)
at 
org.apache.solr.cloud.SaslZkACLProviderTest$SaslZkTestServer.run(SaslZkACLProviderTest.java:225)
at 
org.apache.solr.cloud.SaslZkACLProviderTest.setUp(SaslZkACLProviderTest.java:90)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:861)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestCollectionAPI.test

Error Message:
Could not get the port for ZooKeeper server

Stack Trace:
java.lang.RuntimeException: Could not get the port for ZooKeeper server
at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:506)
at 
org.apache.solr.cloud.AbstractDistribZkT

Re: [VOTE] Release 4.10.4 RC0

[Yonik Seeley]

+1

-Yonik


On Fri, Feb 27, 2015 at 3:27 PM, Michael McCandless
 wrote:
> Artifacts: 
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
>
> Smoke tester: python3 -u dev-tools/scripts/smokeTestRelease.py
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
> 1662817 4.10.4 /tmp/smoke4104 True
>
> SUCCESS! [0:39:34.527017]
>
> I also confirmed Elasticsearch 1.x tests pass after upgrading to this.
>
> Here's my +1
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Windows (32bit/jdk1.8.0_31) - Build # 4518 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Windows/4410/
Java: 32bit/jdk1.8.0_31 -client -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.TestHighlightDedupGrouping.test

Error Message:
Timeout occured while waiting response from server at: 
https://127.0.0.1:60588/wpp/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: https://127.0.0.1:60588/wpp/collection1
at 
__randomizedtesting.SeedInfo.seed([618D59BB719903CD:E9D96661DF656E35]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:570)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:131)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:124)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:110)
at 
org.apache.solr.TestHighlightDedupGrouping.addDoc(TestHighlightDedupGrouping.java:122)
at 
org.apache.solr.TestHighlightDedupGrouping.randomizedTest(TestHighlightDedupGrouping.java:96)
at 
org.apache.solr.TestHighlightDedupGrouping.test(TestHighlightDedupGrouping.java:42)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:945)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:920)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
a

[jira] [Created] (SOLR-7183) SaslZkACLProviderTest reproducible failures due to poor locale blacklisting

[Hoss Man (JIRA)]

Hoss Man created SOLR-7183:
--

 Summary: SaslZkACLProviderTest reproducible failures due to poor 
locale blacklisting
 Key: SOLR-7183
 URL: https://issues.apache.org/jira/browse/SOLR-7183
 Project: Solr
  Issue Type: Bug
Reporter: Hoss Man
Assignee: Gregory Chanan


SaslZkACLProviderTest has this blacklist of locales...
{code}
  // These Locales don't generate dates that are compatibile with Hadoop 
MiniKdc.
  protected final static List brokenLocales =
Arrays.asList(
  "th_TH_TH_#u-nu-thai",
  "ja_JP_JP_#u-ca-japanese",
  "hi_IN");
{code}

..but this list is incomplete -- notably because it only focuses on one 
specific Thai variant, and then does a string Locale.toString() comparison.  so 
at a minimum {{-Dtests.locale=th_TH}} also fails - i suspect there are other 
variants that will fail as well

* if there is a bug in "Hadoop MiniKdc" then that bug should be filed in jira, 
and there should be Solr jira that refers to it -- the Solr jira URL needs to 
be included her in the test case so developers in the future can understand the 
context and have some idea of if/when the third-party lib bug is fixed
* if we need to work around some Locales because of this bug, then Locale 
comparisons need be based on whatever aspects of the Locale are actually 
problematic

see for example SOLR-6387 & this commit: 
https://svn.apache.org/viewvc/lucene/dev/branches/branch_4x/solr/contrib/morphlines-core/src/test/org/apache/solr/morphlines/solr/AbstractSolrMorphlineZkTestBase.java?r1=1618676&r2=1618675&pathrev=1618676

Or SOLR-6991 + TIKA-1526 & this commit: 
https://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_5_0/solr/contrib/extraction/src/test/org/apache/solr/handler/extraction/ExtractingRequestHandlerTest.java?r1=1653708&r2=1653707&pathrev=1653708




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7073) Add an API to add a jar to a collection's classpath

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul updated SOLR-7073:
-
Attachment: SOLR-7073.patch

with basic tests added and some bugs fixed

> Add an API to add a jar to a collection's classpath
> ---
>
> Key: SOLR-7073
> URL: https://issues.apache.org/jira/browse/SOLR-7073
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
> Attachments: SOLR-7073.patch, SOLR-7073.patch
>
>
> The idea of having separate classloaders for each component may be counter 
> productive.  This aims to add a jar dependency to the collection itself and 
> each core belonging to that collection will have the jar in the classpath
> As we load everything from the .system collection , we cannot make the core 
> loading delayed till .system is fully loaded and is available . 
> There is a new  set of  config commands to manage the dependencies on a 
> collection level. It is possible to have more than one jar as a dependency. 
> The "lib" attribute is same as the blob name that we use in the blob store API
> example:
> {code}
> curl http://localhost:8983/solr/collection1/config -H 
> 'Content-type:application/json'  -d '{
> "add-runtimelib" : {"name": "jarname" , "version":2 },
> "update-runtimelib" :{"name": "jarname" ,"version":3},
> "delete-runtimelib" :"jarname" 
> }' 
> {code}
> The same is added to the overlay.json .
> The default SolrResourceLoader does not have visibility to  these jars . 
> There is a classloader that can access these jars which is made available 
> only to those components which are specially annotated
> Every pluggable component can have an optional extra attribute called 
> {{runtimeLib=true}}, which means, these components are not be loaded at core 
> load time. They are tried to be loaded on demand and if all the dependency 
> jars are not available at the component load time an error is thrown . 
> example of registering a valueSourceParser which depends on the runtime 
> classloader
> {code}
> curl http://localhost:8983/solr/collection1/config -H 
> 'Content-type:application/json'  -d '{
> "create-valuesourceparser" : {"name": "nvl" ,
> "runtimeLib": true, 
> "class":"solr.org.apache.solr.search.function.NvlValueSourceParser , 
> "nvlFloatValue":0.0 }  
> }'
> {code} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Anshum Gupta to the PMC

[Sanne Grinovero]

Congratulations Anshum!

Regards,
Sanne

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Tomás Fernández Löbbe]

Welcome Ramkumar!

On Mon, Mar 2, 2015 at 8:01 AM, Ryan Ernst  wrote:

> Welcome!
> On Mar 1, 2015 8:40 PM, "Shalin Shekhar Mangar" 
> wrote:
>
>> I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's
>> invitation to become a committer.
>>
>> Ramkumar, it's tradition that you introduce yourself with a brief bio.
>>
>> Your handle "andyetitmoves" has already added to the “lucene" LDAP group,
>> so you now have commit privileges. Please test this by adding yourself to
>> the committers section of the Who We Are page on the website: <
>> http://lucene.apache.org/whoweare.html> (use the ASF CMS bookmarklet at
>> the bottom of the page here:  - more
>> info here ).
>>
>> The ASF dev page also has lots of useful links: <
>> http://www.apache.org/dev/>.
>>
>> Congratulations and welcome!
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>

[jira] [Commented] (SOLR-6143) Bad facet counts from CollapsingQParserPlugin

[Rebecca Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343576#comment-14343576
 ] 

Rebecca Tang commented on SOLR-6143:


Did anyone log an enhancement bug for the CollapsingQParserPlugin to support 
group.facets?  Please let me know.  If not, I would like to log an enhancement 
bug.

> Bad facet counts from CollapsingQParserPlugin 
> --
>
> Key: SOLR-6143
> URL: https://issues.apache.org/jira/browse/SOLR-6143
> Project: Solr
>  Issue Type: Bug
>  Components: query parsers
>Affects Versions: 4.8.1
> Environment: UNIX
> Tomcat 7.0.33
> SOLR 4.8.1
> 16 GB RAM
>Reporter: David Fennessey
>
> I'm noticing a very weird bug using the CollapsingQParserPlugin. We tried to 
> use this plugin when we realized that faceting on the groups would take a 
> ridiculous amount of time. To its credit, it works very quickly, however the 
> facet counts that it gives are incorrect. 
> We have a smallish index of about 200k documents with about with about 50k 
> distinct groups within it. 
> When we use the group implementation 
> (&group=true&group.field=PrSKU&group.facet=true) which I believe this 
> attempts to emulate, the facet counts are totally correct. 
> When we use the field collapsing implementation, it will show an incorrect 
> count for the non-filtered query, but when we go to the filtered query, the 
> facet count corrects itself and matches the document count. 
> Here are some SOLR responses:
> solrslave01:8983/index/select?q=classIDs:12&fl=PrSKU&fq={!collapse%20field=PrSKU}&facet=true&facet.field=at_12_wood_tone
> The facet field will return 
> 867
> 441
> 253
> When I actually apply a filter query like so:
> solrslave01:8983/index/select?q=classIDs:12&fl=PrSKU&fq={!collapse%20field=PrSKU}&facet=true&facet.field=at_12_wood_tone&fq=at_12_wood_tone:%22Light%20Wood%22
> I actually pull back 270 results and the facet updates itself with the 
> correct number at the bottom
> 270
> 68
> 66
> If this were the same number pre and post filter query I would assume that it 
> was simply my data that was bad, however I've pored over this for the better 
> part of a day and I'm pretty sure it's the plugin. For reference, this field 
> that I'm faceting on is a multiValued field, however I have noticed the exact 
> same behavior on non multiValued fields (such as price). 
> I can provide any other details you might need



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release 4.10.4 RC0

[Shalin Shekhar Mangar]

+1

SUCCESS! [0:46:07.507383]

On Sat, Feb 28, 2015 at 4:57 AM, Michael McCandless <
luc...@mikemccandless.com> wrote:

> Artifacts:
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
>
> Smoke tester: python3 -u dev-tools/scripts/smokeTestRelease.py
>
> http://people.apache.org/~mikemccand/staging_area/lucene-solr-4.10.4-RC0-rev1662817
> 1662817 4.10.4 /tmp/smoke4104 True
>
> SUCCESS! [0:39:34.527017]
>
> I also confirmed Elasticsearch 1.x tests pass after upgrading to this.
>
> Here's my +1
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
Regards,
Shalin Shekhar Mangar.

[jira] [Comment Edited] (SOLR-7109) Indexing threads stuck during network partition can put leader into down state

[Shalin Shekhar Mangar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343551#comment-14343551
 ] 

Shalin Shekhar Mangar edited comment on SOLR-7109 at 3/2/15 6:47 PM:
-

Here's a patch which uses ZooKeeper 'multi' transactions to make sure that the 
LIR state can be set only when the requesting leader node is still alive. This 
ensures that regardless of how long the network partition lasts (long GC, 
whatever), the node setting the LIR state must be the leader or else the LIR 
state cannot be set.

Initially I attempted to use the shard leader path as the 'exists' check in the 
'multi' command but this doesn't work because the leader path is always created 
fresh which means that it's version is always 0 and the check always succeeds 
regardless of who the current leader is. This is why we must use the election's 
leader sequence path.

This is just a first cut of this approach. I intend to refactor some of these 
LIR methods -- they have become too big. I will also write a test which 
exercises these new transactional semantics and reproduces the failure.

Edit - I also remove the replicaUrl parameter from 
ZkController.ensureReplicaInLeaderInitiatedRecovery because replicaProps were 
already being passed as a parameter and the replica url can be derived from it.


was (Author: shalinmangar):
Here's a patch which uses ZooKeeper 'multi' transactions to make sure that the 
LIR state can be set only when the requesting leader node is still alive. This 
ensures that regardless of how long the network partition lasts (long GC, 
whatever), the node setting the LIR state must be the leader or else the LIR 
state cannot be set.

Initially I attempted to use the shard leader path as the 'exists' check in the 
'multi' command but this doesn't work because the leader path is always created 
fresh which means that it's version is always 0 and the check always succeeds 
regardless of who the current leader is. This is why we must use the election's 
leader sequence path.

This is just a first cut of this approach. I intend to refactor some of these 
LIR methods -- they have become too big. I will also write a test which 
exercises these new transactional semantics and reproduces the failure.

> Indexing threads stuck during network partition can put leader into down state
> --
>
> Key: SOLR-7109
> URL: https://issues.apache.org/jira/browse/SOLR-7109
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 4.10.3, 5.0
>Reporter: Shalin Shekhar Mangar
> Fix For: Trunk, 5.1
>
> Attachments: SOLR-7109.patch
>
>
> I found this recently while running some Jepsen tests. I found that some 
> threads get stuck on zk operations for a long time in 
> ZkController.updateLeaderInitiatedRecoveryState method and when they wake up 
> they go ahead with setting the LIR state to down. But in the mean time, new 
> leader has been elected and sometimes you'd get into a state where the leader 
> itself is put into recovery causing the shard to reject all writes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7109) Indexing threads stuck during network partition can put leader into down state

[Shalin Shekhar Mangar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shalin Shekhar Mangar updated SOLR-7109:

Attachment: SOLR-7109.patch

Here's a patch which uses ZooKeeper 'multi' transactions to make sure that the 
LIR state can be set only when the requesting leader node is still alive. This 
ensures that regardless of how long the network partition lasts (long GC, 
whatever), the node setting the LIR state must be the leader or else the LIR 
state cannot be set.

Initially I attempted to use the shard leader path as the 'exists' check in the 
'multi' command but this doesn't work because the leader path is always created 
fresh which means that it's version is always 0 and the check always succeeds 
regardless of who the current leader is. This is why we must use the election's 
leader sequence path.

This is just a first cut of this approach. I intend to refactor some of these 
LIR methods -- they have become too big. I will also write a test which 
exercises these new transactional semantics and reproduces the failure.

> Indexing threads stuck during network partition can put leader into down state
> --
>
> Key: SOLR-7109
> URL: https://issues.apache.org/jira/browse/SOLR-7109
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 4.10.3, 5.0
>Reporter: Shalin Shekhar Mangar
> Fix For: Trunk, 5.1
>
> Attachments: SOLR-7109.patch
>
>
> I found this recently while running some Jepsen tests. I found that some 
> threads get stuck on zk operations for a long time in 
> ZkController.updateLeaderInitiatedRecoveryState method and when they wake up 
> they go ahead with setting the LIR state to down. But in the mean time, new 
> leader has been elected and sometimes you'd get into a state where the leader 
> itself is put into recovery causing the shard to reject all writes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: reuseAddress default in Solr jetty.xml

[Shawn Heisey]

On 3/2/2015 10:34 AM, Reitzel, Charles wrote:
>
> Hi Ram,
>
>  
>
> It appears the problem is that the old solr/jetty process is actually
> still running when the new solr/jetty process is started.   That’s the
> problem that needs fixing.
>
>  
>
> This is not a rare problem in systems with worker threads dedicated to
> different tasks.   These threads need to wake up in response to the
> shutdown signal/command, as well the normal inputs.
>
>  
>
> It’s a bug I’ve created and fixed a couple times over the years …
> :-)I wouldn’t know where to start with Solr.  But, as I say,
> re-using the port is a band-aid.  I’ve yet to see a case where it is
> the best solution.
>

I can't say whether the lack of the reuse option on the stop port
binding is a real problem.  I can say that I've never had a problem with
my init script for the 4.x example jetty, which *DOES* use the STOPPORT
and STOPKEY options.  I know that there have been times when Solr has
been completely unresponsive and the init script has been forced to use
the -9 signal.  You can find this init script (in redhat and ubuntu
varieties) here:

http://wiki.apache.org/solr/ShawnHeisey

Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7171) BaseDistributedSearchTestCase.getSolrXml() not used consistently - multiple divergent ways a JettySolrRunner may be inited

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343510#comment-14343510
 ] 

ASF subversion and git services commented on SOLR-7171:
---

Commit 1663381 from hoss...@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1663381 ]

SOLR-7171: BaseDistributedSearchTestCase now clones getSolrHome() for each 
subclass, and consistently uses getSolrXml()

> BaseDistributedSearchTestCase.getSolrXml() not used consistently - multiple 
> divergent ways a JettySolrRunner may be inited
> --
>
> Key: SOLR-7171
> URL: https://issues.apache.org/jira/browse/SOLR-7171
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
> Attachments: SOLR-7171.patch, SOLR-7171.patch
>
>
> As part of SOLR-6349, i wanted to take advantage of the new features in 
> SOLR-7147 to inspect shard requests in TestDistributedSearch, but even though 
> i added a proper override of getSolrXml...
> {code}
>   @Override
>   protected String getSolrXml() {
> return "solr-trackingshardhandler.xml"; 
>   }
> {code}
> ...that value was being ignored, and i was never getting an instance of 
> TrackingShardHandlerFactory.
> I poked around a bit and confirmed that getSolrXml() is used by 
> "setupJettySolrHome(File)" but that method is never called by 
> BaseDistributedSearchTestCase - it's only called in framework subclasses like 
> AbstractDistribZkTestBase and AbstractFullDistribZkTestBase.  Instead, for 
> simple subclasses of BaseDistributedSearchTestCase the jetty instances seem 
> to be coming from createServers(int)
> 
> I don't really understand why there are so many diff ways for a shard 
> instance to be inited, and presumably that should be refactored? .. but a 
> more immediate concern is that subclasses of BaseDistributedSearchTestCase 
> which attempt to override the solr.xml file used can't (unless they are 
> actually a subclass of AbstractDistribZkTestBase of 
> AbstractFullDistribZkTestBase)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-7073) Add an API to add a jar to a collection's classpath

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343469#comment-14343469
 ] 

Noble Paul edited comment on SOLR-7073 at 3/2/15 6:05 PM:
--

This does a complete refactor of Solr's plugin loading because all components 
now need to be lazily loadable based on the extra flag. I would love to have 
extra eyes on this before it is too late. 

There is a new class called {{PluginRegistry}} which is slightly more than our 
{{Map>}} .{{PluginRegistry}} takes care of loading the plugins 
at startup or lazily depending on the various attributes such as 
{{startup="lazy"}} or {{runtimeLib="true"}} . This patch gets rid of the Lazy 
loading wrappers we had for {{SolrRequestHandler}} and {{QueryResponseWriter}}


was (Author: noble.paul):
This does a complete refactor of Solr's plugin loading because all components 
now need to be lazily loadable based on the extra flag. I would love to have 
extra eyes on this before it is too late. 

> Add an API to add a jar to a collection's classpath
> ---
>
> Key: SOLR-7073
> URL: https://issues.apache.org/jira/browse/SOLR-7073
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
> Attachments: SOLR-7073.patch
>
>
> The idea of having separate classloaders for each component may be counter 
> productive.  This aims to add a jar dependency to the collection itself and 
> each core belonging to that collection will have the jar in the classpath
> As we load everything from the .system collection , we cannot make the core 
> loading delayed till .system is fully loaded and is available . 
> There is a new  set of  config commands to manage the dependencies on a 
> collection level. It is possible to have more than one jar as a dependency. 
> The "lib" attribute is same as the blob name that we use in the blob store API
> example:
> {code}
> curl http://localhost:8983/solr/collection1/config -H 
> 'Content-type:application/json'  -d '{
> "add-runtimelib" : {"name": "jarname" , "version":2 },
> "update-runtimelib" :{"name": "jarname" ,"version":3},
> "delete-runtimelib" :"jarname" 
> }' 
> {code}
> The same is added to the overlay.json .
> The default SolrResourceLoader does not have visibility to  these jars . 
> There is a classloader that can access these jars which is made available 
> only to those components which are specially annotated
> Every pluggable component can have an optional extra attribute called 
> {{runtimeLib=true}}, which means, these components are not be loaded at core 
> load time. They are tried to be loaded on demand and if all the dependency 
> jars are not available at the component load time an error is thrown . 
> example of registering a valueSourceParser which depends on the runtime 
> classloader
> {code}
> curl http://localhost:8983/solr/collection1/config -H 
> 'Content-type:application/json'  -d '{
> "create-valuesourceparser" : {"name": "nvl" ,
> "runtimeLib": true, 
> "class":"solr.org.apache.solr.search.function.NvlValueSourceParser , 
> "nvlFloatValue":0.0 }  
> }'
> {code} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6307) Rename SegmentInfo.docCount -> .maxDoc

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343471#comment-14343471
 ] 

ASF subversion and git services commented on LUCENE-6307:
-

Commit 1663376 from [~mikemccand] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1663376 ]

LUCENE-6307: rename confusing docCount -> maxDoc in several places

> Rename SegmentInfo.docCount -> .maxDoc
> --
>
> Key: LUCENE-6307
> URL: https://issues.apache.org/jira/browse/LUCENE-6307
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: Trunk, 5.x
>
> Attachments: LUCENE-6307.patch, LUCENE-6307.patch
>
>
> We already have maxDoc and numDocs, I think it's crazy we have a 3rd one 
> docCount.
> We should just rename to maxDoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7073) Add an API to add a jar to a collection's classpath

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343469#comment-14343469
 ] 

Noble Paul commented on SOLR-7073:
--

This does a complete refactor of Solr's plugin loading because all components 
now need to be lazily loadable based on the extra flag. I would love to have 
extra eyes on this before it is too late. 

> Add an API to add a jar to a collection's classpath
> ---
>
> Key: SOLR-7073
> URL: https://issues.apache.org/jira/browse/SOLR-7073
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Noble Paul
>Assignee: Noble Paul
> Attachments: SOLR-7073.patch
>
>
> The idea of having separate classloaders for each component may be counter 
> productive.  This aims to add a jar dependency to the collection itself and 
> each core belonging to that collection will have the jar in the classpath
> As we load everything from the .system collection , we cannot make the core 
> loading delayed till .system is fully loaded and is available . 
> There is a new  set of  config commands to manage the dependencies on a 
> collection level. It is possible to have more than one jar as a dependency. 
> The "lib" attribute is same as the blob name that we use in the blob store API
> example:
> {code}
> curl http://localhost:8983/solr/collection1/config -H 
> 'Content-type:application/json'  -d '{
> "add-runtimelib" : {"name": "jarname" , "version":2 },
> "update-runtimelib" :{"name": "jarname" ,"version":3},
> "delete-runtimelib" :"jarname" 
> }' 
> {code}
> The same is added to the overlay.json .
> The default SolrResourceLoader does not have visibility to  these jars . 
> There is a classloader that can access these jars which is made available 
> only to those components which are specially annotated
> Every pluggable component can have an optional extra attribute called 
> {{runtimeLib=true}}, which means, these components are not be loaded at core 
> load time. They are tried to be loaded on demand and if all the dependency 
> jars are not available at the component load time an error is thrown . 
> example of registering a valueSourceParser which depends on the runtime 
> classloader
> {code}
> curl http://localhost:8983/solr/collection1/config -H 
> 'Content-type:application/json'  -d '{
> "create-valuesourceparser" : {"name": "nvl" ,
> "runtimeLib": true, 
> "class":"solr.org.apache.solr.search.function.NvlValueSourceParser , 
> "nvlFloatValue":0.0 }  
> }'
> {code} 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-6307) Rename SegmentInfo.docCount -> .maxDoc

[Michael McCandless (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless resolved LUCENE-6307.

Resolution: Fixed

> Rename SegmentInfo.docCount -> .maxDoc
> --
>
> Key: LUCENE-6307
> URL: https://issues.apache.org/jira/browse/LUCENE-6307
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: Trunk, 5.x
>
> Attachments: LUCENE-6307.patch, LUCENE-6307.patch
>
>
> We already have maxDoc and numDocs, I think it's crazy we have a 3rd one 
> docCount.
> We should just rename to maxDoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7115) UpdateLog can miss closing transaction log objects.

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343464#comment-14343464
 ] 

Yonik Seeley commented on SOLR-7115:


It's weird that existing stress tests don't catch this... I want to see if I 
can get some of those to fail.

> UpdateLog can miss closing transaction log objects.
> ---
>
> Key: SOLR-7115
> URL: https://issues.apache.org/jira/browse/SOLR-7115
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>
> I've seen this happen on YourKit and in various tests - especially since 
> adding resource release tracking to the log objects. Now I've got a test that 
> catches it in SOLR-7113.
> It seems that in precommit, if prevTlog is not null, we need to close it 
> because we are going to overwrite prevTlog with a new log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Anshum Gupta to the PMC

[Noble Paul]

congrats Anshum

On Mon, Mar 2, 2015 at 4:09 PM, Varun Thacker 
wrote:

> Congratulations Anshum!
>
> On Mon, Mar 2, 2015 at 3:31 AM, Jan Høydahl  wrote:
>
>> Welcome Anshum!
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> > 27. feb. 2015 kl. 18.41 skrev Steve Rowe :
>> >
>> > I'm pleased to announce that Anshum Gupta has accepted the PMC’s
>> invitation to join.
>> >
>> > Welcome Anshum!
>> >
>> > Steve
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>
>
> --
>
>
> Regards,
> Varun Thacker
> http://www.vthacker.in/
>



-- 
-
Noble Paul

RE: reuseAddress default in Solr jetty.xml

[Reitzel, Charles]

Hi Ram,

It appears the problem is that the old solr/jetty process is actually still 
running when the new solr/jetty process is started.   That’s the problem that 
needs fixing.

This is not a rare problem in systems with worker threads dedicated to 
different tasks.   These threads need to wake up in response to the shutdown 
signal/command, as well the normal inputs.

It’s a bug I’ve created and fixed a couple times over the years … :-)I 
wouldn’t know where to start with Solr.  But, as I say, re-using the port is a 
band-aid.  I’ve yet to see a case where it is the best solution.

best,
Charlie

From: Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com]
Sent: Saturday, February 28, 2015 8:15 PM
To: dev@lucene.apache.org
Subject: Re: reuseAddress default in Solr jetty.xml

Hey Charles, see my explanation above on why this is needed. If Solr has to be 
killed, it would generally be immediately restarted. This would normally not 
the case, except when things are potentially misconfigured or if there is a 
bug, but not doing so makes the impact worse..
In any case, turns out really that reuseAddress is true by default for the 
connectors we use, so that really isn't the issue. The issue more specifically 
is that the stop port doesn't do it, so the actual port by itself starts just 
fine on a restart, but the stop port fails to bind -- and there's no way 
currently in Jetty to configure that.
Based on my question in the jetty mailing list, I have now created an issue for 
them..

https://bugs.eclipse.org/bugs/show_bug.cgi?id=461133

On Fri, Feb 27, 2015 at 3:03 PM, Reitzel, Charles 
mailto:charles.reit...@tiaa-cref.org>> wrote:
Disclaimer: I’m not a Solr committer.  But, as a developer, I’ve never seen a 
good case for reusing the listening port.   Better to find and fix the root 
cause on the zombie state (or just slow shutdown, sometimes) and release the 
port.

From: Mark Miller [mailto:markrmil...@gmail.com]
Sent: Thursday, February 26, 2015 5:28 PM
To: dev@lucene.apache.org
Subject: Re: reuseAddress default in Solr jetty.xml

+1

- Mark

On Thu, Feb 26, 2015 at 1:54 PM Ramkumar R. Aiyengar 
mailto:andyetitmo...@gmail.com>> wrote:
The jetty.xml we currently ship by default doesn't set reuseAddress=true. If 
you are having a bad GC day with things going OOM and resulting in Solr not 
even being able to shutdown cleanly (or the oom_solr.sh script killing it), 
whatever external service management mechanism you have is probably going to 
try respawn it and fail with the default config because the ports will be in 
TIME_WAIT. I guess there's the usual disclaimer with reuseAddress causing stray 
packets to reach the restarted server, but sounds like at least the default 
should be true..

I can raise a JIRA, but just wanted to check if anyone has any opinions either 
way..


*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA-CREF
*



--
Not sent from my iPhone or my Blackberry or anyone else's

*
This e-mail may contain confidential or privileged information.
If you are not the intended recipient, please notify the sender immediately and 
then delete it.

TIAA-CREF
*

[jira] [Commented] (LUCENE-6307) Rename SegmentInfo.docCount -> .maxDoc

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343420#comment-14343420
 ] 

ASF subversion and git services commented on LUCENE-6307:
-

Commit 1663371 from [~mikemccand] in branch 'dev/trunk'
[ https://svn.apache.org/r1663371 ]

LUCENE-6307: rename confusing docCount -> maxDoc in several places

> Rename SegmentInfo.docCount -> .maxDoc
> --
>
> Key: LUCENE-6307
> URL: https://issues.apache.org/jira/browse/LUCENE-6307
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: Trunk, 5.x
>
> Attachments: LUCENE-6307.patch, LUCENE-6307.patch
>
>
> We already have maxDoc and numDocs, I think it's crazy we have a 3rd one 
> docCount.
> We should just rename to maxDoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_31) - Build # 4518 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4518/
Java: 32bit/jdk1.8.0_31 -client -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ReplicationFactorTest.test

Error Message:
org.apache.http.NoHttpResponseException: The target server failed to respond

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.http.NoHttpResponseException: The target server failed to respond
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:899)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
at 
org.apache.solr.cloud.ReplicationFactorTest.testRf3(ReplicationFactorTest.java:284)
at 
org.apache.solr.cloud.ReplicationFactorTest.test(ReplicationFactorTest.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:945)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:920)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org

[jira] [Commented] (SOLR-5332) Add "preserve original" setting to the EdgeNGramFilterFactory

[Simon Endele (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343414#comment-14343414
 ] 

Simon Endele commented on SOLR-5332:


+1 for this feature.
We use the EdgeNGramFilterFactory on a tokenized field (in order to implement a 
"prefix search" on index time) with minGramSize="3".
Unfortunately we observed that tokens with length 1 or 2 are actually deleted, 
unexpectedly from our point of view.

Using a second field (though complicated IMHO) would address query-issues, but 
it gets awkward when it comes to highlighting or phrase searches.
For instance when searching for "us rep"
- the field with EdgeNGramFilterFactory highlights "rep" in "representative", 
but not "US" as this token has been removed,
- the field without EdgeNGramFilterFactory highlights "US", but not 
"representative" as it has no prefixes indexed.

Bringing these highlightings together in one string is a quite complex task.
Not speaking of a phrase search, which does not work at all for the example 
above.

We use minGramSize="3" to reduce collisions of prefixes and abbreviations (like 
"US" and "usage") and reduce the index size.
I admit, this does not prevent all collisions (e.g. "USA" still collides with 
"usage"), but it's a compromise.

Nevertheless, minGramSize is a nice feature of EdgeNGramFilterFactory, but it 
lacks a "preserveOriginal" flag IMO.

> Add "preserve original" setting to the EdgeNGramFilterFactory
> -
>
> Key: SOLR-5332
> URL: https://issues.apache.org/jira/browse/SOLR-5332
> Project: Solr
>  Issue Type: Wish
>Affects Versions: 4.4, 4.5, 4.5.1, 4.6
>Reporter: Alexander S.
>
> Hi, as described here: 
> http://lucene.472066.n3.nabble.com/Help-to-figure-out-why-query-does-not-match-td4086967.html
>  the problem is in that if you have these 2 strings to index:
> 1. facebook.com/someuser.1
> 2. facebook.com/someveryandverylongusername
> and the edge ngram filter factory with min and max gram size settings 2 and 
> 25, search requests for these urls will fail.
> But search requests for:
> 1. facebook.com/someuser
> 2. facebook.com/someveryandverylonguserna
> will work properly.
> It's because first url has "1" at the end, which is lover than the allowed 
> min gram size. In the second url the user name is longer than the max gram 
> size (27 characters).
> Would be good to have a "preserve original" option, that will add the 
> original string to the index if it does not fit the allowed gram size, so 
> that "1" and "someveryandverylongusername" tokens will also be added to the 
> index.
> Best,
> Alex



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 2703 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/2703/

6 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedMLTComponentTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:53613//collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:53613//collection1
at 
__randomizedtesting.SeedInfo.seed([BB59CB42BFC54E17:330DF498113923EF]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:568)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:131)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:309)
at 
org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:543)
at 
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:591)
at 
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:573)
at 
org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:552)
at 
org.apache.solr.handler.component.DistributedMLTComponentTest.test(DistributedMLTComponentTest.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:945)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:920)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverrides

[jira] [Commented] (SOLR-7173) Fix ReplicationFactorTest on Windows

[Timothy Potter (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343348#comment-14343348
 ] 

Timothy Potter commented on SOLR-7173:
--

[~ichattopadhyaya] thanks for taking this up ... a couple of minor suggestions 
about your patch:

1) please add some logging around the retries

2) I've found it better to use recursion with a decrementing counter vs. nested 
re-try blocks so something like the following, see where I use maxRetries as a 
param:

{code}
protected int sendBatch(List batch, int waitBeforeRetry, int 
maxRetries) throws Exception {
int sent = 0;
final Timer.Context sendTimerCtxt = sendBatchToSolrTimer.time();
try {
  UpdateRequest updateRequest = new UpdateRequest();
  ModifiableSolrParams params = updateRequest.getParams();
  if (params == null) {
params = new ModifiableSolrParams();
updateRequest.setParams(params);
  }
  updateRequest.add(batch);
  cloudSolrServer.request(updateRequest);
  sent = batch.size();
} catch (Exception exc) {
  Throwable rootCause = SolrException.getRootCause(exc);
  boolean wasCommError = ...
  if (wasCommError) {
if (--maxRetries > 0) {
  log.warn("ERROR: " + rootCause + " ... Sleeping for "
  + waitBeforeRetry + " seconds before re-try ...");
  Thread.sleep(waitBeforeRetry * 1000L);
  sent = sendBatch(batch, waitBeforeRetry, maxRetries);
} else {
  log.error("No more retries available! Add batch failed due to: " + 
rootCause);
  throw exc;
}
  }
} finally {
  sendTimerCtxt.stop();
}

batch.clear();
return sent;
  }
{code}

> Fix ReplicationFactorTest on Windows
> 
>
> Key: SOLR-7173
> URL: https://issues.apache.org/jira/browse/SOLR-7173
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
> Fix For: 5.1
>
> Attachments: SOLR-7173.patch
>
>
> The ReplicationFactorTest fails on the Windows build with 
> NoHttpResponseException, as seen here: 
> http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4502/testReport/junit/org.apache.solr.cloud/ReplicationFactorTest/test/
> Adding a retry logic similar to HttpPartitionTest's doSend() method makes the 
> test pass on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6322) IndexSearcher.doc(int docID, SetfieldsToLoad) is slower in Lucene 4.9 when compared to Lucene 2.9

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343325#comment-14343325
 ] 

Adrien Grand commented on LUCENE-6322:
--

Agreed it would be nice to skip over compressed blocks when they are not needed 
instead of decompressing and then discarding the decompressed bytes. I was just 
looking at the impl and it seems that to make it work we would need to store 
the compressed length of each block and implement skipBytes on the anonymous 
DataInput created in CompressingStoredFieldsReader.document.

> IndexSearcher.doc(int docID, SetfieldsToLoad)  is slower in Lucene 4.9 when 
> compared to Lucene 2.9
> --
>
> Key: LUCENE-6322
> URL: https://issues.apache.org/jira/browse/LUCENE-6322
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 4.9
> Environment: Windows, JDK 7/8
>Reporter: Sekhar
> Fix For: 4.10.x
>
>
> We use IndexSearcher.doc(int docID, SetfieldsToLoad) method to get the 
> document with selected stored fields. If we did not mention few stored fields 
> which have data more than 500KB, this call is slower in Lucene 4.9 when 
> compared to Lucene 2.9.
> I debugged the above method with Lucene 4.9 and found that 
> CompressingStoredFieldsReader#visitDocument(int docID, StoredFieldVisitor 
> visitor) is spending more time while loading file content and decompressing 
> in chunks of 16kb, even to skip the fields. It is noticeable degrade if the 
> document's field size is more than 1MB, and we call this method in loop for 
> more than 1000 such documents.
> In case of Lucene 2.9, there was no compression, and if we want to skip the 
> field, it just does file seek to set the next pointer to read the stored 
> field. For example see Lucene3xStoredFieldsReader#skipField() method how it 
> works for skipping a field in Lucene 2.9 which is VERY faster compared to 
> Lucene 4.9.
> We should have something in CompressingStoredFieldsReader to know the field’s 
> compressed length in file and just do the file seek to set the next pointer 
> instead of loading content from file and decompress that in 16KB chunks to 
> just skip the field from the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6326) MultiCollector does not handle CollectionTerminatedException correctly

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343327#comment-14343327
 ] 

Adrien Grand commented on LUCENE-6326:
--

+1 to make MultiCollector catch this exception. I think this is more intuitive 
as it would make the sub collectors collected as if they was no other collector.

> MultiCollector does not handle CollectionTerminatedException correctly
> --
>
> Key: LUCENE-6326
> URL: https://issues.apache.org/jira/browse/LUCENE-6326
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/search
>Affects Versions: 5.0
>Reporter: Christoph Kaser
>Priority: Minor
>
> The javadoc of the *collect*-method of LeafCollector states:
> bq. Note: The collection of the current segment can be terminated by throwing 
> a CollectionTerminatedException.
> However, the Multicollector does not catch this exception, so if one of the 
> wrapped collectors terminates the current segment, it is terminated for every 
> collector.
> The same is true for the *getLeafCollector*-method (even though this is not 
> documented in the JavaDoc).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Ryan Ernst]

Welcome!
On Mar 1, 2015 8:40 PM, "Shalin Shekhar Mangar" 
wrote:

> I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's
> invitation to become a committer.
>
> Ramkumar, it's tradition that you introduce yourself with a brief bio.
>
> Your handle "andyetitmoves" has already added to the “lucene" LDAP group,
> so you now have commit privileges. Please test this by adding yourself to
> the committers section of the Who We Are page on the website: <
> http://lucene.apache.org/whoweare.html> (use the ASF CMS bookmarklet at
> the bottom of the page here:  - more
> info here ).
>
> The ASF dev page also has lots of useful links: <
> http://www.apache.org/dev/>.
>
> Congratulations and welcome!
>
> --
> Regards,
> Shalin Shekhar Mangar.
>

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Adrien Grand]

Welcome Ramkumar!

On Mon, Mar 2, 2015 at 5:39 AM, Shalin Shekhar Mangar
 wrote:
> I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's
> invitation to become a committer.
>
> Ramkumar, it's tradition that you introduce yourself with a brief bio.
>
> Your handle "andyetitmoves" has already added to the “lucene" LDAP group, so
> you now have commit privileges. Please test this by adding yourself to the
> committers section of the Who We Are page on the website:
>  (use the ASF CMS bookmarklet at the
> bottom of the page here:  - more info here
> ).
>
> The ASF dev page also has lots of useful links:
> .
>
> Congratulations and welcome!
>
> --
> Regards,
> Shalin Shekhar Mangar.



-- 
Adrien

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-1518) Merge Query and Filter classes

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343296#comment-14343296
 ] 

Adrien Grand commented on LUCENE-1518:
--

I am considering reverting this change: in the beginning it felt like a nice 
step to help the filter -> query transition but it ends up being quite trappy. 
For instance existing filters inherit from the default clone implementation 
which might not clone deeply enough, and existing "equals" impls do not pass 
QueryUtils checks since they do not take the boost into account.

> Merge Query and Filter classes
> --
>
> Key: LUCENE-1518
> URL: https://issues.apache.org/jira/browse/LUCENE-1518
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 2.4
>Reporter: Uwe Schindler
> Fix For: Trunk, 5.1
>
> Attachments: LUCENE-1518.patch, LUCENE-1518.patch
>
>
> This issue presents a patch, that merges Queries and Filters in a way, that 
> the new Filter class extends Query. This would make it possible, to use every 
> filter as a query.
> The new abstract filter class would contain all methods of 
> ConstantScoreQuery, deprecate ConstantScoreQuery. If somebody implements the 
> Filter's getDocIdSet()/bits() methods he has nothing more to do, he could 
> just use the filter as a normal query.
> I do not want to completely convert Filters to ConstantScoreQueries. The idea 
> is to combine Queries and Filters in such a way, that every Filter can 
> automatically be used at all places where a Query can be used (e.g. also 
> alone a search query without any other constraint). For that, the abstract 
> Query methods must be implemented and return a "default" weight for Filters 
> which is the current ConstantScore Logic. If the filter is used as a real 
> filter (where the API wants a Filter), the getDocIdSet part could be directly 
> used, the weight is useless (as it is currently, too). The constant score 
> default implementation is only used when the Filter is used as a Query (e.g. 
> as direct parameter to Searcher.search()). For the special case of 
> BooleanQueries combining Filters and Queries the idea is, to optimize the 
> BooleanQuery logic in such a way, that it detects if a BooleanClause is a 
> Filter (using instanceof) and then directly uses the Filter API and not take 
> the burden of the ConstantScoreQuery (see LUCENE-1345).
> Here some ideas how to implement Searcher.search() with Query and Filter:
> - User runs Searcher.search() using a Filter as the only parameter. As every 
> Filter is also a ConstantScoreQuery, the query can be executed and returns 
> score 1.0 for all matching documents.
> - User runs Searcher.search() using a Query as the only parameter: No change, 
> all is the same as before
> - User runs Searcher.search() using a BooleanQuery as parameter: If the 
> BooleanQuery does not contain a Query that is subclass of Filter (the new 
> Filter) everything as usual. If the BooleanQuery only contains exactly one 
> Filter and nothing else the Filter is used as a constant score query. If 
> BooleanQuery contains clauses with Queries and Filters the new algorithm 
> could be used: The queries are executed and the results filtered with the 
> filters.
> For the user this has the main advantage: That he can construct his query 
> using a simplified API without thinking about Filters oder Queries, you can 
> just combine clauses together. The scorer/weight logic then identifies the 
> cases to use the filter or the query weight API. Just like the query 
> optimizer of a RDB.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7181) Unable to show the indexed content in Solr 5.0

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343293#comment-14343293
 ] 

Erick Erickson commented on SOLR-7181:
--

I've asked you to move the conversation to the user's list, please do so now as 
this is the last time I'll respond in this forum. I'll admit that this looks 
odd, I don't quite understand what's happening here. The user's list will get a 
lot more eyeballs on the problem.

You still haven't done what I asked, namely search for 

q=content:[* TO *] 

to insure that you did, indeed, get data into content field. It's still quite 
possible that you
didn't for some reason. A query like

q=content:[* TO *]&fl=id,content

is about the minimal query I can imagine. If this comes back with numFound > 0 
and no content
then we need the exact steps you followed, your configuration for Extracting 
Reqeust Handler, how
you sent the command to the handler, a copy of the pdf document, in short 
enough information to
reproduce the steps you followed.

> Unable to show the indexed content in Solr 5.0
> --
>
> Key: SOLR-7181
> URL: https://issues.apache.org/jira/browse/SOLR-7181
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.0
>Reporter: Yeo Zheng Lin
>
> The content field is unable to be shown during searching, even though the 
> following line has been added to the schema using curl from the resource 
> named in 'managedSchemaResourceName'. 
> 
> I'm using the schema from ManagedIndexSchemaFactory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

[Per Steffensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
 ] 

Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--

I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK 
operations, like uploading and downloading. Changing then content of a 
json-file is not a ZK operation. Why not see it as different things. So what 
you need to do if you want to make arbitrary changes to json-content of a znode 
in ZK is
{code}
zkcli getfile thefile.json

zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the 
different types of jsons we have in a safe way - ensuring e.g. consistency, 
integrity, validity etc. But I think those tools should use the classes already 
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for 
manipulating clusterstate.json should use classes from 
org.apache.solr.common.cloud in solrj project, like ZkStateReader, 
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are 
used to operate on a specific type of json, and we only have to build 
consistency, integrity, validity etc into those classes (separation of 
concern). The new thing should be that those classes can be used via an 
external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually 
reading/writing/verifying the jsons, and make separate tool-classes. Changing 
ZkCLI to be able to trigger operations in those tool-classes is less important, 
but personally I do not like - actually, if it has to be, I would rather see 
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool 
to download first, do its manipulations and then use ZkCLI to upload again. 
That is, new tools that use ZkCLI (if you are too lazy doing the 
download/upload using ZkCLI yourself), instead of the other way around, where 
ZkCLI uses other tools or even just does json-manipulation itself. Please do 
not try to implement rules about what can and cannot be in a specific type of 
json, what operations you can do on it etc two places in the code-base - use 
what we already have.

Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being 
consistent, valid etc. You never do changes to clusterstate.json with using 
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json. 
You never manipulate a clusterstate.json without going through 
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use 
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so 
that it support receiving jobs not coming from overseer queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be 
the same. It is just that I know more about which classes handle cluster-state 
than which handles cluster-props.


That said, maybe you can live with using e.g. http://trentm.com/json/ (or one 
of the million other command-line json tools available) in  above?


was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK 
operations, like uploading and downloading. Changing then content of a 
json-file is not a ZK operation. Why not see it as different things. So what 
you need to do if you want to make arbitrary changes to json-content of a znode 
in ZK is
{code}
zkcli getfile thefile.json

zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the 
different types of jsons we have in a safe way - ensuring e.g. consistency, 
integrity, validity etc. But I think those tools should use the classes already 
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for 
manipulating clusterstate.json should use classes from 
org.apache.solr.common.cloud in solrj project, like ZkStateReader, ClusterState 
etc. Then it is always those classes that are used to operate on a specific 
type of json, and we only have to build consistency, integrity, validity etc 
into those classes (separation of concern). The new thing should be that those 
classes can be used via an external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually 
reading/writing/verifying the jsons, and make separate tool-classes. Changing 
ZkCLI to be able to trigger operations in those tool-classes is less important, 
but personally I do not like - actually, if it has to be, I would rather see 
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool 
to download first, do its manipulations and then use ZkCLI to upload again. 
That is, new tools that use ZkCLI (if you are too lazy doing the 
download/

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

[Per Steffensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
 ] 

Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:48 PM:
--

I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK 
operations, like uploading and downloading. Changing then content of a 
json-file is not a ZK operation. Why not see it as different things. So what 
you need to do if you want to make arbitrary changes to json-content of a znode 
in ZK is
{code}
zkcli getfile thefile.json

zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the 
different types of jsons we have in a safe way - ensuring e.g. consistency, 
integrity, validity etc. But I think those tools should use the classes already 
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for 
manipulating clusterstate.json should use classes from 
org.apache.solr.common.cloud in solrj project, like ZkStateReader, 
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are 
used to operate on a specific type of json, and we only have to build 
consistency, integrity, validity etc into those classes (separation of 
concern). The new thing should be that those classes can be used via an 
external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually 
reading/writing/verifying the jsons, and make separate tool-classes. Changing 
ZkCLI to be able to trigger operations in those tool-classes is less important, 
but personally I do not like - actually, if it has to be, I would rather see 
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool 
to download first, do its manipulations and then use ZkCLI to upload again. 
That is, new tools that use ZkCLI (if you are too lazy doing the 
download/upload using ZkCLI yourself), instead of the other way around, where 
ZkCLI uses other tools or even just does json-manipulation itself. Please do 
not try to implement rules about what can and cannot be in a specific type of 
json, what operations you can do on it etc two places in the code-base - use 
what we already have.

Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being 
consistent, valid etc. You never do changes to clusterstate.json with using 
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json. 
You never manipulate a clusterstate.json without going through 
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use 
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so 
that it support receiving jobs not coming from other sources than overseer 
queue.

Yes, I know you primarily talk about ClusterProps, but the principle will be 
the same. It is just that I know more about which classes handle cluster-state 
than which handle cluster-props.

That said, maybe you can live with using e.g. http://trentm.com/json/ (or one 
of the million other command-line json tools available) in  above?


was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK 
operations, like uploading and downloading. Changing then content of a 
json-file is not a ZK operation. Why not see it as different things. So what 
you need to do if you want to make arbitrary changes to json-content of a znode 
in ZK is
{code}
zkcli getfile thefile.json

zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the 
different types of jsons we have in a safe way - ensuring e.g. consistency, 
integrity, validity etc. But I think those tools should use the classes already 
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for 
manipulating clusterstate.json should use classes from 
org.apache.solr.common.cloud in solrj project, like ZkStateReader, 
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are 
used to operate on a specific type of json, and we only have to build 
consistency, integrity, validity etc into those classes (separation of 
concern). The new thing should be that those classes can be used via an 
external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually 
reading/writing/verifying the jsons, and make separate tool-classes. Changing 
ZkCLI to be able to trigger operations in those tool-classes is less important, 
but personally I do not like - actually, if it has to be, I would rather see 
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool 
to download first, do its manipulations and then use ZkCLI to upload again. 
That is, new tools that use ZkCLI 

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

[Per Steffensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
 ] 

Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--

I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK 
operations, like uploading and downloading. Changing then content of a 
json-file is not a ZK operation. Why not see it as different things. So what 
you need to do if you want to make arbitrary changes to json-content of a znode 
in ZK is
{code}
zkcli getfile thefile.json

zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the 
different types of jsons we have in a safe way - ensuring e.g. consistency, 
integrity, validity etc. But I think those tools should use the classes already 
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for 
manipulating clusterstate.json should use classes from 
org.apache.solr.common.cloud in solrj project, like ZkStateReader, 
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are 
used to operate on a specific type of json, and we only have to build 
consistency, integrity, validity etc into those classes (separation of 
concern). The new thing should be that those classes can be used via an 
external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually 
reading/writing/verifying the jsons, and make separate tool-classes. Changing 
ZkCLI to be able to trigger operations in those tool-classes is less important, 
but personally I do not like - actually, if it has to be, I would rather see 
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool 
to download first, do its manipulations and then use ZkCLI to upload again. 
That is, new tools that use ZkCLI (if you are too lazy doing the 
download/upload using ZkCLI yourself), instead of the other way around, where 
ZkCLI uses other tools or even just does json-manipulation itself. Please do 
not try to implement rules about what can and cannot be in a specific type of 
json, what operations you can do on it etc two places in the code-base - use 
what we already have.

Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being 
consistent, valid etc. You never do changes to clusterstate.json with using 
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json. 
You never manipulate a clusterstate.json without going through 
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use 
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so 
that it support receiving jobs not coming from overseer queue.

Yes, I know you primarily talk about ClusterProps, but the principle will be 
the same. It is just that I know more about which classes handle cluster-state 
than which handles cluster-props.


That said, maybe you can live with using e.g. http://trentm.com/json/ (or one 
of the million other command-line json tools available) in  above?


was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK 
operations, like uploading and downloading. Changing then content of a 
json-file is not a ZK operation. Why not see it as different things. So what 
you need to do if you want to make arbitrary changes to json-content of a znode 
in ZK is
{code}
zkcli getfile thefile.json

zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the 
different types of jsons we have in a safe way - ensuring e.g. consistency, 
integrity, validity etc. But I think those tools should use the classes already 
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for 
manipulating clusterstate.json should use classes from 
org.apache.solr.common.cloud in solrj project, like ZkStateReader, 
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are 
used to operate on a specific type of json, and we only have to build 
consistency, integrity, validity etc into those classes (separation of 
concern). The new thing should be that those classes can be used via an 
external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually 
reading/writing/verifying the jsons, and make separate tool-classes. Changing 
ZkCLI to be able to trigger operations in those tool-classes is less important, 
but personally I do not like - actually, if it has to be, I would rather see 
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool 
to download first, do its manipulations and then use ZkCLI to upload again. 
That is, new tools that use ZkCLI (if you are too la

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

[Per Steffensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343271#comment-14343271
 ] 

Per Steffensen commented on SOLR-7176:
--

I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK 
operations, like uploading and downloading. Changing then content of a 
json-file is not a ZK operation. Why not see it as different things. So what 
you need to do if you want to make arbitrary changes to json-content of a znode 
in ZK is
{code}
zkcli getfile thefile.json

zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the 
different types of jsons we have in a safe way - ensuring e.g. consistency, 
integrity, validity etc. But I think those tools should use the classes already 
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for 
manipulating clusterstate.json should use classes from 
org.apache.solr.common.cloud in solrj project, like ZkStateReader, ClusterState 
etc. Then it is always those classes that are used to operate on a specific 
type of json, and we only have to build consistency, integrity, validity etc 
into those classes (separation of concern). The new thing should be that those 
classes can be used via an external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually 
reading/writing/verifying the jsons, and make separate tool-classes. Changing 
ZkCLI to be able to trigger operations in those tool-classes is less important, 
but personally I do not like - actually, if it has to be, I would rather see 
e.g. an ClusterPropCLI-tool support a "do-in-zk"-flag making it use ZkCLI tool 
to download first, do its manipulations and then use ZkCLI to upload again. 
That is, new tools that use ZkCLI (if you are too lazy doing the 
download/upload using ZkCLI yourself), instead of the other way around, where 
ZkCLI uses other tools or even just does json-manipulation itself. Please do 
not try to implement rules about what can and cannot be in a specific type of 
json, what operations you can do on it etc two places in the code-base - use 
what we already have.

That said, maybe you can live with using e.g. http://trentm.com/json/ (or one 
of the million other command-line json tools available) in  above?

> allow zkcli to modify JSON
> --
>
> Key: SOLR-7176
> URL: https://issues.apache.org/jira/browse/SOLR-7176
> Project: Solr
>  Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Minor
>
> To enable SSL, we have instructions like the following:
> {code}
> server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
> /clusterprops.json '{"urlScheme":"https"}'
> {code}
> Overwriting the value won't work well when we have more properties to put in 
> clusterprops.  We should be able to change individual values or perhaps merge 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6320) speed up checkindex

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343260#comment-14343260
 ] 

Michael McCandless commented on LUCENE-6320:


+1

> speed up checkindex
> ---
>
> Key: LUCENE-6320
> URL: https://issues.apache.org/jira/browse/LUCENE-6320
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Robert Muir
> Attachments: LUCENE-6320.patch, LUCENE-6320.patch
>
>
> This is fairly slow today, very ram intensive, and has some buggy stuff (e.g. 
> postingsenum reuse bugs). We can do better...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6321) Make oal.index.Term more defensive

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-6321:
-
Attachment: LUCENE-6321.patch

Here is a patch which makes TermQuery.clone() also clone the wrapped term. The 
patch also fixes:
 - (Multi)PhraseQuery,
 - SpanTermQuery,
 - CommonTermsQuery,
 - BooleanQuery (which did not clone the wrapped queries)
 - DisjunctionMaxQuery (same reason)

> Make oal.index.Term more defensive
> --
>
> Key: LUCENE-6321
> URL: https://issues.apache.org/jira/browse/LUCENE-6321
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Adrien Grand
>Assignee: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-6321.patch
>
>
> oal.index.Term has a Term(String field, BytesRef termBytes) constructor. Even 
> though it warns that the term bytes should not be reused, I'm wondering that 
> we should make it more defensive.
> {noformat}
>* WARNING: the provided BytesRef is not copied, but used directly.
>* Therefore the bytes should not be modified after construction, for
>* example, you should clone a copy by {@link BytesRef#deepCopyOf}
>* rather than pass reused bytes from a TermsEnum.
> {noformat} 
> For example if you have term queries in your query cache and they are 
> modified in-place, it would have very bad consequences and would be hard to 
> diagnose.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-6326) MultiCollector does not handle CollectionTerminatedException correctly

[Christoph Kaser (JIRA)]

Christoph Kaser created LUCENE-6326:
---

 Summary: MultiCollector does not handle 
CollectionTerminatedException correctly
 Key: LUCENE-6326
 URL: https://issues.apache.org/jira/browse/LUCENE-6326
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/search
Affects Versions: 5.0
Reporter: Christoph Kaser
Priority: Minor


The javadoc of the *collect*-method of LeafCollector states:
bq. Note: The collection of the current segment can be terminated by throwing a 
CollectionTerminatedException.
However, the Multicollector does not catch this exception, so if one of the 
wrapped collectors terminates the current segment, it is terminated for every 
collector.
The same is true for the *getLeafCollector*-method (even though this is not 
documented in the JavaDoc).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-MacOSX (64bit/jdk1.7.0) - Build # 2024 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-MacOSX/1981/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.TestSolrConfigHandlerCloud.test

Error Message:
Could not get expected value  'A val' for path 'params/a' full output: null

Stack Trace:
java.lang.AssertionError: Could not get expected value  'A val' for path 
'params/a' full output: null
at 
__randomizedtesting.SeedInfo.seed([EAC109BAFBAD3143:6295366055515CBB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:399)
at 
org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqParams(TestSolrConfigHandlerCloud.java:166)
at 
org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:945)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:920)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apac

[GitHub] lucene-solr pull request: close connection when done processing ea...

[rust82]

Github user rust82 closed the pull request at:

https://github.com/apache/lucene-solr/pull/130


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Ramkumar R. Aiyengar]

Thanks everyone!

I lead the News Search backend team at the London R&D office of Bloomberg.
I have been at this job for a little more than 7 years, and joined straight
from my university at IIT Madras, Chennai, India where I did my bachelors
and masters in CS. I started working with Lucene/Solr around two and a half
years back, when we decided to rewrite our entire search and alerting
backend, something used by every Bloomberg subscriber to get near real-time
access to news with sub-second search/alerting latencies. I have mostly
dabbled with Solr's search distribution and cloud code since, and have had
some "fun" experiences with it when SolrCloud was, well, let's say less
mature.. :)

I consider myself a Linux geek/evangelist, and outside Java, still use
Emacs and describe Lisp as a beautiful language (As for Java, was Eclipse,
and then thanks Alan for showing me IntelliJ! :) )

On Mon, Mar 2, 2015 at 4:39 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's
> invitation to become a committer.
>
> Ramkumar, it's tradition that you introduce yourself with a brief bio.
>
> Your handle "andyetitmoves" has already added to the “lucene" LDAP group,
> so you now have commit privileges. Please test this by adding yourself to
> the committers section of the Who We Are page on the website: <
> http://lucene.apache.org/whoweare.html> (use the ASF CMS bookmarklet at
> the bottom of the page here:  - more
> info here ).
>
> The ASF dev page also has lots of useful links: <
> http://www.apache.org/dev/>.
>
> Congratulations and welcome!
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
Not sent from my iPhone or my Blackberry or anyone else's

[jira] [Commented] (SOLR-7139) SolrContentHandler for TIKA is broken by TikaOCR (caused by multiple startDocument events)

[Chris A. Mattmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343233#comment-14343233
 ] 

Chris A. Mattmann commented on SOLR-7139:
-

awesome thanks Uwe and Mike. Will report back.

> SolrContentHandler for TIKA is broken by TikaOCR (caused by multiple 
> startDocument events)
> --
>
> Key: SOLR-7139
> URL: https://issues.apache.org/jira/browse/SOLR-7139
> Project: Solr
>  Issue Type: Bug
>  Components: contrib - Solr Cell (Tika extraction)
>Affects Versions: 4.10.3
>Reporter: Chris A. Mattmann
>Assignee: Uwe Schindler
>Priority: Critical
> Fix For: 4.10.4, Trunk, 5.1
>
> Attachments: SOLR-7139.Mattmann.022115.patch.txt, SOLR-7139.patch
>
>
> While testing my large scale Tika/SolrCell indexing (great work on 
> /extraction guys, really really appreciate it) on my 40M image dataset, I was 
> pulling my frickin' hair out trying to figure out why the TesseractOCR 
> extracted content wasn't actually making it into the index. Well I figured it 
> out lol (many many System.out.printlns later) - it's the disabling of div 
> tags (=>ignored) in the default solrconfig.xml. This basically renders 
> TesseractOCR output in SolrCell useless since it is surrounded by a div tag.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request: close connection when done processing ea...

[rust82]

GitHub user rust82 opened a pull request:

https://github.com/apache/lucene-solr/pull/130

close connection when done processing each entity



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rust82/lucene-solr lucene_solr_5_0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/130.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #130


commit f453007d986050c03cb3a126d747c3e1f44458b5
Author: rust82 
Date:   2015-03-02T15:05:39Z

close connection when done processing each entity




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Steve Rowe]

Welcome Ramkumar!

Steve

> On Mar 1, 2015, at 11:39 PM, Shalin Shekhar Mangar  
> wrote:
> 
> I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's 
> invitation to become a committer.
> 
> Ramkumar, it's tradition that you introduce yourself with a brief bio.
> 
> Your handle "andyetitmoves" has already added to the “lucene" LDAP group, so 
> you now have commit privileges. Please test this by adding yourself to the 
> committers section of the Who We Are page on the website: 
>  (use the ASF CMS bookmarklet at the 
> bottom of the page here:  - more info here 
> ).
> 
> The ASF dev page also has lots of useful links: .
> 
> Congratulations and welcome!
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 774 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/774/

1 tests failed.
REGRESSION:  
org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage

Error Message:
forceMerge used too much temporary space: starting usage was 379542 bytes; 
final usage was 442916 bytes; max temp usage was 1669519 but should have been 
1328748 (= 3X starting usage)

Stack Trace:
java.lang.AssertionError: forceMerge used too much temporary space: starting 
usage was 379542 bytes; final usage was 442916 bytes; max temp usage was 
1669519 but should have been 1328748 (= 3X starting usage)
at 
__randomizedtesting.SeedInfo.seed([AD6008DD6F02F612:B7A2CB2E011215D5]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.lucene.index.TestIndexWriterForceMerge.testForceMergeTempSpaceUsage(TestIndexWriterForceMerge.java:181)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 999 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterForceMerge
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 
'ant get-jenkin

Re: DocValues instead of stored values

[david.w.smi...@gmail.com]

On Mon, Mar 2, 2015 at 9:13 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> The problem with fetching from DocValue automatically is that it may cause
> as many number of disk seeks as the number of doc value fields being
> retrieved.


True in the worst-case (cold disk cache).  If you have warm-up and only use
this technique judiciously (on a few docvalues fields), then one can assume
the data is cached.  And you can explicitly use a memory based DV format
too, which puts all the values in a memory-efficient FST assuming
string/text data.

~ David Smiley
Freelance Apache Lucene/Solr Search Consultant/Developer
http://www.linkedin.com/in/davidwsmiley

Re: DocValues instead of stored values

[Shalin Shekhar Mangar]

Hmm, this is indeed not documented. I'll fix.

On Mon, Mar 2, 2015 at 7:54 PM, Toke Eskildsen 
wrote:

> On Mon, 2015-03-02 at 15:13 +0100, Shalin Shekhar Mangar wrote:
> > It is already possible. Just use fl=field(my_dv_field) or with an
> > alias fl=my_alias:field(my_dv_field).
> >
> Happy days!
>
> Thank you, Shalin. Can you point me to a description of this mechanism?
> I could not find it on the wiki or in the reference guide (my Google-fu
> is weak).
>
> - Toke Eskildsen, State and University Library, Denmark
>
>
>
>


-- 
Regards,
Shalin Shekhar Mangar.

Re: Welcome Ramkumar Aiyengar as Lucene/Solr committer

[Joel Bernstein]

Congratulations Ram!

Joel Bernstein
Search Engineer at Heliosearch

On Mon, Mar 2, 2015 at 8:46 AM, david.w.smi...@gmail.com <
david.w.smi...@gmail.com> wrote:

> Welcome Ram!
>
> ~ David Smiley
> Freelance Apache Lucene/Solr Search Consultant/Developer
> http://www.linkedin.com/in/davidwsmiley
>
> On Sun, Mar 1, 2015 at 11:39 PM, Shalin Shekhar Mangar <
> shalinman...@gmail.com> wrote:
>
>> I'm pleased to announce that Ramkumar Aiyengar has accepted the PMC's
>> invitation to become a committer.
>>
>> Ramkumar, it's tradition that you introduce yourself with a brief bio.
>>
>> Your handle "andyetitmoves" has already added to the “lucene" LDAP group,
>> so you now have commit privileges. Please test this by adding yourself to
>> the committers section of the Who We Are page on the website: <
>> http://lucene.apache.org/whoweare.html> (use the ASF CMS bookmarklet at
>> the bottom of the page here:  - more
>> info here ).
>>
>> The ASF dev page also has lots of useful links: <
>> http://www.apache.org/dev/>.
>>
>> Congratulations and welcome!
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>
>

[jira] [Commented] (SOLR-5743) Faceting with BlockJoin support

[Dr Oleg Savrasov (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343189#comment-14343189
 ] 

Dr Oleg Savrasov commented on SOLR-5743:


Video from the Lucene Revolution talk is available here 
http://www.youtube.com/watch?v=Su5SHc_uJw8

> Faceting with BlockJoin support
> ---
>
> Key: SOLR-5743
> URL: https://issues.apache.org/jira/browse/SOLR-5743
> Project: Solr
>  Issue Type: New Feature
>Reporter: abipc
>  Labels: features
> Attachments: SOLR-5743.patch, SOLR-5743.patch, SOLR-5743.patch, 
> SOLR-5743.patch, SOLR-5743.patch
>
>
> For a sample inventory(note - nested documents) like this -   
>  
> 10
> parent
> Nike
> 
> 11
> Red
> XL
> 
> 
> 12
> Blue
> XL
> 
> 
> Faceting results must contain - 
> Red(1)
> XL(1) 
> Blue(1) 
> for a "q=*" query. 
> PS : The inventory example has been taken from this blog - 
> http://blog.griddynamics.com/2013/09/solr-block-join-support.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: DocValues instead of stored values

[Toke Eskildsen]

On Mon, 2015-03-02 at 15:13 +0100, Shalin Shekhar Mangar wrote:
> It is already possible. Just use fl=field(my_dv_field) or with an
> alias fl=my_alias:field(my_dv_field).
> 
Happy days!

Thank you, Shalin. Can you point me to a description of this mechanism?
I could not find it on the wiki or in the reference guide (my Google-fu
is weak).

- Toke Eskildsen, State and University Library, Denmark




-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org