Re: How to make solr fault tolerant for query?
i think with params is ok. -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-make-solr-fault-tolerant-for-query-tp4158411p4159574.html Sent from the Solr - User mailing list archive at Nabble.com.
Fast question about replication between solr version
I cant't find anywhere inofrmation about replication between solr version. Is it possible replicate between example solr 4.2 and 4.10 without any problems? What i need is to upgrade version and don't wona re-index all from beginning. -- View this message in context: http://lucene.472066.n3.nabble.com/Fast-question-about-replication-between-solr-version-tp4159575.html Sent from the Solr - User mailing list archive at Nabble.com.
AW: Solr(j) API for manipulating the schema(.xml)?
as our framework so far only knows a few field types dynamic fields may be the way to go... And if there are new fieldtypes the new schema can be distributed through ZooKeeper -Ursprüngliche Nachricht- Von: Erick Erickson [mailto:erickerick...@gmail.com] Gesendet: Mittwoch, 17. September 2014 19:56 An: solr-user@lucene.apache.org Betreff: Re: Solr(j) API for manipulating the schema(.xml)? Right, you can create new cores over the rest api. As far as changing the schema, there's no good way to do that that I know of programmatically. In the SolrCloud world, you can upload the schema to ZooKeeper and have it automatically distributed to all the nodes though. Best, Erick On Wed, Sep 17, 2014 at 2:28 AM, Clemens Wyss DEV clemens...@mysign.ch wrote: Is there an API to manipulate/consolidate the schema(.xml) of a Solr-core? Through SolrJ? Context: We already have a generic indexing/searching framework (based on lucene) where any component can act as a so called IndexDataPorvider. This provider delivers the field-types and also the entities to be (converted into documents and then) indexed. Each of these IndexProviders has ist own lucene index. So we kind of have the information for the Solr schema.xml. Hope the intention is clear. And yes the manipulation of the schema.xml is basically only needed when the field types change. Thats why I am looking for a way to consolidate the schema.xml (upon boot, initialization oft he IndexDataProviders ...). In 99,999% it won't change, But I'd like to keep the possibility of an IndexDataProvider to hand in its schema. Also, again driven by the dynamic nature of our framework, can I easily create new cores over Sorj or the Solr-REST API ?
Increase maxBooleanClauses limit at run time
Hi I am using solr 4.9. I have set maxBooleanClauses to 2000 in solrconfig.xml. But my application need to change this limit at run time depending upon the query. So can I set this limit at run time for e.g. in SolrParam or in other ways. -- View this message in context: http://lucene.472066.n3.nabble.com/Increase-maxBooleanClauses-limit-at-run-time-tp4159578.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Solr Suggestion not working in solr PLZ HELP
Hello Thanks for response. :) I am new in solr it better can you tell me steps of configration I need to do to reslove this issue. Appericate your help. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Suggestion-not-working-in-solr-PLZ-HELP-tp4159351p4159582.html Sent from the Solr - User mailing list archive at Nabble.com.
How to summarize a String Field ?
Hi One of my filed called AMOUNT is String,and I want to calculate the sum of the this filed. I have try it with the stats component,it only give out the stats information without sum item just as following: lst name=AMOUNT str name=min/str str name=max5000/str long name=count24230/long long name=missing26362/long lst name=facets/ /lst Is there any ways to achieve this object? Regards
Re: Implementing custom analyzer for multi-language stemming
Hi, The author of Solr in Action has produced something similar to what you want. I even has used it for one of my projects where I needed to automatically analyze languages. Here is the link to its code https://github.com/treygrainger/solr-in-action/tree/master/src/main/java/sia/ch14 https://github.com/treygrainger/solr-in-action/tree/master/src/main/java/sia/ch14 . Nevertheless, you need to pay attention that not all languages are supported by Lucene or Solr. Therefore, some of the languages detected by Google API will not have their responding chain analysis. You need to develop that. In another project, I am following the same approach to develop an AutoAnalyzer for Lucene without using Solr. So, let me know if you want directions in how to do it. Regards Ameer -- View this message in context: http://lucene.472066.n3.nabble.com/Implementing-custom-analyzer-for-multi-language-stemming-tp4150156p4159588.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Implementing custom analyzer for multi-language stemming
Is there a way to set attribute in tokenizer to document to search by word and this attribute? -- View this message in context: http://lucene.472066.n3.nabble.com/Implementing-custom-analyzer-for-multi-language-stemming-tp4150156p4159594.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Error - Parent /overseer/queue missing for /overseer/queue/
Hi, I delete data folder and issue is resolved. It seems some of the files in data folder got corrupt. Naresh -- View this message in context: http://lucene.472066.n3.nabble.com/Error-Parent-overseer-queue-missing-for-overseer-queue-tp4158208p4159601.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Ping handler during initial wamup
So, is it possible to configure a ping handler to return quickly with non-OK status if a search handler is not yet available? This would allow the load balancer to quickly fail over to another server. I couldn't find anything like this in the docs, but I'm still hopeful. I'm aware of the possibility of using a health state file, but I'd rather have a way of doing this automatically. If it's not horribly messy to implement, returning a non-OK status immediately when there is no available searcher seems like a good idea. Please file an improvement issue in Jira. Thanks, I've filed https://issues.apache.org/jira/browse/SOLR-6532. --Ere
Re[2]: Implementing custom analyzer for multi-language stemming
Thu, 18 Sep 2014 01:34:29 -0700 (PDT) от roman-v1 [via Lucene] ml-node+s472066n415959...@n3.nabble.com: Is there a way to set attribute in tokenizer to document to search by word and this attribute? -- If you reply to this email, your message will be added to the discussion below: http://lucene.472066.n3.nabble.com/Implementing-custom-analyzer-for-multi-language-stemming-tp4150156p4159594.html To unsubscribe from Implementing custom analyzer for multi-language stemming, click here . NAML -- View this message in context: http://lucene.472066.n3.nabble.com/Implementing-custom-analyzer-for-multi-language-stemming-tp4150156p4159617.html Sent from the Solr - User mailing list archive at Nabble.com.
Large Transaction Logs Out of memory
Hello, i have encountered this problem where one of the transaction logs size reached about 60GB in the solr log i have seen outOfMemeoryError during delta import. autoCommit is configured as : maxDocs2/maxDocs maxTime3/maxTime openSearcherfalse/openSearcher * i think a 15min time for hard commit might be better buffers are configured as: ramBufferSizeMB128/ramBufferSizeMB maxBufferedDocs2/maxBufferedDocs solr is used for indexing a small amount of documents, few thousands at most. jvm is running with -XX:MaxPermSize=256m -Xms512m -Xmx2048m the delta import is scheduled to run every minute. - what is the cause of the other? the outOfMemory causes the tlog to get bigger (cause the autoCommit does not happen) or the tlog size causes the outOfMemorty error? another weird thing im seeing is that every minute in the solr log im seeing the exact same lines 13 Sep 2014 00:04:50,845 [Thread-119233] INFO DocBuilder - Completed ModifiedRowKey for Entity: document rows obtained : 6203 13 Sep 2014 00:04:50,845 [Thread-119233] INFO DocBuilder - Completed DeletedRowKey for Entity: document rows obtained : 308 13 Sep 2014 00:39:43,618 [Thread-119303] INFO DocBuilder - Completed ModifiedRowKey for Entity: permission_set rows obtained : 12699 i think this shows that the last_index_time property is not being updated (maybe due to outOfMem error) since the same data is being retrieved over and over. Thanks for the help -- View this message in context: http://lucene.472066.n3.nabble.com/Large-Transaction-Logs-Out-of-memory-tp4159636.html Sent from the Solr - User mailing list archive at Nabble.com.
join on same field in same core
hi All, I am solr schema like id | task 1|t1 1|t2 2|t2 2|t3 I would like to get all the id's where task t2 t1 are associated with same id. Is there anyway we can do inner query or self join to get results? -- View this message in context: http://lucene.472066.n3.nabble.com/join-on-same-field-in-same-core-tp4159657.html Sent from the Solr - User mailing list archive at Nabble.com.
slorj - httpclient 4, but we already have httpclient 3 in use
I doing initial steps with solrj which is based on httpclient 4. Unfortunately parts of our framework are based on httpclient 3. So when I instantiate an HttpSolrServer I run into: java.lang.VerifyError: Cannot inherit from final class ... at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157) at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:447) at org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(HttpClientUtil.java:255) ... Can these be run side-by-side at all?
Solr optimistic concurrency update: version conflict
Hi, could you please help me to understand why I have such results for optimistic concurrency update. Say, I have the following document: In solrconfig.xml: In schema.xml: Because I use SignatureUpdate, I can't use partial update requests containing signature field (known Solr limitation). So I try to update the documents with optimistic concurrency update. According to the document Optimistic Concurrency http://heliosearch.org/solr/optimistic-concurrency/ if specify document version on an update, then the presence of the _version_ field instructs Solr to accept the update only if the document version matches exactly. After the update has successfully completed, the document will have a new _version_ that is guaranteed to be higher than the last. If I send the update request (the /phrase/ field is changed) as follows: I get the conflict error: Why such results? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-optimistic-concurrency-update-version-conflict-tp4159663.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to exclude a mimetype in tika?
eShard wrote Good afternoon, I'm using solr 4.0 Final I need movies hidden in zip files that need to be excluded from the index. I can't filter movies on the crawler because then I would have to exclude all zip files. I was told I can have tika skip the movies. the details are escaping me at this point. How do I exclude a file in the tika configuration? I assume it's something I add in the update/extract handler but I'm not sure. Thanks, I am having the same issue. I need to exlcude some mime types from the zip files and using SOLR 4.8. Did you ever get an answer to this? THanks. -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-exclude-a-mimetype-in-tika-tp4127168p4159676.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: slorj - httpclient 4, but we already have httpclient 3 in use
Hi Clemens, I think you need to upgrade you framework * AFAIK is httpclient 3 4 uses the same package names - which is slightly unfortunate * assuming that they are using the same package name it is non-deterministic which httpclient library is loaded - might work on your local box but not on the production server or might change to a change in the project Cheers, Siegfried Goeschl On 18.09.14 15:08, Clemens Wyss DEV wrote: I doing initial steps with solrj which is based on httpclient 4. Unfortunately parts of our framework are based on httpclient 3. So when I instantiate an HttpSolrServer I run into: java.lang.VerifyError: Cannot inherit from final class ... at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157) at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:447) at org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(HttpClientUtil.java:255) ... Can these be run side-by-side at all?
Re: Increase maxBooleanClauses limit at run time
On 9/18/2014 1:28 AM, Vishnu Mishra wrote: I am using solr 4.9. I have set maxBooleanClauses to 2000 in solrconfig.xml. But my application need to change this limit at run time depending upon the query. So can I set this limit at run time for e.g. in SolrParam or in other ways. This is a global Lucene setting. If you change the setting in a core's config and then reload that core, it would *probably* change it globally for the entire application. I'm not 100% positive about that, though. Your best bet is to try it. You would want to change it in all your configs, just to be sure you didn't lose the setting on a full solr restart. Some important people would really like to eliminate that limitation completely, but when I raised an issue to do so, I met with resistance from pure Lucene folks and I haven't made another attempt. Thanks, Shawn
Re: Fast question about replication between solr version
On 9/18/2014 1:15 AM, mizayah wrote: I cant't find anywhere inofrmation about replication between solr version. Is it possible replicate between example solr 4.2 and 4.10 without any problems? What i need is to upgrade version and don't wona re-index all from beginning. Solr 4.10 should be able to replicate from Solr 4.2, and read the index format. If it doesn't work, it's considered a bug. A full reindex is *strongly* recommended with any upgrade. Thanks, Shawn
Explanation Solr Synonym file format
Can someone give me a good explanation on how the Synonym file format works in Solr? Cause I really don't get it anymore. If I have the following synonym rule: dog = hound and I search for dog OR hound I receive the following rules: dog hound If I have the following synonym rule: hound = dog and I search for hound OR dog I receive the following rules: dog hound So weird, I dont get it what = exactly means. I read the solr docs but I'm completely confused right now, I'm trying different things for a couple of hours now and I cannot explain any of the results. What I would like to accomplish is when a user searches for dog it receives hound AND dog. If it searches for hound it just receives hound. This works: hound = dog, hound Now I receive only Hound when I search for Hound and if I search for Dog I receive Dog AND Hound -- View this message in context: http://lucene.472066.n3.nabble.com/Explanation-Solr-Synonym-file-format-tp4159710.html Sent from the Solr - User mailing list archive at Nabble.com.
AW: slorj - httpclient 4, but we already have httpclient 3 in use
I guess you are right ;) -Ursprüngliche Nachricht- Von: Siegfried Goeschl [mailto:sgoes...@gmx.at] Gesendet: Donnerstag, 18. September 2014 16:38 An: solr-user@lucene.apache.org Betreff: Re: slorj - httpclient 4, but we already have httpclient 3 in use Hi Clemens, I think you need to upgrade you framework * AFAIK is httpclient 3 4 uses the same package names - which is slightly unfortunate * assuming that they are using the same package name it is non-deterministic which httpclient library is loaded - might work on your local box but not on the production server or might change to a change in the project Cheers, Siegfried Goeschl On 18.09.14 15:08, Clemens Wyss DEV wrote: I doing initial steps with solrj which is based on httpclient 4. Unfortunately parts of our framework are based on httpclient 3. So when I instantiate an HttpSolrServer I run into: java.lang.VerifyError: Cannot inherit from final class ... at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157) at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:447) at org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(Ht tpClientUtil.java:255) ... Can these be run side-by-side at all?
Re: Large Transaction Logs Out of memory
On 9/18/2014 5:36 AM, or gerson wrote: what is the cause of the other? the outOfMemory causes the tlog to get bigger (cause the autoCommit does not happen) or the tlog size causes the outOfMemorty error? Program behavior (any Java program, not just Solr) becomes completely undefined when you get out of memory errors. It's exceptionally difficult to write programs that have defined behavior in an OOM situation. Not impossible, but *very* hard, so I'm reasonably sure that no attempt has been made in the Lucene/Solr codebase. To eliminate the OOM, you need to either make your Java heap larger, or find ways to reduce Solr's heap requirements. http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap Thanks, Shawn
AND operator in multi valued fields
Dear all, let's say you have two multivalued fields with two different complex analyzers in a quite complex schema. I would like to match specific combinations of values in these fields. For instance : Field1 : Value1, Value2 Field2 : Value3, Value4 I would like to match this document with a query like this one : +Field1:Value1 +Field2:Value3 But not with this one : +Field1:Value1 +Field2:Value4 I tried to check the PayloadNearQuery class but this class cannot use two different fields (due to the SpanNearQuery inheritance). Is there an easy way to do that ? Ludovic. - Jouve France. -- View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to summarize a String Field ?
You cannot do this as far as I know, it must be a numeric field (float/int/tint/tfloat whatever). Best Erick On Thu, Sep 18, 2014 at 12:46 AM, YouPeng Yang yypvsxf19870...@gmail.com wrote: Hi One of my filed called AMOUNT is String,and I want to calculate the sum of the this filed. I have try it with the stats component,it only give out the stats information without sum item just as following: lst name=AMOUNT str name=min/str str name=max5000/str long name=count24230/long long name=missing26362/long lst name=facets/ /lst Is there any ways to achieve this object? Regards
Re: Increase maxBooleanClauses limit at run time
Just ran in to this. Since it's global, the last core loaded rules... and you don't necessarily know which core that will be. So change this in _all_ your solrconfig.xml files. There's no external API that I know of to set it higher on a running instance. Best, Erick On Thu, Sep 18, 2014 at 8:00 AM, Shawn Heisey s...@elyograg.org wrote: On 9/18/2014 1:28 AM, Vishnu Mishra wrote: I am using solr 4.9. I have set maxBooleanClauses to 2000 in solrconfig.xml. But my application need to change this limit at run time depending upon the query. So can I set this limit at run time for e.g. in SolrParam or in other ways. This is a global Lucene setting. If you change the setting in a core's config and then reload that core, it would *probably* change it globally for the entire application. I'm not 100% positive about that, though. Your best bet is to try it. You would want to change it in all your configs, just to be sure you didn't lose the setting on a full solr restart. Some important people would really like to eliminate that limitation completely, but when I raised an issue to do so, I met with resistance from pure Lucene folks and I haven't made another attempt. Thanks, Shawn
Re: AND operator in multi valued fields
Both queries seem valid. The values are there and you asking to match them. They both should match. Can you explain how query 2 is actually different from query 1? Are you saying you want to match 1st value with 1st value (like positional constraints?). Regards, Alex. Personal: http://www.outerthoughts.com/ and @arafalov Solr resources and newsletter: http://www.solr-start.com/ and @solrstart Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 On 18 September 2014 11:24, lboutros boutr...@gmail.com wrote: Dear all, let's say you have two multivalued fields with two different complex analyzers in a quite complex schema. I would like to match specific combinations of values in these fields. For instance : Field1 : Value1, Value2 Field2 : Value3, Value4 I would like to match this document with a query like this one : +Field1:Value1 +Field2:Value3 But not with this one : +Field1:Value1 +Field2:Value4 I tried to check the PayloadNearQuery class but this class cannot use two different fields (due to the SpanNearQuery inheritance). Is there an easy way to do that ? Ludovic. - Jouve France. -- View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: AND operator in multi valued fields
Alexandre Rafalovitch wrote Are you saying you want to match 1st value with 1st value (like positional constraints?). That's exactly what I would like to do. :) - Jouve France. -- View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159728.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: AW: slorj - httpclient 4, but we already have httpclient 3 in use
Hi Clemens, If you are going thru the effort of migrating from SolrJ 3 to 4 and HTTP client 3 to 4 make sure you do it using HTTP client 4.3.x (Latest is 4.3.5) since there are deprecations and stuff from 3.x to 4.0.x, to 4.1.x, to ..., to 4.3.x It will be painful but it is better do it one time and not later needed to do it again. I was on a similar situation (well my company) and I had to suffer such migration (not my company but myself since I'm the one that keeps all those things up to date) Best regards, Guido. On 18/09/14 16:14, Clemens Wyss DEV wrote: I guess you are right ;) -Ursprüngliche Nachricht- Von: Siegfried Goeschl [mailto:sgoes...@gmx.at] Gesendet: Donnerstag, 18. September 2014 16:38 An: solr-user@lucene.apache.org Betreff: Re: slorj - httpclient 4, but we already have httpclient 3 in use Hi Clemens, I think you need to upgrade you framework * AFAIK is httpclient 3 4 uses the same package names - which is slightly unfortunate * assuming that they are using the same package name it is non-deterministic which httpclient library is loaded - might work on your local box but not on the production server or might change to a change in the project Cheers, Siegfried Goeschl On 18.09.14 15:08, Clemens Wyss DEV wrote: I doing initial steps with solrj which is based on httpclient 4. Unfortunately parts of our framework are based on httpclient 3. So when I instantiate an HttpSolrServer I run into: java.lang.VerifyError: Cannot inherit from final class ... at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157) at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:447) at org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(Ht tpClientUtil.java:255) ... Can these be run side-by-side at all?
Re: AND operator in multi valued fields
Do you know the position when you are doing the search? Or just that they need to be parallel within their tokenized groups? Regards, Alex. P.s. It may help if you explain a business level issue here. There might be a completely different approach to that as well. Personal: http://www.outerthoughts.com/ and @arafalov Solr resources and newsletter: http://www.solr-start.com/ and @solrstart Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 On 18 September 2014 11:42, lboutros boutr...@gmail.com wrote: Alexandre Rafalovitch wrote Are you saying you want to match 1st value with 1st value (like positional constraints?). That's exactly what I would like to do. :) - Jouve France. -- View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159728.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: AW: slorj - httpclient 4, but we already have httpclient 3 in use
AFAIK even the different minor versions are source/binary compatible so you might need to tinker with the right version to get your server running Cheers, Siegfried Goeschl On 18.09.14 17:45, Guido Medina wrote: Hi Clemens, If you are going thru the effort of migrating from SolrJ 3 to 4 and HTTP client 3 to 4 make sure you do it using HTTP client 4.3.x (Latest is 4.3.5) since there are deprecations and stuff from 3.x to 4.0.x, to 4.1.x, to ..., to 4.3.x It will be painful but it is better do it one time and not later needed to do it again. I was on a similar situation (well my company) and I had to suffer such migration (not my company but myself since I'm the one that keeps all those things up to date) Best regards, Guido. On 18/09/14 16:14, Clemens Wyss DEV wrote: I guess you are right ;) -Ursprüngliche Nachricht- Von: Siegfried Goeschl [mailto:sgoes...@gmx.at] Gesendet: Donnerstag, 18. September 2014 16:38 An: solr-user@lucene.apache.org Betreff: Re: slorj - httpclient 4, but we already have httpclient 3 in use Hi Clemens, I think you need to upgrade you framework * AFAIK is httpclient 3 4 uses the same package names - which is slightly unfortunate * assuming that they are using the same package name it is non-deterministic which httpclient library is loaded - might work on your local box but not on the production server or might change to a change in the project Cheers, Siegfried Goeschl On 18.09.14 15:08, Clemens Wyss DEV wrote: I doing initial steps with solrj which is based on httpclient 4. Unfortunately parts of our framework are based on httpclient 3. So when I instantiate an HttpSolrServer I run into: java.lang.VerifyError: Cannot inherit from final class ... at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157) at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:447) at org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(Ht tpClientUtil.java:255) ... Can these be run side-by-side at all?
Re: Access solr cloud via ssh tunnel? (Workaround/Jsch)
On 09/16/2014 04:03 PM, Doug Balog wrote:
Not sure if this will work, but try to use ssh to setup a SOCKS proxy via
the -D command option.
Then use the socksProxyHost and socksProxyPort via the java command line
(ie java -DsocksProxyHost=localhost) or
System.setProperty(socksProxyHost,localhost) from your code. Make sure
to specify both the host and the port.
See
http://docs.oracle.com/javase/7/docs/api/java/net/doc-files/net-properties.html
Unfortunately Jsch does not seem to provide the -D socks5 over
ssh option.
- In case this may help others -
Because the production system will have direct access to the cluster and
this is being setup for accessing the production cloud from our office
we instead did the following:
SolrTunnels t = new SolrTunnels();
t.connect();
LBHttpSolrServer server = new LBHttpSolrServer();
server.setParser(new BinaryResponseParser());
server.setAliveCheckInterval(500);
for (SolrHost solr: t.getEndpoints()) {
server.addSolrServer(http://127.0.0.1:+solr.forward+/solr/test;);
}
WHERE:
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
import com.newsrx.util.NrxLog;
public class SolrTunnels {
static final private String sshUser = autossh;
static final private String sshPass = LETMEIN;
static private String sshHost = public.solr.gateway.host.com;
static final private int sshPort = 22;
static volatile private JSch jsch = new JSch();
static private Session solrSSH = null;
public static class SolrHost {
public String host;
public int port;
public int forward;
public SolrHost() {}
public SolrHost(String host, int port) {
super();
this.host = host;
this.port = port;
this.forward = -1;
}
}
final static private ListSolrHost nodes;
static {
nodes=new ArrayList();
nodes.add(new SolrHost(solr1.private, 8983));
nodes.add(new SolrHost(solr2.private, 8983));
}
public SolrTunnels() {
}
public void connect() {
if (solrSSH != null) {
if (solrSSH.isConnected()) {
return;
}
}
JSch.setConfig(StrictHostKeyChecking, no);
JSch.setConfig(Compression, none);//compressionsometimes
causes ssh transport breakage
int maxTries = 100;
do {
try {
if (solrSSH !=null) {
solrSSH.disconnect();
}
solrSSH = jsch.getSession(sshUser, sshHost, sshPort);
solrSSH.setPassword(sshPass);
solrSSH.connect(1000);
IteratorSolrHost isolr = nodes.iterator();
while (isolr.hasNext()) {
SolrHost solr = isolr.next();
solr.forward=solrSSH.setPortForwardingL(0, solr.host,solr.port);
Console.log(http://127.0.0.1:+solr.forward+/solr;);
}
} catch (JSchException e) {
e.printStackTrace();
try {
Console.log(Sleeping 100 ms);
Thread.sleep(100);
} catch (InterruptedException e1) {
}
}
} while (maxTries-- 0!solrSSH.isConnected());
}
public CollectionSolrHost getEndpoints() {
ListSolrHost list = new ArrayList();
IteratorSolrHost isolr = nodes.iterator();
while (isolr.hasNext()) {
SolrHost solr = isolr.next();
if (solr.forward0) {
list.add(solr);
}
}
return list;
}
public void disconnect() {
if (solrSSH !=null) {
IteratorSolrHost isolr = nodes.iterator();
while (isolr.hasNext()) {
SolrHost solr = isolr.next();
try {
solrSSH.delPortForwardingL(solr.forward);
} catch (JSchException e) {
}
}
solrSSH.disconnect();
}
}
}
Re: AW: slorj - httpclient 4, but we already have httpclient 3 in use
SolrJ client after 4.8 I think requires HTTP client 4.3.x so why not just start there as base version? Guido. On 18/09/14 16:49, Siegfried Goeschl wrote: AFAIK even the different minor versions are source/binary compatible so you might need to tinker with the right version to get your server running Cheers, Siegfried Goeschl On 18.09.14 17:45, Guido Medina wrote: Hi Clemens, If you are going thru the effort of migrating from SolrJ 3 to 4 and HTTP client 3 to 4 make sure you do it using HTTP client 4.3.x (Latest is 4.3.5) since there are deprecations and stuff from 3.x to 4.0.x, to 4.1.x, to ..., to 4.3.x It will be painful but it is better do it one time and not later needed to do it again. I was on a similar situation (well my company) and I had to suffer such migration (not my company but myself since I'm the one that keeps all those things up to date) Best regards, Guido. On 18/09/14 16:14, Clemens Wyss DEV wrote: I guess you are right ;) -Ursprüngliche Nachricht- Von: Siegfried Goeschl [mailto:sgoes...@gmx.at] Gesendet: Donnerstag, 18. September 2014 16:38 An: solr-user@lucene.apache.org Betreff: Re: slorj - httpclient 4, but we already have httpclient 3 in use Hi Clemens, I think you need to upgrade you framework * AFAIK is httpclient 3 4 uses the same package names - which is slightly unfortunate * assuming that they are using the same package name it is non-deterministic which httpclient library is loaded - might work on your local box but not on the production server or might change to a change in the project Cheers, Siegfried Goeschl On 18.09.14 15:08, Clemens Wyss DEV wrote: I doing initial steps with solrj which is based on httpclient 4. Unfortunately parts of our framework are based on httpclient 3. So when I instantiate an HttpSolrServer I run into: java.lang.VerifyError: Cannot inherit from final class ... at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(DefaultHttpClient.java:157) at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHttpClient.java:447) at org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects(Ht tpClientUtil.java:255) ... Can these be run side-by-side at all?
Re: Access solr cloud via ssh tunnel? (Workaround/Jsch)
This would make for a nice blog post!
On Thu, Sep 18, 2014 at 9:38 PM, Michael Joyner mich...@newsrx.com wrote:
On 09/16/2014 04:03 PM, Doug Balog wrote:
Not sure if this will work, but try to use ssh to setup a SOCKS proxy via
the -D command option.
Then use the socksProxyHost and socksProxyPort via the java command line
(ie java -DsocksProxyHost=localhost) or
System.setProperty(socksProxyHost,localhost) from your code. Make
sure
to specify both the host and the port.
See
http://docs.oracle.com/javase/7/docs/api/java/net/doc-files/
net-properties.html
Unfortunately Jsch does not seem to provide the -D socks5 over ssh
option.
- In case this may help others -
Because the production system will have direct access to the cluster and
this is being setup for accessing the production cloud from our office we
instead did the following:
SolrTunnels t = new SolrTunnels();
t.connect();
LBHttpSolrServer server = new LBHttpSolrServer();
server.setParser(new BinaryResponseParser());
server.setAliveCheckInterval(500);
for (SolrHost solr: t.getEndpoints()) {
server.addSolrServer(http://127.0.0.1:+solr.forward+/solr/test;);
}
WHERE:
import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.JSchException;
import com.jcraft.jsch.Session;
import com.newsrx.util.NrxLog;
public class SolrTunnels {
static final private String sshUser = autossh;
static final private String sshPass = LETMEIN;
static private String sshHost = public.solr.gateway.host.com;
static final private int sshPort = 22;
static volatile private JSch jsch = new JSch();
static private Session solrSSH = null;
public static class SolrHost {
public String host;
public int port;
public int forward;
public SolrHost() {}
public SolrHost(String host, int port) {
super();
this.host = host;
this.port = port;
this.forward = -1;
}
}
final static private ListSolrHost nodes;
static {
nodes=new ArrayList();
nodes.add(new SolrHost(solr1.private, 8983));
nodes.add(new SolrHost(solr2.private, 8983));
}
public SolrTunnels() {
}
public void connect() {
if (solrSSH != null) {
if (solrSSH.isConnected()) {
return;
}
}
JSch.setConfig(StrictHostKeyChecking, no);
JSch.setConfig(Compression, none);//compressionsometimes
causes ssh transport breakage
int maxTries = 100;
do {
try {
if (solrSSH !=null) {
solrSSH.disconnect();
}
solrSSH = jsch.getSession(sshUser, sshHost, sshPort);
solrSSH.setPassword(sshPass);
solrSSH.connect(1000);
IteratorSolrHost isolr = nodes.iterator();
while (isolr.hasNext()) {
SolrHost solr = isolr.next();
solr.forward=solrSSH.setPortForwardingL(0, solr.host,solr.port);
Console.log(http://127.0.0.1:+solr.forward+/solr;);
}
} catch (JSchException e) {
e.printStackTrace();
try {
Console.log(Sleeping 100 ms);
Thread.sleep(100);
} catch (InterruptedException e1) {
}
}
} while (maxTries-- 0!solrSSH.isConnected());
}
public CollectionSolrHost getEndpoints() {
ListSolrHost list = new ArrayList();
IteratorSolrHost isolr = nodes.iterator();
while (isolr.hasNext()) {
SolrHost solr = isolr.next();
if (solr.forward0) {
list.add(solr);
}
}
return list;
}
public void disconnect() {
if (solrSSH !=null) {
IteratorSolrHost isolr = nodes.iterator();
while (isolr.hasNext()) {
SolrHost solr = isolr.next();
try {
solrSSH.delPortForwardingL(solr.forward);
} catch (JSchException e) {
}
}
solrSSH.disconnect();
}
}
}
--
Regards,
Shalin Shekhar Mangar.
Re: Solr 4.10 termsIndexInterval and termsIndexDivisor not supported with default PostingsFormat?
: I think the documentation and example files for Solr 4.x need to be : updated. If someone will let me know I'll be happy to fix the example : and perhaps someone with edit rights could fix the reference guide. I think you're correct - can you open a Jira with suggested improvements for the configs? (i see you commented on the ref guide page which is helpful - but the jira issue wil also help serve sa a reminder to audit *all* the pages for refrences to these options, ie: in config snippets, etc...) : According to the JavaDocs for IndexWriterConfig, the Lucene level : implementations of these do not apply to the default PostingsFormat : implementation. : http://lucene.apache.org/core/4_10_0/core/org/apache/lucene/index/IndexWriterConfig.html#setReaderTermsIndexDivisor%28int%29 : : Despite this statement in the Lucene JavaDocs, in the : example/solrconfig.xml there is the following: Yeah ... I'm not sure what (if anything?) we should say about these in the example configs -- the *setting* is valid and supported by IndexWriterConfig no matter what posting format you use, so it's not an error to configure this, but it can be ignored in many cases. : Can someone please confirm that these two parameter settings : termIndexInterval and termsIndexDivisor, do not apply to the default : PostingsFormat for Solr 4.10? I was taking your word for it :) -Hoss http://www.lucidworks.com/
running solr in debug through eclipse
Dear Solr users, I am new to Solr dev community and trying to setup eclipse to debug a running solr server. Please suggest if anyone of you have tried doing the same. Once above is done. Also suggest the entry point in code where breakpoint can be placed. Thanks Anurag
Re: How to summarize a String Field ?
Do a copyField to a numeric field. -- Jack Krupansky -Original Message- From: Erick Erickson Sent: Thursday, September 18, 2014 11:35 AM To: solr-user@lucene.apache.org Subject: Re: How to summarize a String Field ? You cannot do this as far as I know, it must be a numeric field (float/int/tint/tfloat whatever). Best Erick On Thu, Sep 18, 2014 at 12:46 AM, YouPeng Yang yypvsxf19870...@gmail.com wrote: Hi One of my filed called AMOUNT is String,and I want to calculate the sum of the this filed. I have try it with the stats component,it only give out the stats information without sum item just as following: lst name=AMOUNT str name=min/str str name=max5000/str long name=count24230/long long name=missing26362/long lst name=facets/ /lst Is there any ways to achieve this object? Regards
Re: AND operator in multi valued fields
Thx Alex. We have main documents in the index. (more than 100 complex fields). Each document can have events attached. An event contains 4 fields with 3 different analyzers. We need more than just filtering on them (highlighting on documents and events at the same time for instance). That means that nested documents cannot be used. These events are indexed as additional multi valued fields in each documents. They are searched like any other field. The issue here is that the operator 'AND' between event fields can match false positives. We do not know the position during search. We just want to respect the event integrity in the search. So you are right, we just want them to be parallel within their tokenized groups ? The first idea was to index the event in only one field and use proximity/phrase search in order to prevent false positives. But that means that we need to index dates, ids and text in one unique field. Do you think this could be a better/easier approach ? Ludovic - Jouve France. -- View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159797.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: AND operator in multi valued fields
Well, I can think of four ways, increasingly complicated. 1) You could have both parent record with unzipped events and also child events as individual documents. Then, you do filtering based on children and highlighting based on parent documents. 2) The other way is to have a custom post filter that looks at the matches and discards the ones that have different offset (by using very large positionIncrementGap to create clear group boundaries). But I don't know whether you can access the match token offsets in the post filter, so this is more of a thought experiment. 3) You could also duplicate main field contents and be the document one per event. If most of the fields are indexed, it's ok and no real duplication. But you may need to store fields for highlighter and those are not de-duplicated internally, as far as I know. 4) You could create zipped pairs of values in a dedicated field and search that as near-queries. But than you do have to have the same analyzer for all members. Sounds like this may not be an option for you. Regards, Alex. Personal: http://www.outerthoughts.com/ and @arafalov Solr resources and newsletter: http://www.solr-start.com/ and @solrstart Solr popularizers community: https://www.linkedin.com/groups?gid=6713853 On 18 September 2014 16:41, lboutros boutr...@gmail.com wrote: Thx Alex. We have main documents in the index. (more than 100 complex fields). Each document can have events attached. An event contains 4 fields with 3 different analyzers. We need more than just filtering on them (highlighting on documents and events at the same time for instance). That means that nested documents cannot be used. These events are indexed as additional multi valued fields in each documents. They are searched like any other field. The issue here is that the operator 'AND' between event fields can match false positives. We do not know the position during search. We just want to respect the event integrity in the search. So you are right, we just want them to be parallel within their tokenized groups ? The first idea was to index the event in only one field and use proximity/phrase search in order to prevent false positives. But that means that we need to index dates, ids and text in one unique field. Do you think this could be a better/easier approach ? Ludovic - Jouve France. -- View this message in context: http://lucene.472066.n3.nabble.com/AND-operator-in-multi-valued-fields-tp4159715p4159797.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Fatal full GC
mark -- View this message in context: http://lucene.472066.n3.nabble.com/Fatal-full-GC-tp4158429p4159827.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud deleted all existing indexes after update query
should be master node. replia maybe appear this. I don't know why? -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-deleted-all-existing-indexes-after-update-query-tp4159566p4159828.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr/lucene 4.10 out of memory issues
mark. -- View this message in context: http://lucene.472066.n3.nabble.com/solr-lucene-4-10-out-of-memory-issues-tp4158262p4159829.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: What's the actual story with new morphline and hadoop contribs?
I think it is useful to dirstribute index and then merge to solr. Cloudear use it often. But reference is too less to understand. -- View this message in context: http://lucene.472066.n3.nabble.com/What-s-the-actual-story-with-new-morphline-and-hadoop-contribs-tp4130999p4159830.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: running solr in debug through eclipse
There are two approaches that work: 1 (preferred IMO) is to debug through the Junit test cases. It's far easier than remote debugging usually, with quicker turnaround times. 2 Set up remote debugging, see: http://wiki.apache.org/solr/HowToConfigureEclipse which is linked from the how to contribute page here: http://wiki.apache.org/solr/HowToContribute#Eclipse As to where to set your first breakpoint, I have no clue. You haven't told us a thing about what you're trying to accomplish, so somewhere in the code is about the best I can say ;). Best Erick On Thu, Sep 18, 2014 at 11:50 AM, Anurag Sharma anura...@gmail.com wrote: Dear Solr users, I am new to Solr dev community and trying to setup eclipse to debug a running solr server. Please suggest if anyone of you have tried doing the same. Once above is done. Also suggest the entry point in code where breakpoint can be placed. Thanks Anurag
AW: AW: slorj - httpclient 4, but we already have httpclient 3 in use
I'd like to mention, that substituting the httpcore.jar with the latest (4.3) sufficed... -Ursprüngliche Nachricht- Von: Guido Medina [mailto:guido.med...@temetra.com] Gesendet: Donnerstag, 18. September 2014 18:20 An: solr-user@lucene.apache.org Betreff: Re: AW: slorj - httpclient 4, but we already have httpclient 3 in use SolrJ client after 4.8 I think requires HTTP client 4.3.x so why not just start there as base version? Guido. On 18/09/14 16:49, Siegfried Goeschl wrote: AFAIK even the different minor versions are source/binary compatible so you might need to tinker with the right version to get your server running Cheers, Siegfried Goeschl On 18.09.14 17:45, Guido Medina wrote: Hi Clemens, If you are going thru the effort of migrating from SolrJ 3 to 4 and HTTP client 3 to 4 make sure you do it using HTTP client 4.3.x (Latest is 4.3.5) since there are deprecations and stuff from 3.x to 4.0.x, to 4.1.x, to ..., to 4.3.x It will be painful but it is better do it one time and not later needed to do it again. I was on a similar situation (well my company) and I had to suffer such migration (not my company but myself since I'm the one that keeps all those things up to date) Best regards, Guido. On 18/09/14 16:14, Clemens Wyss DEV wrote: I guess you are right ;) -Ursprüngliche Nachricht- Von: Siegfried Goeschl [mailto:sgoes...@gmx.at] Gesendet: Donnerstag, 18. September 2014 16:38 An: solr-user@lucene.apache.org Betreff: Re: slorj - httpclient 4, but we already have httpclient 3 in use Hi Clemens, I think you need to upgrade you framework * AFAIK is httpclient 3 4 uses the same package names - which is slightly unfortunate * assuming that they are using the same package name it is non-deterministic which httpclient library is loaded - might work on your local box but not on the production server or might change to a change in the project Cheers, Siegfried Goeschl On 18.09.14 15:08, Clemens Wyss DEV wrote: I doing initial steps with solrj which is based on httpclient 4. Unfortunately parts of our framework are based on httpclient 3. So when I instantiate an HttpSolrServer I run into: java.lang.VerifyError: Cannot inherit from final class ... at org.apache.http.impl.client.DefaultHttpClient.createHttpParams(Defa ultHttpClient.java:157) at org.apache.http.impl.client.AbstractHttpClient.getParams(AbstractHt tpClient.java:447) at org.apache.solr.client.solrj.impl.HttpClientUtil.setFollowRedirects (Ht tpClientUtil.java:255) ... Can these be run side-by-side at all?
Re: running solr in debug through eclipse
Hi Erick, Thanks a lot for your response. I am trying to fix the issue: https://issues.apache.org/jira/browse/SOLR-6307. I guess this require change in the 'update' component. Am not able to locate the code for 'update' component nor the testcases. To understand the code base I think I may need to go through both the #1 #2 approaches. Regards, Anurag On Fri, Sep 19, 2014 at 8:38 AM, Erick Erickson erickerick...@gmail.com wrote: There are two approaches that work: 1 (preferred IMO) is to debug through the Junit test cases. It's far easier than remote debugging usually, with quicker turnaround times. 2 Set up remote debugging, see: http://wiki.apache.org/solr/HowToConfigureEclipse which is linked from the how to contribute page here: http://wiki.apache.org/solr/HowToContribute#Eclipse As to where to set your first breakpoint, I have no clue. You haven't told us a thing about what you're trying to accomplish, so somewhere in the code is about the best I can say ;). Best Erick On Thu, Sep 18, 2014 at 11:50 AM, Anurag Sharma anura...@gmail.com wrote: Dear Solr users, I am new to Solr dev community and trying to setup eclipse to debug a running solr server. Please suggest if anyone of you have tried doing the same. Once above is done. Also suggest the entry point in code where breakpoint can be placed. Thanks Anurag
