Re: Number of clustering labels to show
The scope in there is to try to make clustering lighter and more related to the query. The summary produced is a fragment that is surrounding the query terms in the document content. Actually this is arguably a way to improve the quality of clusters, but for sure it makes the clustering operation lighter, as the content used to produce the clusters is much smaller than the full content. We can discuss of course if the window of text surrounding queries match is really helpful to cluster the documents in a more precise way. That is not an easy research topic, and for sure it depends strictly on the use cases. For this reason a user should decide if going with the summary ( lighter) approach or the more comprehensive , full content approach. Cheers 2015-06-02 3:21 GMT+01:00 Zheng Lin Edwin Yeo edwinye...@gmail.com: Thank you so much Alessandro. But i do not find any difference with the quality of the clustering results when I change the hl.fragszie to a even though I've set my carrot.produceSummary to true. Regards, Edwin On 1 June 2015 at 17:31, Alessandro Benedetti benedetti.ale...@gmail.com wrote: Only to clarify the initial mail, The carrot.fragSize has nothing to do with the number of clusters produced. When you select to work with field summary ( you will work only on snippets from the original content, snippets produced by the highlight of the query in the content), the fragSize will specify the size of these fragments. From Carrot documentation : carrot.produceSummary When true, the carrot.snippet https://wiki.apache.org/solr/ClusteringComponent#carrot.snippet field (if no snippet field, then the carrot.title https://wiki.apache.org/solr/ClusteringComponent#carrot.title field) will be highlighted and the highlighted text will be used for clustering. Highlighting is recommended when the snippet field contains a lot of content. Highlighting can also increase the quality of clustering because the clustered content will get an additional query-specific context. carrot.fragSize The frag size to use for highlighting. Meaningful only when carrot.produceSummary https://wiki.apache.org/solr/ClusteringComponent#carrot.produceSummary is true. If not specified, the default highlighting fragsize (hl.fragsize) will be used. If that isn't specified, then 100. Cheers 2015-06-01 2:00 GMT+01:00 Zheng Lin Edwin Yeo edwinye...@gmail.com: Thank you Stanislaw for the links. Will read them up to better understand how the algorithm works. Regards, Edwin On 29 May 2015 at 17:22, Stanislaw Osinski stanislaw.osin...@carrotsearch.com wrote: Hi, The number of clusters primarily depends on the parameters of the specific clustering algorithm. If you're using the default Lingo algorithm, the number of clusters is governed by the LingoClusteringAlgorithm.desiredClusterCountBase parameter. Take a look at the documentation ( https://cwiki.apache.org/confluence/display/solr/Result+Clustering#ResultClustering-TweakingAlgorithmSettings ) for some more details (the Tweaking at Query-Time section shows how to pass the specific parameters at request time). A complete overview of the Lingo clustering algorithm parameters is here: http://doc.carrot2.org/#section.component.lingo. Stanislaw -- Stanislaw Osinski, stanislaw.osin...@carrotsearch.com http://carrotsearch.com On Fri, May 29, 2015 at 4:29 AM, Zheng Lin Edwin Yeo edwinye...@gmail.com wrote: Hi, I'm trying to increase the number of cluster result to be shown during the search. I tried to set carrot.fragSize=20 but only 15 cluster labels is shown. Even when I tried to set carrot.fragSize=5, there's also 15 labels shown. Is this the correct way to do this? I understand that setting it to 20 might not necessary mean 20 lables will be shown, as the setting is for maximum number. But when I set this to 5, it should reduce the number of labels to 5? I'm using Solr 5.1. Regards, Edwin -- -- Benedetti Alessandro Visiting card : http://about.me/alessandro_benedetti Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry? William Blake - Songs of Experience -1794 England -- -- Benedetti Alessandro Visiting card : http://about.me/alessandro_benedetti Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry? William Blake - Songs of Experience -1794 England
Need help in velocity search UI
Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ?
Re: Derive suggestions across multiple fields
Hi Edwin, I have worked extensively recently in Suggester and the blog I feel to suggest is Erick's one. It's really detailed and good for a beginner and expert as well. [1] Apart that let's see you particular use case : 1) Do you want to be able to get also where the suggestions are coming from ? e.g. suggestion1 from field1 suggestion2 from field2 ? In this case I would try with multiple dictionaries but I am not sure Solr allows you to use them concurrently. But can be a really nice extension to develop. 2) If you don't care where the suggestions are coming from, just use a copy field, where you copy the content of the interesting fields. The suggestions will come from the fields you have copied in the copy field, without distinction. Hope this helps you Cheers [1] http://lucidworks.com/blog/solr-suggester/ 2015-06-02 4:22 GMT+01:00 Zheng Lin Edwin Yeo edwinye...@gmail.com: Hi, Does anyone knows if we can derive suggestions across multiple fields? I tried to set something like this in my field in suggest searchComponents in solrconfig.xml, but nothing is returned. It only works when I set a single field, and not multiple field. searchComponent class=solr.SpellCheckComponent name=suggest lst name=spellchecker str name=namesuggest/str str name=classnameorg.apache.solr.spelling.suggest.Suggester/str str name=lookupImplorg.apache.solr.spelling.suggest.tst.TSTLookupFactory/str str name=fieldContent, Summary/str !-- the indexed field to derive suggestions from -- float name=threshold0.005/float str name=buildOnCommittrue/str /lst /searchComponent I'm using solr 5.1. Regards, Edwin -- -- Benedetti Alessandro Visiting card : http://about.me/alessandro_benedetti Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry? William Blake - Songs of Experience -1794 England
Re: Need help in velocity search UI
Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ? -- Michał Bieńkowski
Re: Need help in velocity search UI
Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); /script - On 2 June 2015 at 14:29, Michał B. . m.bienkow...@gmail.com wrote: Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ? -- Michał Bieńkowski
Re: SolrCloud 5.1 startup looking for standalone config
I ran this command with Solr hosts s1 s2 running. http://s1:8983/solr/admin/collections?action=CREATEname=mycollectionnumShards=2collection.configName=mycollection_cloud_confcreateNodeSet=s1:8983,s2:8983 I referred to this link http://heliosearch.org/solrcloud-assigning-nodes-machines/ which looks like it is only passing the desired leaders to createNodeSet. But I'm getting this error - Cannot create collection mycollection. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 0. This allows a maximum of 0 to be created. Value of numShards is 2 and value of replicationFactor is 1. This requires 2 shards to be created (higher than the allowed number) I get the same error with createNodeSet=s1:8983,s2:8983,s3:8983,s4:8983 with all four Solr hosts running. But the service status command shows that Zookeeper sees all my running nodes Solr process 24603 running on port 8983 { solr_home:/volume/solr/data/, version:5.1.0 1672403 - timpotter - 2015-04-09 10:37:54, startTime:2015-06-02T18:00:06.665Z, uptime:0 days, 0 hours, 4 minutes, 35 seconds, memory:19.6 MB (%4) of 490.7 MB, cloud:{ ZooKeeper:zk1:2181,zk2:2181,k3:2181, liveNodes:4, collections:0}} I was expecting the absent maxShardsPerNode param to default to 1 and give me 2 leaders, 2 replicas. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5-1-startup-looking-for-standalone-config-tp4209118p4209294.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Need help in velocity search UI
Jeetendra - Forget about url_for_facet_filter and any other macros (since these were built for the techproducts /browse UI, not necessarily general purpose and not for use with multi-select facets)… just concern yourself with the actual Solr URL that you need to submit. In the example you provided you probably want to use OR instead of AND - unless you’re trying to find documents that have actually have all of those values, not just all documents that all documents that have _any_ of those values. Once you’ve got the URLs to Solr right and getting the data you want, then you can macro things as desired. You’re asking about some fiddly niche stuff with those macros that are just getting in your way. Again, don’t worry about the macros and just concern yourself with the Solr actions until you get that where you want. But yes, in general /browse will work with any arbitrary fq passed to it - it’s just a Solr request that displays the results through templates rather than as JSON, XML, or some other format. — Erik Hatcher, Senior Solutions Architect http://www.lucidworks.com http://www.lucidworks.com/ On Jun 2, 2015, at 12:58 PM, Jeetendra Gangele gangele...@gmail.com wrote: Thanks Erik for reply. I am building some kind of semantic search for one of the google Hackathon project so I need this UI for Demo only. I already have java script which is allowing user to select multiple facets and I am getting all the facets selecting by user in array in java script. Now I want to pass this to solr? by using url_for_facet_filter ? below is my UI and java script which allow use to select multiple things and capture in java scripts array, now the question is how can I make rest call using these captured data to slor. One way I can thing of is like create the URI like below and when user hot on submit call the below URI(highlights things I will pass as variable from user selected facets) http://54.242.13.110:8080/solr/collection1/browse?fq=idx_type:companiesq=fq=(*:* AND topics:*revenue* AND topics:*reports/results* AND NOT topics: *aerospace*”) Will this work? or I need to create the search page on which this data will be shown or on the existing page data will be filtered and rendered? http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#else http://facet.name/#else emmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); On 2 June 2015 at 21:56, Erik Hatcher erik.hatc...@lucidworks.com wrote: Jeetendra - First steps - implement what you want with static HTML/JavaScript so that you have the interface in place. It can get hairy with the Velocity/HTML/JavaScript/Solr stuff all intermingled*. By doing things statically (without Velocity, and even without Solr) you’ll get the UI pieces implemented and then can make it dynamic. The #url_for_facet_filter is designed for single select faceting. For multi-select faceting (within the Velocity UI) you’ll need JavaScript that can allow a user to select multiple facets before submitting (or maybe submit each one and allow refining). There is no built-in example with multi-select faceting to go by.
SolrCloud Composite ID routing problem
Hi, I have two set of data(Collections). I want to make a join from one to the other. Is that possible using SolrCloud. I also tried the following. 1. Tried to load both set of documents in a common schema(single collection) so that we can make joins. From some of the posts, I heard joins can be done only if the documents are in the same shard. So tried to co-locate them using composite-id routing. But my id field is of TrieIntField type and when I try to index using the following. curl http://localhost:8983/solr/collection1/update?commit=true -H 'Content-type:application/json' -d ' [ { id: solrcheck!3719797, company : XXX } ]' But am getting Bad request error. What is the correct way to index documents with prefix for co-location? Thanks Bharath -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-Composite-ID-routing-problem-tp4209316.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud 5.1 startup looking for standalone config
Take a closer look at the admin UIcloudtree and you'll see that the nodes have names like s1:8983_solr as does the example at the link, did you try it with that kind of labeling? Best, Erick On Tue, Jun 2, 2015 at 11:16 AM, tuxedomoon dancolem...@yahoo.com wrote: I ran this command with Solr hosts s1 s2 running. http://s1:8983/solr/admin/collections?action=CREATEname=mycollectionnumShards=2collection.configName=mycollection_cloud_confcreateNodeSet=s1:8983,s2:8983 I referred to this link http://heliosearch.org/solrcloud-assigning-nodes-machines/ which looks like it is only passing the desired leaders to createNodeSet. But I'm getting this error - Cannot create collection mycollection. Value of maxShardsPerNode is 1, and the number of nodes currently live or live and part of your createNodeSet is 0. This allows a maximum of 0 to be created. Value of numShards is 2 and value of replicationFactor is 1. This requires 2 shards to be created (higher than the allowed number) I get the same error with createNodeSet=s1:8983,s2:8983,s3:8983,s4:8983 with all four Solr hosts running. But the service status command shows that Zookeeper sees all my running nodes Solr process 24603 running on port 8983 { solr_home:/volume/solr/data/, version:5.1.0 1672403 - timpotter - 2015-04-09 10:37:54, startTime:2015-06-02T18:00:06.665Z, uptime:0 days, 0 hours, 4 minutes, 35 seconds, memory:19.6 MB (%4) of 490.7 MB, cloud:{ ZooKeeper:zk1:2181,zk2:2181,k3:2181, liveNodes:4, collections:0}} I was expecting the absent maxShardsPerNode param to default to 1 and give me 2 leaders, 2 replicas. -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5-1-startup-looking-for-standalone-config-tp4209118p4209294.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Derive suggestions across multiple fields
Thank you for your explanation. I'll not need to care where the suggestions are coming from. All the suggestions from different fields can be consolidate and display together. I've tried to put those field into a new Suggestion copy field, but no suggestion is shown when I set: str name=fieldSuggestion/str !-- the indexed field to derive suggestions from -- Is there a need to re-index the documents in order for this to work? Regards, Edwin On 2 June 2015 at 17:25, Alessandro Benedetti benedetti.ale...@gmail.com wrote: Hi Edwin, I have worked extensively recently in Suggester and the blog I feel to suggest is Erick's one. It's really detailed and good for a beginner and expert as well. [1] Apart that let's see you particular use case : 1) Do you want to be able to get also where the suggestions are coming from ? e.g. suggestion1 from field1 suggestion2 from field2 ? In this case I would try with multiple dictionaries but I am not sure Solr allows you to use them concurrently. But can be a really nice extension to develop. 2) If you don't care where the suggestions are coming from, just use a copy field, where you copy the content of the interesting fields. The suggestions will come from the fields you have copied in the copy field, without distinction. Hope this helps you Cheers [1] http://lucidworks.com/blog/solr-suggester/ 2015-06-02 4:22 GMT+01:00 Zheng Lin Edwin Yeo edwinye...@gmail.com: Hi, Does anyone knows if we can derive suggestions across multiple fields? I tried to set something like this in my field in suggest searchComponents in solrconfig.xml, but nothing is returned. It only works when I set a single field, and not multiple field. searchComponent class=solr.SpellCheckComponent name=suggest lst name=spellchecker str name=namesuggest/str str name=classnameorg.apache.solr.spelling.suggest.Suggester/str str name=lookupImplorg.apache.solr.spelling.suggest.tst.TSTLookupFactory/str str name=fieldContent, Summary/str !-- the indexed field to derive suggestions from -- float name=threshold0.005/float str name=buildOnCommittrue/str /lst /searchComponent I'm using solr 5.1. Regards, Edwin -- -- Benedetti Alessandro Visiting card : http://about.me/alessandro_benedetti Tyger, tyger burning bright In the forests of the night, What immortal hand or eye Could frame thy fearful symmetry? William Blake - Songs of Experience -1794 England
Re: Best strategy for logging security
I am using log4j.properties which logs core name with each query... Is there a way to generate separate logs for each core ? Regards On Mon, Jun 1, 2015 at 2:13 PM, Rajesh Hazari rajeshhaz...@gmail.com wrote: Logging : Just use logstash to a parse your logs for all collection and logstash forwarder and lumberjack at your solr replicas in your solr cloud to send the log events to you central logstash server and send it to back to solr (either the same or different instance) to a different collection. The default log4j.properties that comes with solr dist can log core name with each query log. Security: suggest you to go through this wiki https://wiki.apache.org/solr/SolrSecurity *Thanks,* *Rajesh,* *(mobile) : 8328789519.* On Mon, Jun 1, 2015 at 11:20 AM, Vishal Swaroop vishal@gmail.com wrote: It will be great if you can provide your valuable inputs on strategy for logging security... Thanks a lot in advance... Logging : - Is there a way to implement logging for each cores separately. - What will be the best strategy to log every query details (like source IP, search query, etc.) at some point we will need monthly reports for analysis. Securing SOLR : - We need to implement SOLR security from client as well as server side... requests will be performed via web app as well as other server side apps e.g. curl... Please suggest about the best approach we can follow... link to any documentation will also help. Environment : SOLR 4.7 configured on Tomcat 7 (Linux)
Re: Need help in velocity search UI
It's 'Boost by Price' field in default velocity template. You can find it in query.vm. R. Michał 2015-06-02 13:28 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: what is bf field and where can I find? On 2 June 2015 at 15:42, Michał B. . m.bienkow...@gmail.com wrote: HI, Look how bf field is done. You could simply add hidden field with name 'fq' and in JS append values to it. Regards, Michal 2015-06-02 11:17 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); /script - On 2 June 2015 at 14:29, Michał B. . m.bienkow...@gmail.com wrote: Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ? -- Michał Bieńkowski -- Michał Bieńkowski -- Michał Bieńkowski
Re: Need help in velocity search UI
HI, Look how bf field is done. You could simply add hidden field with name 'fq' and in JS append values to it. Regards, Michal 2015-06-02 11:17 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); /script - On 2 June 2015 at 14:29, Michał B. . m.bienkow...@gmail.com wrote: Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ? -- Michał Bieńkowski -- Michał Bieńkowski
all terms and frequency
Hi, There is a way to use the Solr API to see all the words in all the documents and their frequency in each document? I saw the term vector component but couldn't see any way to see the vector for each word... Thanks, Shani - Intel Electronics Ltd. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Re: Need help in velocity search UI
Hey Michal thanks for your reply. Actually I am beginner I am listing below steps to perform. 1. add entry in query.vm add hidden field with name 'fq' 2. now through js append in this field My problem is ,facet_fields.vm is another file how can I get 'fq' access this in another file? Can you help me in writing sequence of steps to make it work? On 2 June 2015 at 17:01, Michał B. . m.bienkow...@gmail.com wrote: It's 'Boost by Price' field in default velocity template. You can find it in query.vm. R. Michał 2015-06-02 13:28 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: what is bf field and where can I find? On 2 June 2015 at 15:42, Michał B. . m.bienkow...@gmail.com wrote: HI, Look how bf field is done. You could simply add hidden field with name 'fq' and in JS append values to it. Regards, Michal 2015-06-02 11:17 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); /script - On 2 June 2015 at 14:29, Michał B. . m.bienkow...@gmail.com wrote: Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com : Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ? -- Michał Bieńkowski -- Michał Bieńkowski -- Michał Bieńkowski -- Hi, Find my attached resume. I have total around 7 years of work experience. I worked for Amazon and Expedia in my previous assignments and currently I am working with start- up technology company called Insideview in hyderabad. Regards Jeetendra
Spellcheck across multiple collections
Hi, Is there a way to collate the spellcheck across different collection? I understand that for select query, this can be done by setting collection=collection1,collection2 at the query. However, when I do that for spellcheck, Solr did not return me any result on the spellcheck when I entered a wrong spelling in the query. I can only get results when I search on a single collection. I'm using Solr 5.1 Regards, Edwin
AW: Occasionally getting error in solr suggester component.
Thanks! -Ursprüngliche Nachricht- Von: Erick Erickson [mailto:erickerick...@gmail.com] Gesendet: Montag, 1. Juni 2015 17:26 An: solr-user@lucene.apache.org Betreff: Re: Occasionally getting error in solr suggester component. Attach suggester.build=true or suggester.buildAll=true to any request to suggester to rebuild. OR add buildOnStartup or buildOnCommit or buildOnOptimize to the definition in solrConfig. BUT: building can be a _very_ expensive operation. For document-based indexes, the build process reads through _all_ of the _stored_ documents in your index, and that can take many minutes so I recommend against these options for a large index, and strongly recommend you test these with a large corpus. Best, Erick On Mon, Jun 1, 2015 at 4:01 AM, Clemens Wyss DEV clemens...@mysign.ch wrote: Lucene 5.1: I am (also) facing java.lang.IllegalStateException: suggester was not built At the very moment no new documents seem tob e added to the index/core. Will a reboot sanitize the index/core? I (still) have str name=buildOnCommittrue/str How can I tell Solr to peridoically update the suggestions? If not possible per configuration (in solrconfig.xml), what ist he preferred approach through SolrJ? Thx Clemens -Ursprüngliche Nachricht- Von: Michael Sokolov [mailto:msoko...@safaribooksonline.com] Gesendet: Donnerstag, 15. Januar 2015 19:52 An: solr-user@lucene.apache.org Betreff: Re: Occasionally getting error in solr suggester component. That sounds like a good approach to me. Of course it depends how often you commit, and what your tolerance is for delay in having suggestions appear, but it sounds as if you have a good understanding of the tradeoffs there. -Mike On 1/15/15 10:31 AM, Dhanesh Radhakrishnan wrote: Hi, From Solr 4.7 onwards, the implementation of this Suggester is changed. The old SpellChecker based search component is replaced with a new suggester that utilizes Lucene suggester module. The latest Solr download is preconfigured with this new suggester I;m using Solr 4.10 and suggestion are based on query /suggest instead of /spell. So what I did is that in changed to str name=buildOnCommitfalse/str Its not good that each time rebuild the index on commit , however, I would like to build the index on certain time period, say 1 hour. The lookup data will be built only when requested by URL parameter suggest.build=true http://localhost:8983/solr/ha/suggest?suggest.build=true; So this will rebuild the index again and the changes will reflect in the suggester. There are certain pros and cons for this. Issue is that the change will reflect only on certain time interval, here 1 hour. Advantage is that we can avoid the rebuilt index on every commit or optimize. Is this the right way ?? or any that I missed ??? Regards dhanesh s.r On Thu, Jan 15, 2015 at 3:20 AM, Michael Sokolov msoko...@safaribooksonline.com wrote: did you build the spellcheck index using spellcheck.build as described here: https://cwiki.apache.org/confluence/display/solr/Spell+Checking ? -Mike On 01/14/2015 07:19 AM, Dhanesh Radhakrishnan wrote: Hi, Thanks for the reply. As you mentioned in the previous mail I changed buildOnCommit=false in solrConfig. After that change, suggestions are not working. In Solr 4.7 introduced a new approach based on a dedicated SuggestComponent I'm using that component to build suggestions and lookup implementation is AnalyzingInfixLookupFactory Is there any work around ?? On Wed, Jan 14, 2015 at 12:47 AM, Michael Sokolov msoko...@safaribooksonline.com wrote: I think you are probably getting bitten by one of the issues addressed in LUCENE-5889 I would recommend against using buildOnCommit=true - with a large index this can be a performance-killer. Instead, build the index yourself using the Solr spellchecker support (spellcheck.build=true) -Mike On 01/13/2015 10:41 AM, Dhanesh Radhakrishnan wrote: Hi all, I am experiencing a problem in Solr SuggestComponent Occasionally solr suggester component throws an error like Solr failed: {responseHeader:{status:500,QTime:1},error:{msg:sugges t er was not built,trace:java.lang.IllegalStateException: suggester was not built\n\tat org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester. lookup(AnalyzingInfixSuggester.java:368)\n\tat org.apache.lucene.search.suggest.analyzing.AnalyzingInfixSuggester. lookup(AnalyzingInfixSuggester.java:342)\n\tat org.apache.lucene.search.suggest.Lookup.lookup(Lookup.java:240)\n \ tat org.apache.solr.spelling.suggest.SolrSuggester. getSuggestions(SolrSuggester.java:199)\n\tat org.apache.solr.handler.component.SuggestComponent. process(SuggestComponent.java:234)\n\tat org.apache.solr.handler.component.SearchHandler.handleRequestBody ( SearchHandler.java:218)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(
Problem add document
Hi, I have a problem with an encode to index in solr. You can have more information on : http://stackoverflow.com/questions/30531051/solr-error-when-update-a-document?noredirect=1#comment49152956_30531051 Thanks a lot, Sincerely, Uruca
Re: Sorting in Solr
Hi Erick, I didn't know this isn't memory bound, but it got to be CPU bound, no? Thanks Steve On Mon, Jun 1, 2015 at 12:27 PM, Erick Erickson erickerick...@gmail.com wrote: Steve: Surprisingly, the number of hits is completely irrelevant for the memory requirements for sorting. The base memory size is, AFAIK, an array of maxDoc ints (you can find maxDoc on the admin screen). There's some additional overhead, but that's the base size. If you sue DocValues, much of the overhead is kept in the MMapDirectory space IIRC. Best, Erick On Mon, Jun 1, 2015 at 8:41 AM, Shawn Heisey apa...@elyograg.org wrote: On 6/1/2015 9:29 AM, Steven White wrote: I need to be able to sot in Solr. Obviously, I need to do this in a way sorting won't cause OOM when a result may contain 1000's of hits if not millions. Can you guide me on how I can do this? Is there a way to tell Solr sort top N results (discarding everything else) or must such sorting be cone on the client side? Solr supports sorting. https://wiki.apache.org/solr/CommonQueryParameters#sort https://cwiki.apache.org/confluence/display/solr/Common+Query+Parameters#CommonQueryParameters-ThesortParameter I think we may have an omission from the docs -- docValues can also be used for sorting, and may also offer a performance advantage. Thanks, Shawn
Re: Need help in velocity search UI
what is bf field and where can I find? On 2 June 2015 at 15:42, Michał B. . m.bienkow...@gmail.com wrote: HI, Look how bf field is done. You could simply add hidden field with name 'fq' and in JS append values to it. Regards, Michal 2015-06-02 11:17 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); /script - On 2 June 2015 at 14:29, Michał B. . m.bienkow...@gmail.com wrote: Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ? -- Michał Bieńkowski -- Michał Bieńkowski
Solr Atomic Updates
Hi! I'm using *SOLR 4.4.0* for searching in my project. Now I am facing a problem of atomic updates in multiple cores. From wiki: curl *http://localhost:8983/solr/update http://localhost:8983/solr/update *-H 'Content-type:application/json' -d ' [ { *id*: *TestDoc1*, title : {set:test1}, revision : {inc:3}, publisher : {add:TestPublisher} }, { id: TestDoc2, publisher : {add:TestPublisher} } ]' As well as I understand, this means that the document, for example, with id *TestDoc1*, will be searched for updating *only in one core*. And if there is no any document with id *TestDoc1*, the document will be created. Can I somehow to specify the* list of cores* for searching and then updating necessary document with specific id? It's something like *shards *parameter in *select* query. From wiki: #now do a distributed search across both servers with your browser or curl curl 'http://localhost:8983/solr/*select*?*shards*=localhost:8983/solr,localhost:7574/solrindent=trueq=ipod+solr' Or is it planned in the future? Thanks in advance. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Best regards, Batalova Kseniya
Looking for doc on LimitTokenCountFilterFactory
Hi everyone I cannot find much useful info on LimitTokenCountFilterFactory other than the format and the attributes it takes per this link: http://lucene.apache.org/core/4_3_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LimitTokenCountFilterFactory.html. It is also not mentioned at this link: https://cwiki.apache.org/confluence/display/solr/ I'm looking for it to find out what happens if this filter is missing from a fieldType. For example, when it is missing, is that the same as no-limit or some hard-wired limit takes effect? Does anyone know? With older versions of Solr, I had to modify maxFieldLength to get all my data into the index so I was looking for it and found that maxFieldLength was removed as of 4.0. I'm using Solr 5.1. Thanks Steve
Re: Need help in velocity search UI
Jeetendra, It doesn't matter if it is a different file it is included in result page so its content is available for you. At the beginning I suggest for you an exercise in which you could at simple html input and find out that it is in URL. Than try to modify it in JS and than name it fq :) regards, Michał 2015-06-02 13:58 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hey Michal thanks for your reply. Actually I am beginner I am listing below steps to perform. 1. add entry in query.vm add hidden field with name 'fq' 2. now through js append in this field My problem is ,facet_fields.vm is another file how can I get 'fq' access this in another file? Can you help me in writing sequence of steps to make it work? On 2 June 2015 at 17:01, Michał B. . m.bienkow...@gmail.com wrote: It's 'Boost by Price' field in default velocity template. You can find it in query.vm. R. Michał 2015-06-02 13:28 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: what is bf field and where can I find? On 2 June 2015 at 15:42, Michał B. . m.bienkow...@gmail.com wrote: HI, Look how bf field is done. You could simply add hidden field with name 'fq' and in JS append values to it. Regards, Michal 2015-06-02 11:17 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $ field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); /script - On 2 June 2015 at 14:29, Michał B. . m.bienkow...@gmail.com wrote: Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com : Hi All I was trying to modify the facet_fields.vm do anybody know where is the location of the function url_for_facet_filter ? Also where will be log stored ? -- Michał Bieńkowski -- Michał Bieńkowski -- Michał Bieńkowski -- Hi, Find my attached resume. I have total around 7 years of work experience. I worked for Amazon and Expedia in my previous assignments and
Re: SolrCloud 5.1 startup looking for standalone config
ok thanks, continuing... numShards in SOLR_OPTS isn't a good idea, what happens if you want to create a collection with 5 shards?) yes I was following my old pattern CATALINA_OPTS=${CATALINA_OPTS} -DnumShards=n down the nodes and nuke the directories you created by hand and bring the nodes back up yes I did this create the collection via the Collections API CREATE I did this but kept getting not running in SolrCloud mode. Added the -c option to my service script like this su -c SOLR_INCLUDE=$SOLR_ENV $SOLR_INSTALL_DIR/bin/solr $SOLR_CMD -c - $RUNAS and it did start in cloud mode. Is the -c necessary and is that the right place for it? I thought uncommenting the ZK param in solr.in.sh would put it in cloud mode. Reran the CREATE and got a shard1 and shard2 in the GUI cloud view. New directories are arc_search_shard1_replica1 and arc_search_shard2_replica1. Is this because I have only 2 Solr hosts running? I'm used to adding nodes one by one and having the replica assignments start when numShards count is exceeded. Transitioning from 4.2 to 5.1 and it's quite different! -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5-1-startup-looking-for-standalone-config-tp4209118p4209222.html Sent from the Solr - User mailing list archive at Nabble.com.
index time boosting through partial update
Hi I'm using solr-4.7.2 and trying to set boost at field level at index time in atomic updates(tried though xml update handler). But its not happening. Also if you index the full doc with index time boost and update any other field partially then also the boost value of other field changes to 1.0. There must be a JIRA filed for this bug. But I was not able to find that. Could anyone please help? Regards Sankalp *SSE* 246 OKHLA PHASE III, NEW DELHI 110 020, INDIA
Re: Index optimize runs in background.
Erick! I could not find any underlying setting of 10 minutes. It is not only optimize but commit is also behaving in the same fashion and is taking lesser time than usually had taken. As per my observation both are running in background. On Fri, May 29, 2015 at 7:21 PM, Erick Erickson erickerick...@gmail.com wrote: I'm not talking about you setting a timeout, but the underlying connection timing out... The 10 minutes then the indexer exits comment points in that direction. Best, Erick On Thu, May 28, 2015 at 11:43 PM, Modassar Ather modather1...@gmail.com wrote: I have not added any timeout in the indexer except zk client time out which is 30 seconds. I am simply calling client.close() at the end of indexing. The same code was not running in background for optimize with solr-4.10.3 and org.apache.solr.client.solrj.impl.CloudSolrServer. On Fri, May 29, 2015 at 11:13 AM, Erick Erickson erickerick...@gmail.com wrote: Are you timing out on the client request? The theory here is that it's still a synchronous call, but you're just timing out at the client level. At that point, the optimize is still running it's just the connection has been dropped Shot in the dark. Erick On Thu, May 28, 2015 at 10:31 PM, Modassar Ather modather1...@gmail.com wrote: I could not notice it but with my past experience of commit which used to take around 2 minutes is now taking around 8 seconds. I think this is also running as background. On Fri, May 29, 2015 at 10:52 AM, Modassar Ather modather1...@gmail.com wrote: The indexer takes almost 2 hours to optimize. It has a multi-threaded add of batches of documents to org.apache.solr.client.solrj.impl.CloudSolrClient. Once all the documents are indexed it invokes commit and optimize. I have seen that the optimize goes into background after 10 minutes and indexer exits. I am not sure why this 10 minutes it hangs on indexer. This behavior I have seen in multiple iteration of the indexing of same data. There is nothing significant I found in log which I can share. I can see following in log. org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} On Wed, May 27, 2015 at 10:59 PM, Erick Erickson erickerick...@gmail.com wrote: All strange of course. What do your Solr logs show when this happens? And how reproducible is this? Best, Erick On Wed, May 27, 2015 at 4:00 AM, Upayavira u...@odoko.co.uk wrote: In this case, optimising makes sense, once the index is generated, you are not updating It. Upayavira On Wed, May 27, 2015, at 06:14 AM, Modassar Ather wrote: Our index has almost 100M documents running on SolrCloud of 5 shards and each shard has an index size of about 170+GB (for the record, we are not using stored fields - our documents are pretty large). We perform a full indexing every weekend and during the week there are no updates made to the index. Most of the queries that we run are pretty complex with hundreds of terms using PhraseQuery, BooleanQuery, SpanQuery, Wildcards, boosts etc. and take many minutes to execute. A difference of 10-20% is also a big advantage for us. We have been optimizing the index after indexing for years and it has worked well for us. Every once in a while, we upgrade Solr to the latest version and try without optimizing so that we can save the many hours it take to optimize such a huge index, but find optimized index work well for us. Erick I was indexing today the documents and saw the optimize happening in background. On Tue, May 26, 2015 at 9:12 PM, Erick Erickson erickerick...@gmail.com wrote: No results yet. I finished the test harness last night (not really a unit test, a stand-alone program that endlessly adds stuff and tests that every commit returns the correct number of docs). 8,000 cycles later there aren't any problems reported. Siiigh. On Tue, May 26, 2015 at 1:51 AM, Modassar Ather modather1...@gmail.com wrote: Hi, Erick you mentioned about a unit test to test the optimize running in background. Kindly share your findings if any. Thanks, Modassar On Mon, May 25, 2015 at 11:47 AM, Modassar Ather modather1...@gmail.com wrote: Thanks everybody for your replies. I have noticed the optimization running in background every time I indexed. This is 5 node cluster with solr-5.1.0 and uses the CloudSolrClient. Kindly share your findings on this issue. Our index has almost 100M documents running on SolrCloud. We have
Re: Need help in velocity search UI
Jeetendra, Just be aware that /browse is great for demo UIs, but shouldn't be used in production, as it has no authentication. Also, from a development perspective, it lacks a controller meaning you may need to use javascript to fix URLs/etc. Upayavira On Tue, Jun 2, 2015, at 02:50 PM, Michał B. . wrote: Jeetendra, It doesn't matter if it is a different file it is included in result page so its content is available for you. At the beginning I suggest for you an exercise in which you could at simple html input and find out that it is in URL. Than try to modify it in JS and than name it fq :) regards, Michał 2015-06-02 13:58 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hey Michal thanks for your reply. Actually I am beginner I am listing below steps to perform. 1. add entry in query.vm add hidden field with name 'fq' 2. now through js append in this field My problem is ,facet_fields.vm is another file how can I get 'fq' access this in another file? Can you help me in writing sequence of steps to make it work? On 2 June 2015 at 17:01, Michał B. . m.bienkow...@gmail.com wrote: It's 'Boost by Price' field in default velocity template. You can find it in query.vm. R. Michał 2015-06-02 13:28 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: what is bf field and where can I find? On 2 June 2015 at 15:42, Michał B. . m.bienkow...@gmail.com wrote: HI, Look how bf field is done. You could simply add hidden field with name 'fq' and in JS append values to it. Regards, Michal 2015-06-02 11:17 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $ field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); /script - On 2 June 2015 at 14:29, Michał B. . m.bienkow...@gmail.com wrote: Look at the VM_global_library.vm file. Regards, Michal 2015-06-02 10:47 GMT+02:00 Jeetendra Gangele gangele...@gmail.com : Hi All I was trying to
Re: all terms and frequency
Yes, we can get all the distinct terms from each collection solr index using the below call http://{host}:{port}/solr/collection1/admin/luke?wt=jsonfl=fieldnumTerms=2 the fields:{.} of response is your distinct terms in that collection. I'm sure we can get this using solrj also but never tried. *Rajesh**.* On Tue, Jun 2, 2015 at 6:32 AM, Chaushu, Shani shani.chau...@intel.com wrote: Hi, There is a way to use the Solr API to see all the words in all the documents and their frequency in each document? I saw the term vector component but couldn't see any way to see the vector for each word... Thanks, Shani - Intel Electronics Ltd. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Re: Problem add document
Please post the whole question here. Many people read mail offline - they won't be able to understand your request. Thanks! On Tue, Jun 2, 2015, at 02:43 PM, antoine charron wrote: Hi, I have a problem with an encode to index in solr. You can have more information on : http://stackoverflow.com/questions/30531051/solr-error-when-update-a-document?noredirect=1#comment49152956_30531051 Thanks a lot, Sincerely, Uruca
Looking for help in building a configuration that should be simple
I've been asked to build a sample configuration of SolrCloud using Solr 4.10. I want to have two instances (virtual machines) each with two solr nodes. Let's call the instances 1 and 2, and the nodes 1AO, 1BB, 2AB, and 2BO. I want 1AO to be the owner of that shard with 2AB as the backup, and 2BO to be the owner of its data and have 1BB as its backup. I also want to use an external ZooKeeper that we already have and trust for all 4 solr nodes. Is this something that is doable, and what does it take to make it so? Thanks. Dave Patterson
Re: Solr Atomic Updates
What exactly is the problem? And why do you care about cores, per se - other than to send the update to the core/collection you are trying to update? You should specify the core/collection name in the URL. You should also be using the Solr reference guide rather than the (old) wiki: https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents -- Jack Krupansky On Tue, Jun 2, 2015 at 10:15 AM, Ксения Баталова batalova...@gmail.com wrote: Hi! I'm using *SOLR 4.4.0* for searching in my project. Now I am facing a problem of atomic updates in multiple cores. From wiki: curl *http://localhost:8983/solr/update http://localhost:8983/solr/update *-H 'Content-type:application/json' -d ' [ { *id*: *TestDoc1*, title : {set:test1}, revision : {inc:3}, publisher : {add:TestPublisher} }, { id: TestDoc2, publisher : {add:TestPublisher} } ]' As well as I understand, this means that the document, for example, with id *TestDoc1*, will be searched for updating *only in one core*. And if there is no any document with id *TestDoc1*, the document will be created. Can I somehow to specify the* list of cores* for searching and then updating necessary document with specific id? It's something like *shards *parameter in *select* query. From wiki: #now do a distributed search across both servers with your browser or curl curl ' http://localhost:8983/solr/*select*?*shards*=localhost:8983/solr,localhost:7574/solrindent=trueq=ipod+solr ' Or is it planned in the future? Thanks in advance. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Best regards, Batalova Kseniya
Re: Looking for doc on LimitTokenCountFilterFactory
For most of the filter factory javadoc you need to click on the underlying filter to see the detail doc for the filter. So, see: http://lucene.apache.org/core/4_3_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LimitTokenCountFilter.html If this filter is not used, there is no limit other than the Lucene limit of unique terms per segment, which is in the billions. Yeah, that should be more clearly documented. -- Jack Krupansky On Tue, Jun 2, 2015 at 10:29 AM, Steven White swhite4...@gmail.com wrote: Hi everyone I cannot find much useful info on LimitTokenCountFilterFactory other than the format and the attributes it takes per this link: http://lucene.apache.org/core/4_3_0/analyzers-common/org/apache/lucene/analysis/miscellaneous/LimitTokenCountFilterFactory.html . It is also not mentioned at this link: https://cwiki.apache.org/confluence/display/solr/ I'm looking for it to find out what happens if this filter is missing from a fieldType. For example, when it is missing, is that the same as no-limit or some hard-wired limit takes effect? Does anyone know? With older versions of Solr, I had to modify maxFieldLength to get all my data into the index so I was looking for it and found that maxFieldLength was removed as of 4.0. I'm using Solr 5.1. Thanks Steve
Re: Sorting in Solr
Never meant to imply that it wasn't memory bound! It _is_ memory bound (or at least can be), it's just that the memory consumed isn't proportional to the number of documents matched, but to the total number of documents in your index. Although that all said, once all the structures are in memory, the load shifts to CPU. Best, Erick On Tue, Jun 2, 2015 at 7:20 AM, Steven White swhite4...@gmail.com wrote: Hi Erick, I didn't know this isn't memory bound, but it got to be CPU bound, no? Thanks Steve On Mon, Jun 1, 2015 at 12:27 PM, Erick Erickson erickerick...@gmail.com wrote: Steve: Surprisingly, the number of hits is completely irrelevant for the memory requirements for sorting. The base memory size is, AFAIK, an array of maxDoc ints (you can find maxDoc on the admin screen). There's some additional overhead, but that's the base size. If you sue DocValues, much of the overhead is kept in the MMapDirectory space IIRC. Best, Erick On Mon, Jun 1, 2015 at 8:41 AM, Shawn Heisey apa...@elyograg.org wrote: On 6/1/2015 9:29 AM, Steven White wrote: I need to be able to sot in Solr. Obviously, I need to do this in a way sorting won't cause OOM when a result may contain 1000's of hits if not millions. Can you guide me on how I can do this? Is there a way to tell Solr sort top N results (discarding everything else) or must such sorting be cone on the client side? Solr supports sorting. https://wiki.apache.org/solr/CommonQueryParameters#sort https://cwiki.apache.org/confluence/display/solr/Common+Query+Parameters#CommonQueryParameters-ThesortParameter I think we may have an omission from the docs -- docValues can also be used for sorting, and may also offer a performance advantage. Thanks, Shawn
Re: Looking for help in building a configuration that should be simple
What have you tried? Have you worked with the Collections API? See: https://cwiki.apache.org/confluence/display/solr/Collections+API, and note that in th eupper left you can download older versions of the ref guide, including 4.10. I'd really advise against getting hung up on this bit of your question I want 1AO to be the owner of that shard with 2AB as the backup, and 2BO to be the owner of its data and have 1BB as its backup. First off, the default behavior is to distribute shards and replicas across the nodes available in a balanced fashion. Then one of the points of SolrCloud is that you shouldn't have to worry about who is or is not the leader, and it'll shift around anyway as you bring nodes up/down in different sequences. Leader is a lightweight role, especially in a 2x2 setup and completely not worth obsessing over. You can't really assign specific shards to specific nodes in 4.10, but you can in the 5.1 with the createNodeSet.shuffle parameter, create a one-replica (i.e. leader-only) collection then use ADDREPLICA to add the replicas for the shards exactly where you want. . But again, IMO it's a waste of time. Up to you of course, Erick On Tue, Jun 2, 2015 at 9:08 AM, David Patterson patt...@gmail.com wrote: I've been asked to build a sample configuration of SolrCloud using Solr 4.10. I want to have two instances (virtual machines) each with two solr nodes. Let's call the instances 1 and 2, and the nodes 1AO, 1BB, 2AB, and 2BO. I want 1AO to be the owner of that shard with 2AB as the backup, and 2BO to be the owner of its data and have 1BB as its backup. I also want to use an external ZooKeeper that we already have and trust for all 4 solr nodes. Is this something that is doable, and what does it take to make it so? Thanks. Dave Patterson
Re: Best strategy for logging security
Logstash is open-source and free. At some point Sematext contributed Solr connector/output to Logstash. Here are some numbers about Logstash (and rsyslog, which is also an option, though it doesn't have Solr output): http://blog.sematext.com/2015/05/18/tuning-elasticsearch-indexing-pipeline-for-logs/ If you are new to Logstash, this is a good one: http://blog.sematext.com/2013/12/19/getting-started-with-logstash/ Note: Solr was mentioned as the destination for logs here, but it's not the only option. You can send your logs to other systems and services, including off-site ones, those that also archive your old logs for audit or other purposes, have more than just basic log search functionality, etc. HTH Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr Elasticsearch Support * http://sematext.com/ On Mon, Jun 1, 2015 at 4:47 PM, Vishal Swaroop vishal@gmail.com wrote: Thanks Rajesh... just trying to figure out if *logstash *is opensource and free ? On Mon, Jun 1, 2015 at 2:13 PM, Rajesh Hazari rajeshhaz...@gmail.com wrote: Logging : Just use logstash to a parse your logs for all collection and logstash forwarder and lumberjack at your solr replicas in your solr cloud to send the log events to you central logstash server and send it to back to solr (either the same or different instance) to a different collection. The default log4j.properties that comes with solr dist can log core name with each query log. Security: suggest you to go through this wiki https://wiki.apache.org/solr/SolrSecurity *Thanks,* *Rajesh,* *(mobile) : 8328789519.* On Mon, Jun 1, 2015 at 11:20 AM, Vishal Swaroop vishal@gmail.com wrote: It will be great if you can provide your valuable inputs on strategy for logging security... Thanks a lot in advance... Logging : - Is there a way to implement logging for each cores separately. - What will be the best strategy to log every query details (like source IP, search query, etc.) at some point we will need monthly reports for analysis. Securing SOLR : - We need to implement SOLR security from client as well as server side... requests will be performed via web app as well as other server side apps e.g. curl... Please suggest about the best approach we can follow... link to any documentation will also help. Environment : SOLR 4.7 configured on Tomcat 7 (Linux)
Re: Need help in velocity search UI
Jeetendra - First steps - implement what you want with static HTML/JavaScript so that you have the interface in place. It can get hairy with the Velocity/HTML/JavaScript/Solr stuff all intermingled*. By doing things statically (without Velocity, and even without Solr) you’ll get the UI pieces implemented and then can make it dynamic. The #url_for_facet_filter is designed for single select faceting. For multi-select faceting (within the Velocity UI) you’ll need JavaScript that can allow a user to select multiple facets before submitting (or maybe submit each one and allow refining). There is no built-in example with multi-select faceting to go by. You’ll need to get savvy with the tag/ex faceting “local” parameters - see https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-LocalParametersforFaceting https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-LocalParametersforFaceting for more details. As has been said on this thread, the /browse UI may not be the right way for you to build this - if your purpose is a proof of concept or demo that’s fine. If it is a production system for other users to access you must be aware that /browse (or any wt=velocity) UI is *directly* Solr and any user that can hit that could also affect the actual Solr server and indexed data, unless you employed some kind of proxy or firewall to prevent unauthorized access. Erik — Erik Hatcher, Senior Solutions Architect http://www.lucidworks.com http://www.lucidworks.com/ * It got cleaned up a lot in 5x/trunk of Solr, see http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/contrib/velocity/src/resources/velocity/facets.vm http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/contrib/velocity/src/resources/velocity/facets.vm and http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/contrib/velocity/src/resources/_macros.vm http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/contrib/velocity/src/resources/_macros.vm On Jun 2, 2015, at 7:58 AM, Jeetendra Gangele gangele...@gmail.com wrote: Hey Michal thanks for your reply. Actually I am beginner I am listing below steps to perform. 1. add entry in query.vm add hidden field with name 'fq' 2. now through js append in this field My problem is ,facet_fields.vm is another file how can I get 'fq' access this in another file? Can you help me in writing sequence of steps to make it work? On 2 June 2015 at 17:01, Michał B. . m.bienkow...@gmail.com wrote: It's 'Boost by Price' field in default velocity template. You can find it in query.vm. R. Michał 2015-06-02 13:28 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: what is bf field and where can I find? On 2 June 2015 at 15:42, Michał B. . m.bienkow...@gmail.com wrote: HI, Look how bf field is done. You could simply add hidden field with name 'fq' and in JS append values to it. Regards, Michal 2015-06-02 11:17 GMT+02:00 Jeetendra Gangele gangele...@gmail.com: Hi Michael Thanks for your reply. I got this is an macro. I am trying to modify the facet_fields.vm. how it works is like for each $ field.name, $facet.name it gives the result. But now what I want is I want user to select multiple facets from fields and once user hit the filter button which I provided below I need to send all the select fields with selected facets? how can I do this . since url_for_facet_filter is executed server side how can I pass all the selected facets of the fields, you can have a look at the UI below http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#elseemmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val();
Re: Need help in velocity search UI
Thanks Erik for reply. I am building some kind of semantic search for one of the google Hackathon project so I need this UI for Demo only. I already have java script which is allowing user to select multiple facets and I am getting all the facets selecting by user in array in java script. Now I want to pass this to solr? by using url_for_facet_filter ? below is my UI and java script which allow use to select multiple things and capture in java scripts array, now the question is how can I make rest call using these captured data to slor. One way I can thing of is like create the URI like below and when user hot on submit call the below URI(highlights things I will pass as variable from user selected facets) http://54.242.13.110:8080/solr/collection1/browse?fq=idx_type:companiesq=fq=(*:* AND topics:*revenue* AND topics:*reports/results* AND NOT topics: *aerospace*”) Will this work? or I need to create the search page on which this data will be shown or on the existing page data will be filtered and rendered? http://54.146.195.106:8080/solr/collection1/browse Also below is my facet_fields.vm source code. please help me how can I selected things to url_for_facet_filter #** * Display facets based on field values * e.g.: fields specified by facet.field= *# #if($response.facetFields) h2 #annTitle(Facets generated by adding facet.field= to the request) Filters /h2 #foreach($field in $response.facetFields) ## Hide facets without value #if($field.values.size() 0) span class=facet-field$field.name/span ul #foreach($facet in $field.values) li span class=showhide#if($facet.name!=$null)$ facet.name#else http://facet.name/#else emmissing/em#end($facet.count)/span div class=showhidecontainer select name=$facet.name class=filter $field.name option value=-1 selectedselect/option option value=1Yes/option option value=0No/option /select /div /li #end /ul #end ## end if 0 #end## end for each facet field #end ## end if response has facet fields input type=button value=filter id=filterme/ script function filter() { var topicsYes = [], topicsNo = [], semsYes = [], semsNo = []; $(.filter).each(function() { //decide if its topic or sem var filter = $(this); var value = filter.val(); if(filter.is(.topics)) { if(value === 1) topicsYes.push(filter.attr(name)); if(value === 0) topicsNo.push(filter.attr(name)); } else if(filter.is(.semTags)) { if(value === 1) semsYes.push(filter.attr(name)); if(value === 0) semsNo.push(filter.attr(name)); } else { console.log(something wrong); } }); console.log(topicsYes, topicsNo, semsYes, semsNo); } $(function() { $(#filterme).click(filter); $(.showhide).click(function() { $(this).next().slideToggle(); }); }); On 2 June 2015 at 21:56, Erik Hatcher erik.hatc...@lucidworks.com wrote: Jeetendra - First steps - implement what you want with static HTML/JavaScript so that you have the interface in place. It can get hairy with the Velocity/HTML/JavaScript/Solr stuff all intermingled*. By doing things statically (without Velocity, and even without Solr) you’ll get the UI pieces implemented and then can make it dynamic. The #url_for_facet_filter is designed for single select faceting. For multi-select faceting (within the Velocity UI) you’ll need JavaScript that can allow a user to select multiple facets before submitting (or maybe submit each one and allow refining). There is no built-in example with multi-select faceting to go by. You’ll need to get savvy with the tag/ex faceting “local” parameters - see https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-LocalParametersforFaceting https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-LocalParametersforFaceting for more details. As has been said on this thread, the /browse UI may not be the right way for you to build this - if your purpose is a proof of concept or demo that’s fine. If it is a production system for other users to access you must be aware that /browse (or any wt=velocity) UI is *directly* Solr and any user that can hit that could also affect the actual Solr server and indexed data, unless you employed some kind of proxy or firewall to prevent unauthorized access. Erik — Erik Hatcher, Senior Solutions Architect http://www.lucidworks.com http://www.lucidworks.com/ * It got cleaned up a lot in 5x/trunk of Solr, see http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/contrib/velocity/src/resources/velocity/facets.vm http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/contrib/velocity/src/resources/velocity/facets.vm and http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/contrib/velocity/src/resources/_macros.vm