Trying to index document in Solr with solr-spark library

I'm getting some errors when I try to use the solr-sparl library getting the error *KeeperErrorCode = NoNode for /live_nodes*. I download the library and compile with the branch_4.x since I'm using Cloudera 5.5.1 and Solr 4.10.3. I checked the logs of Solr and Zookeeper and I didn't find any erro

Re: solr cloud invalid shard/collection configuration

Can someone please advise considering my previous answer? -- View this message in context: http://lucene.472066.n3.nabble.com/solr-cloud-invalid-shard-collection-configuration-tp4245151p4245986.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Issues when indexing PDF files

They could be using custom fonts and non-Unicode characters. That's probably something to explore with PDF specific tools. On 17 Dec 2015 1:37 pm, "Zheng Lin Edwin Yeo" wrote: > I've checked all the files which has problem with the content in the Solr > index using the Tika app. All of them shows

Re: query to get parents without childs

You could try simply doing a not query to find all those docs that do not contain the child fields like -fq=:* Since the index is flat, the "children" are like any other fields to lucene and so this should work On Thu, 17 Dec 2015, 04:33 Novin wrote: > "Index the number of children into the pare

Re: Issues when indexing PDF files

I've checked all the files which has problem with the content in the Solr index using the Tika app. All of them shows the same issues as what I see in the Solr index. So does the issues lies with the encoding of the file? Are we able to check the encoding of the file? Regards, Edwin On 17 Dece

Re: faceting is unusable slow since upgrade to 5.3.0

Same question here Wondering if faceting performance is fixed and how to take advantage of it ? On Wed, Dec 16, 2015 at 2:57 AM, Vincenzo D'Amore wrote: > Hi all, > > given that solr 5.4 is finally released, is this what's more stable and > efficient version of solrcloud ? > > I have a webs

Re: warning while indexing

Ah. Then it might be that DIH cannot be run in parallel. Though the exception is much lower in the stack. Not sure. Maybe somebody else with more knowledge in the commit path can comment on it. On 17 Dec 2015 12:21 pm, "Midas A" wrote: > Alexandre, > > *Only two DIH, indexing different data. *

Re: warning while indexing

Alexandre, *Only two DIH, indexing different data. * On Thu, Dec 17, 2015 at 10:46 AM, Alexandre Rafalovitch wrote: > How many? On the same node? > > I am not sure if running multiple DIH is a popular case. > > My theory, still, that you are running out of a pool size there. Though if > it hap

Re: warning while indexing

How many? On the same node? I am not sure if running multiple DIH is a popular case. My theory, still, that you are running out of a pool size there. Though if it happens with even just two DIH, it could be a different issue. On 17 Dec 2015 12:01 pm, "Midas A" wrote: > Alexandre , > > we are ru

Re: warning while indexing

Alexandre , we are running multiple DIH to index data. On Thu, Dec 17, 2015 at 12:40 AM, Alexandre Rafalovitch wrote: > Are you sending documents from one client or many? > > Looks like an exhaustion of some sort of pool related to Commit within, > which I assume you are using. > > Regards, >

Re: Strange debug output for a slow query

Hmmm, take a look at the individual queries on a shard, i.e. peek at the Solr logs and see if the fq clause comes through cleanly when you see &distrib=false. I suspect this is just a glitch in assembling the debug response. If it is, it probably deserves a JIRA. In fact it deserves a JIRA in eithe

Re: Solr Basic Configuration - Highlight - Begginer

bq: but when highlight, using the text field...nothing comes up... http://localhost:8983/solr/techproducts/select?q=text:nietava&fq=id:pdf1&wt=json&indent=true&hl=true&hl.fl=text&hl.simple.pre=%3Cem%3E&hl.simple.post=%3C%2Fem%3E It's unclear what this means. No results showed up (i.e. numFound==0

Re: Solr Basic Configuration - Highlight - Begginer

Hi Erick and Teague, I found that when using the field 'text' it shows the pdf file result id:pdf1 in this case, like: http://localhost:8983/solr/techproducts/select?fq=id:pdf1&q=nietava but when highlight, using the text field...nothing comes up... http://localhost:8983/solr/techproducts/sele

Re: Append fields to a document

If you enable LazyLoading and do not request them in your 'fl' list, they should be mostly just size on disk AFAIK. Regards, Alex. Newsletter and resources for Solr beginners and intermediates: http://www.solr-start.com/ On 17 December 2015 at 08:09, Jamie Johnson wrote: > The expense i

Re: Append fields to a document

The expense is in gathering the pieces to do the indexing. There isn't much that I can do in that regard unfortunately. I need to investigate storing the fields, if they aren't returned is the expense just size on disk or is there a memory cost as well? On Dec 16, 2015 7:43 PM, "Alexandre Rafalov

Re: Append fields to a document

ExternalFileField might be useful in some situations. But also, is it possible that your Solr schema configuration is not best suited for your domain? Is it - for example - possible that the additional data should be in child records? Pure guesswork here, not enough information. But, as described

Re: Where/howto store store.xml in Zookeeper?

On 12/16/2015 5:18 AM, Andrej van der Zee wrote: > I have tried several variations to upload solr.xml to Zookeeper like these: > > /opt/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -confdir > /etc/zookeeper/solr.xml -confname solr -z 1.2.3.4:2181 > > But somehow the Solr instances cant

Strange debug output for a slow query

Here is the query URL that I did. The info included in this message is slightly redacted. http://bigindy5.REDACTED.com:8982/solr/sparkmain/search?q=%28german+shepherd%29&qt=/search&start=0&fq=NOT%28feature:redact1+OR+feature:spkhistorical%29&fq=%28ip:%28AP%29+AND+price:0%29+OR+%28ip:%28BB%29%29+O

Re: Solr Basic Configuration - Highlight - Begginer

I think you're still missing the critical bit. Highlighting is completely separate from searching. In other words, you can search on one field and highlight another. What field is searched is governed by the "qf" parameter when using edismax and by the the "df" parameter configured in your request

Re: query to get parents without childs

"Index the number of children into the parent as an integer" is nice and easy solution. But I would like to know about" You could probably do that inside an UpdateProcessor, even using the Javascript ScriptUpdateProcessor. Probably simpler though in the code that pushes the docs to Solr." either

Re: JVM error v ~StubRoutines::jbyte_disjoint_arraycopy

https://wiki.apache.org/lucene-java/JavaBugs See the last entry in the OpenJDK section, you're using one of the Java versions that has issues. So the first thing I'd try is up grading my JVM. Best, Erick On Wed, Dec 16, 2015 at 2:01 PM, abhayd wrote: > hi > > I have more than 50Gb in /tmp index

Re: query to get parents without childs

So that's a good question - how do you identify parent documents that *do not* have child documents. I'm not sure how you would do that. However, you could index the number of children into the parent as an integer, then it would be easy. You could probably do that inside an UpdateProcessor, even

RE: DIH Caching w/ BerkleyBackedCache

Todd, I have no idea if this will perform acceptable with so many multiple values. I doubt the solr/patch code was really optimized for such a use case. In my production environment, I have je-6.2.31.jar on the classpath. I don't think I've tried it with other versions. James Dyer Ingram Co

Re: query to get parents without childs

Hi Scott, Actually, it is not multi value field. it is nested document. Novin On 16 December 2015 at 20:33, Scott Stults < sstu...@opensourceconnections.com> wrote: > Hi Novin, > > How are you associating parents with children? Is it a "children" > multivalued field in the parent record? If so

Re: Append fields to a document

What is the nature of your documents that reproducing them is so expensive? Whatever it is, you should spend some time trying to reduce it to something more manageable and performant. Generally, the primary recommendation is to simply reindex any documents that need to be updated since atomic updat

Re: query to get parents without childs

Hi Novin, How are you associating parents with children? Is it a "children" multivalued field in the parent record? If so you could query for records that don't have a value in that field like "-children:[* TO *]" k/r, Scott On Wed, Dec 16, 2015 at 7:29 AM, Novin Novin wrote: > Hi guys, > > I

RE: Solr Basic Configuration - Highlight - Begginer

Sorry to hear that didn't work! Let me ask a couple of questions... Have you tried the analyzer inside of the Admin Interface? It has helped me sort out a number of highlighting issues in the past. To access it, go to your Admin interface, select your core, then select Analysis from the list of

Re: Security Problems

I have opened https://issues.apache.org/jira/browse/SOLR-8429 On Wed, Dec 16, 2015 at 9:32 PM, Noble Paul wrote: > I don't this behavior is intuitive. It is very easy to misunderstand > > I would rather just add a flag to "authentication" plugin section > which says "blockUnauthenticated" : true

Re: warning while indexing

Are you sending documents from one client or many? Looks like an exhaustion of some sort of pool related to Commit within, which I assume you are using. Regards, Alex On 16 Dec 2015 4:11 pm, "Midas A" wrote: > Getting following warning while indexing ..Anybody please tell me the > reason .

Re: Solr Basic Configuration - Highlight - Begginer

Hi Teague! I configured the solrconf.xml and schema.xml exactly the way you did, only substituting the word 'documentText' per 'content' used by the techproducts sample, I reindex through : curl ' http://localhost:8983/solr/techproducts/update/extract?literal.id=pdf1&commit=true' -F "Emmanuel=@/

Re: Solr High Availability

Thanx for the response. There were few occurrences of our SolrCloud cluster where when a primary went down in a shard, the replica didn't get promoted which eventually led to downtime. We had to restart zookeeper services (we have three zookeeper nodes) to promote the replica into primary. But I

Re: Solr Basic Configuration - Highlight - Begginer

Hi Erick, I think you are right! When I use the form 'features:accents' in my case 'content:nietava', it show as if there was not matching words... but if I take the field off having only the 'q=searchword' (q=nietava) it brings the pdf content file, as below (in XML out type): #partial snip:

Re: Solr High Availability

If you have two replicas (one leader/one replica) for each shard of your collection, and you ensure that no two replicas are on the same node, and you have three independent Zookeeper nodes, then yes, you should have HA. Upayavira On Wed, Dec 16, 2015, at 05:48 PM, Peter Tan wrote: > Hi Jack, >

Re: Solr High Availability

Hi Jack, Appreciate you helping me to clear this up. For replicationFactor = 1, that means only keeping one copy of document in the cluster. Currently, for our SolrCloud setup, we have two replicas (primary and replica) per each shard (total of 5 shards). This should achieve the HA already, cor

RE: Solr Basic Configuration - Highlight - Begginer

Hi Evert, I recently needed help with phrase highlighting and was pointed to the FastVectorHighlighter which worked out great. I just made a change to the configuration to add generateWordParts="0" and generateNumberParts="0" so that searches for things like "1a" would get highlighted correctly

Re: SolrCloud 4.8.1 - commit wait

Quick scan, but probably this: INFO o.a.solr.spelling.suggest.Suggester - build() The suggester build process can easily take many minutes, there's some explanation here: https://lucidworks.com/blog/2015/03/04/solr-suggester/ the short form is that depending on how it's defined, it may have to

Re: Solr cloud instance does not read cores from Zookeeper whilst connected

At a random guess, how are you starting Zookeeper and Solr? Is it possible that you're running the Zookeeper embedded in Solr but have an external Zookeeper running also? In that scenario you might be seeing one Zookeeper in the admin UI and another when trying to create the collection. Could you

Re: Append fields to a document

The only way to do this currently is with Atomic Updates, which require all fields to be stored except the destinations of copyField directives. see: https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents Best, Erick On Wed, Dec 16, 2015 at 7:09 AM, Jamie Johnson wrote: >

Re: Solr Basic Configuration - Highlight - Begginer

Ok, you're getting confused by all the options, an easy thing to do. You're trying to do too many things at once without making sure the basics work 1> Forget all about the f.content.hl stuff. That's there in case you want to specify different parameters for different fields in the same hi

Re: Solr 6 Distributed Join

Hi Dennis, Thank you for your help. I used your explanation to construct an innerJoin query; I think I am getting further but didn't get the results I expected. The following describes what I did – is there any chance you can tell where I am going wrong: Solr 6 Developer Builds: #2738 and #274

Re: Ugh! My term is the entire record

Yup! That was it! Thanks! (I changed "string" to "text_en" in my backup copy, too, so this doesn't happen again.) Mark On 12/16/2015 10:44 AM, Binoy Dalal wrote: What is the type of the fields in question? What you're seeing will happen if a field is of type string. If this is the case then t

Re: Issues when indexing PDF files

Hi Erik, I've shared the file on dropbox, which you can access via the link here: https://www.dropbox.com/s/rufi9esmnsmzhmw/Desmophen%2B670%2BBAe.pdf?dl=0 This is what I get from the Tika app after dropping the file in. Content-Length: 75092 Content-Type: application/pdf Type: COSName{Info} X-Pa

Re: Solr Basic Configuration - Highlight - Begginer

Hi Andrea, ok, let´s do it: 1. it does has the 'nietava' term, so it brings the only book (pdf file) has this word, and all its content as my previous message to Erick, so the content field is there. 2. using content:nietava it does not show any result as below: { "responseHeader": { "statu

RE: DIH Caching w/ BerkleyBackedCache

James, I apologize for the late response. Dyer, James-2 wrote > With the DIH request, are you specifying "cacheDeletePriorData=false" We are not specifying that property (it looks like it defaults to "false"). I'm actually seeing this issue when running a full clean/import. It appears that the

Re: Issues when indexing PDF files

Edwin - Can you share one of those PDF files? Also, drop the file into the Tika app and see what it sees directly - get the tika-app JAR and run that desktop application. Could be an encoding issue? Erik — Erik Hatcher, Senior Solutions Architect http://www.lucidworks.com

Re: Security Problems

I don't this behavior is intuitive. It is very easy to misunderstand I would rather just add a flag to "authentication" plugin section which says "blockUnauthenticated" : true which means all unauthenticated requests must be blocked. On Tue, Dec 15, 2015 at 7:09 PM, Jan Høydahl wrote: > Yes,

Issues when indexing PDF files

Hi, I'm using Solr 5.3.0 I'm indexing some PDF documents. However, for certain PDF files, there are chinese text in the documents, but after indexing, what is indexed in the content is either a series of "??" or an empty content. I'm using the post.jar that comes together with Solr. What co

Re: Ugh! My term is the entire record

What is the type of the fields in question? What you're seeing will happen if a field is of type string. If this is the case then try changing your field type to text_en or text_general depending on your requirements. On Wed, 16 Dec 2015, 19:51 Mark Fenbers wrote: > Greetings, > > I had my Solr

Re: Timeouts for create_collection

Hi, I completely started over again. Now I get the following error upon create_collection: solr@ip-172-31-11-63:/opt/solr$ ./bin/solr create_collection -c connects -replicationFactor 2 Connecting to ZooKeeper at 172.31.11.65:2181 ... Re-using existing configuration directory connects Creating n

Append fields to a document

I have a use case where we only need to append some fields to a document. To retrieve the full representation is very expensive but I can easily get the deltas. Is it possible to just add fields to an existing Solr document? I experimented with using overwrite=false, but that resulted in two docu

Timeouts for create_collection

Hi, I am newby to Solr and I am having difficulties setting up a cluster with a single Zookeeper instance and two Solr instances. The Solr intances both successfully establish sessions with the Zookeeper and I am able to upload collection configs to Zookeeper, but somehow creating a collection fro

Re: Collection API migrate statement

Hello, Thanks for your reply. As you suggested, I've tried running the operation along with the async command and it works- thank you. My next question is: Is there any way of finding out more information on the completed task? As I'm currently testing the new solr configuration, it would be

Re: Solr Basic Configuration - Highlight - Begginer

hl=f.content.hl.content (I guess) is definitely wrong. Some questions: - First, sorry, the obvious question: are you sure the documents contain the "nietava" term? - Could you try to use q=content:nietaval? - Could you paste the definition (field & fieldtype) of the content field?

Re: Solr Basic Configuration - Highlight - Begginer

Hi Andrea, Thanks for the reply! I tried with the hl.fl parameter as well, using as below: http://localhost:8983/solr/techproducts/select?q=nietava&fl=id%2C+content&wt=json&indent=true&hl=true&; hl.fl=f.content.hl.content%3D4&hl.simple.pre=%3Cem%3E&hl.simple.post=%3C%2Fem%3E with the parameter

Ugh! My term is the entire record

Greetings, I had my Solr searching capabilities working for a while. But today I inadvertently "unload"d my core from the Admin Interface. After adding it back in, it is not working right. Because Solr was down for a while in recent weeks, I have also done a full import with the clean option.

Re: integrate solr with preprocessor tools

Hi Sara, I would recommend looking at code of some component that you use currently and start from that - you can extend that class or use it as template for your own. Thanks, Emir On 16.12.2015 09:58, sara hajili wrote: hi Emir,tnx for answering now my question is how i write this class? i

Permutations of entries in a multivalued field

Hello all, we are facing the following problem: we use a multivalued string field that contains entries of the kind A/B/C/, where A,B,C are terms. We are now looking for a simple way to also find all permutations of A/B/C, so e.g. B/A/C. As a workaround we added a new field that contains all e

Re: Solr Basic Configuration - Highlight - Begginer

Hi Evert, what is the configuration of the default request handler? Did you set the hl.fl parameter? Please check here [1] the parameters that the highlighting component expects. Required parameters should be in the query string or declared within the request handler which answers to your query.

Re: similarity as a parameter

Hi Markus, I confirm (if that counts) that all current built-in similarities (expect Sweet spot) save same stuff into the norms. They can be switched/changed at search time. Actually, I am doing this today with Lucene, experimenting different term-weighting models using a single index. It would

query to get parents without childs

Hi guys, I have few parent index without child, what would wold be the query for those to get? Thanks, Novin

Solr cloud instance does not read cores from Zookeeper whilst connected

Hi, I have setup Zookeer and uploaded a collection config. But somehow it seems that Solr keeps reading core definitions locally ("Looking for core definitions underneath /opt/solr/server/solr") instead of getting it from Zookeep. Below the logs. Probably some kind of config thingy, unfortunately

Re: minimum should match, cant explain the amount of hits

Thanks! This makes sense, I will change my configuration to 2<-35% On 16-12-15 13:11, Binoy Dalal wrote: The edismax documentation confirms that when a positive % value is provided, solr will round down. If you want solr to round up set your parameter value as '-35%' On Wed, 16 Dec 2015, 17:28

Where/howto store store.xml in Zookeeper?

Hi, When I start a Solr cloud instance, I keep getting this in the log: 800 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper 800 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider 805 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.x

Re: minimum should match, cant explain the amount of hits

The edismax documentation confirms that when a positive % value is provided, solr will round down. If you want solr to round up set your parameter value as '-35%' On Wed, 16 Dec 2015, 17:28 Binoy Dalal wrote: > My guess is that solr is rounding down while calculating number of > mandatory terms.

Re: minimum should match, cant explain the amount of hits

My guess is that solr is rounding down while calculating number of mandatory terms. In your case, there are 3 terms, 65% of which is 1.95 which rounded down is 1, but 67% is 2.01 which rounded down is 2 which conforms with the results you're seeing. Maybe someone else can confirm this. On Wed, 16

Re: Solr Basic Configuration - Highlight - Begginer

Hi everyone! I think I should not have posted my server name... never had that many access attempts... 2015-12-16 9:03 GMT-02:00 Evert R. : > Hello Erick, > > Thanks again for your time. > > Here is as far as I have gone: > > 1. I started a fresh install and did the following: > > [evert@nix]$

minimum should match, cant explain the amount of hits

Hi, I'm currently searching with the following query: q="sony+led+tv". The minimum should match setting is set on: mm=2<65%. So when there are more then two terms, at least 65% of the terms should match. I'm not using the StopFilterFactory. When turning on debug, this is the parsedquery_toStri

Re: Solr Basic Configuration - Highlight - Begginer

Hello Erick, Thanks again for your time. Here is as far as I have gone: 1. I started a fresh install and did the following: [evert@nix]$ bin/solr start -e techproducts [evert@nix]$ curl ' http://localhost:8983/solr/techproducts/update/extract?literal.id=pdf1&commit=true' -F "Emmanuel=@/home/sol

Re: SolrCloud 4.8.1 - commit wait

Hi, an update. Hope you can help me. I have stopped all the other working collections, in order to have a clean log file. at 11:01:16 an hard commit has been issued 2015-12-16 11:01:49,839 [http-bio-8080-exec-824] INFO org.apache.solr.update.UpdateHandler - start commit{,optimize=false,openSea

Re: pf2 pf3 and stopwords

What is your exact use case? On Wed, 16 Dec 2015, 13:40 elisabeth benoit wrote: > Thanks for your answer. > > Actually, using a slop of 1 is something I can't do (because of other > specifications) > > I guess I'll index differently. > > Best regards, > Elisabeth > > 2015-12-14 16:24 GMT+01:00 B

Re: faceting is unusable slow since upgrade to 5.3.0

Hi all, given that solr 5.4 is finally released, is this what's more stable and efficient version of solrcloud ? I have a website which receives many search requests. It serve normally about 2000 concurrent requests, but sometime there are peak from 4000 to 1 requests in few seconds. On Janu

warning while indexing

Getting following warning while indexing ..Anybody please tell me the reason . java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@9916a67 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@79f8b5f[Terminated, poo

Re: integrate solr with preprocessor tools

hi Emir,tnx for answering now my question is how i write this class? i must use solr interfaces? i see in above link that i can use solr analyzer.but how i use that? plz say me how i start to write my own analyzer step by step... which interface i can use and change to achieve my goal? tnx On Wed,

Re: Is DIH going to be removed from Solr future versions?

Are you saying to do a local mini-collection and then mirror final result to the real one? What about deletions? Per-entry cleanup statements and so on? DIH does full updates, not just additions. Or did I miss the focus? Regards, Alex On 15 Dec 2015 11:46 pm, "Erik Hatcher" wrote: > With t

Custom auth plugin not loaded in SolrCloud

Hi, I'm trying to include a custom authentication plugin in my SolrCloud installation. It only works when I add it to server\solr-webapp\webapp\WEB-INF\lib or to the solr home directory of each node. If I add it as described here https://cwiki.apache.org/confluence/display/solr/Adding+Cust

Re: pf2 pf3 and stopwords

Thanks for your answer. Actually, using a slop of 1 is something I can't do (because of other specifications) I guess I'll index differently. Best regards, Elisabeth 2015-12-14 16:24 GMT+01:00 Binoy Dalal : > Moreover, the stopword de will work on your queries and not on your > documents, mean

Re: Partial sentence match with block join

For example: If company A is { name:"Apple Inc", location:"Los Alamos"} and company B is { name:"Banana Inc", location:"Los Angeles"} then if you only want to retrieve company A you must use "Apple AND Inc AND Los AND Alamos"}, otherwise it will also retrieve company B. However if you use AND for