Re: Substring in filter query...

On Wed, May 27, 2009 at 6:07 AM, escher2k wrote: > > Example: > Root cat -> Digital -> Watch -> "Timex Essentials" > Root cat -> Men -> Accessories -> "Timex Essentials" > > In the example above, we essentially have a product that is a part of two > separate hierarchies. > In the use case that I

Re: issues with shards

You are using the binary response writer (the request params have wt=javabin). Somehow the binary response writer is being passed a character stream, hence the error. Off hand, I'm not sure how that can happen. Just remove wt=javabin from the request parameters and try again. Are you using your o

Re: commit question

Hey Ashish, If commit fails, the documents won't be indexed! You can look at your index by pointing luke to your data folder (a Solr index is a Lucene index) or hit: http://host:port/solr/admin/luke/ to get an xml reply of what your index looks like. You can commit

Re: commit question

Hi, Any idea if documents from solr server are cleared even if commit fails or I can still again try commit after some time?? Thanks, Ashish Ashish P wrote: > > If I add 10 document to solrServer as in solrServer.addIndex(docs) ( Using > Embedded ) and then I commit and commit fails for for som

Re: Solr query performance issue

Aha, then no fq for you. :) Those two queries you wrote should be equivalent under the hood.   Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Larry He > To: solr-user@lucene.apache.org > Sent: Tuesday, May 26, 2009 5:26:34 PM > Subject: Re

Re: Substring in filter query (fq)...

Hi, It looks like you don't have about the hierarchy when doing your queries.  That makes things simpler, and you already have the answer, more or less: ?q=name:Timex&fq=category:Digital Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: e

Re: Recover crashed solr index

Hi Mike, The index is autoCommit every 1000 docs. I set this to increase the indexing speed. What is the best configuration do you suggest for each commit cycle? Thank you very much for your help. Following is the original exception: java.lang.RuntimeException: java.io.FileNotFoundException: /s

Re: [Solr Wiki] Update of "FrontPage" by OscarBernal

Oscar - are you on either of these lists? The front page does not seem like an appropriate place to list a blog entry. And the technique used in that blog entry doesn't seem like a best practice to me anyway, though I'd be curious to hear it debated a bit on solr-user on ways to offer sug

Substring in filter query (fq)...

I have a requirement where I have to search a product catalog and filter on a substring. I am not sure if this is supported. Any help is appreciated. I have included the use case below. Example: Root cat -> Digital -> Watch -> "Timex Essentials" Root cat -> Men -> Accessories -> "Timex Essent

Substring in filter query...

I have a requirement where I have to search a product catalog and filter on a substring. I am not sure if this is supported. Any help is appreciated. I have included the use case below. Example: Root cat -> Digital -> Watch -> "Timex Essentials" Root cat -> Men -> Accessories -> "Timex Essentials

Re: How to deal with hyphens in PDF documents?

Otis Gospodnetic wrote on 05/26/2009 11:06 PM: Hello, You really want to fix this before indexing, so you don't index garbage. One way to fix this is to make use of dictionaries while looking at two tokens at a time (current + next). Then you might see that neither "fo" or "cus" are in the di

Re: issues with shards

regarding the error, there is one webapp=null and path=null in the requests: INFO: webapp=null path=null > params={q=static+ firstSearcher+warming+query+from+solrconfig.xml} hits=0 status=0 QTime=63 , can it be the root of the error? 2009/5/26 Valdir Salgueiro > "WARNING: Could not getStat

Re: Solr query performance issue

Another little optimization would be to flatten the query. Instead of "field1:(02 04 05) field2:(01 02 03)" use "field1:02 field1:04 field1:05 field2:01 field2:02 field2:03" But I'd try and narrow down what queries are taking a long time, and see if there is a common element that could be optimize

Re: Solr query performance issue

We actually want OR operator on those values. Filters can only do AND, right? Is it better performance to have the query as field1:01 field1:02 field1:03 instead of field1:(01 02 03)? BR, Larry On Tue, May 26, 2009 at 5:15 PM, Otis Gospodnetic < otis_gospodne...@yahoo.com> wrote: > > What abo

Re: Solr query performance issue

What about field1:01 . field:100 being used as separate filters (that would then get ANDed) -- doable? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Development Team > To: solr-user@lucene.apache.org; yo...@lucidimagination.com > S

Re: solr machine freeze up during first replication after optimization

Not only does the whole index have to be rsynced, everything that the FS cached from the index can now be thrown away and the index has to slowly get cached anew. Even if you warm things up with good queries. No machine will like that. Even if this doesn't kill the machine (it shouldn't!) it

Re: Solr distributed, questions about upgrading

Valdir, Yes, although I can't give you exact pointers about what changed since 1.1 (2006?), so my suggestion is that you take the example solrconfig/schema and inject your settings in it. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From:

Re: How to deal with hyphens in PDF documents?

Hello, You really want to fix this before indexing, so you don't index garbage. One way to fix this is to make use of dictionaries while looking at two tokens at a time (current + next). Then you might see that neither "fo" or "cus" are in the dictionary, but that "focus" is, so you might co

Re: Solr query performance issue

Yes, those terms are important in calculating the relevancy scores so they are not in the filter queries. I was hoping if I can cache everything about a field, any combinations on the field values will be read from cache. Then it does not matter if I query for field1:(02 04 05), or field1:(01 02)

Re: Index size concerns

Thank you Otis, I will for sure check on this wa salaam, Muhammed Sameer --- On Tue, 5/26/09, Otis Gospodnetic wrote: > From: Otis Gospodnetic > Subject: Re: Index size concerns > To: solr-user@lucene.apache.org > Date: Tuesday, May 26, 2009, 1:01 PM > > Muhammed, > > It sounds like you are

How to deal with hyphens in PDF documents?

Good day, fellow solr users, Fair warning: - I am utterly new to solr and to this mailing list (and to lucene for that matter). I have been playing with solr for about two weeks. Goal: - I would like to index several thousand OCR'd newspaper articles, stored as PDF documents. I

RE: Solr statistics of top searches and results returned

Hi, In our specific implementation this is not really an issue, but I can imagine it could impact performance. I guess a new thread could spawned, which takes care of any performance issues, thanks for pointing it out. I'll post a message when I coded the change. Regards, Patrick -Origi

Re: Multicore Solr not returning expects results from search

I too am using 1.3. They way you specified shards is correct. For instance, I normally make the request to core0, and in the shards parameter, I put the addresses of both core0 and core1. I am using Tomcat though, so that may be different... Is there anything in the logs that strikes you as odd

Re: Multicore Solr not returning expects results from search

Thanks for the reply ahammad, that helps. Are you specifying them both in a URL, or in the localhost:8983/solr/core0,localhost:8983/solr/core1 like I have? I should add that I now have two indices that have different data in them. That is to say the ids are unique across both shards and I am sti

Re: Solr query performance issue

On Tue, May 26, 2009 at 3:42 PM, Larry He wrote: > We have about 100 different fields and 1 million documents we indexed with > Solr.  Many of the fields are multi-valued, and some are numbers (for range > search).  We are expecting to perform solr queries contains over 30 terms > and often the re

Solr query performance issue

Hi All, We have about 100 different fields and 1 million documents we indexed with Solr. Many of the fields are multi-valued, and some are numbers (for range search). We are expecting to perform solr queries contains over 30 terms and often the response time is well over a second. I found that

Solr distributed, questions about upgrading

hi, i have an application, and im wanting to port from solr 1.1 to 1.3 so i can have distribution (various shards), do i have to change something on solrconfig.xml or schema.xml?

Re: Multicore Solr not returning expects results from search

I have a multicore setup as well, and when I query something, I do it through core0, then specify both core0 and core1 ins the "shards" parameter. However, I don't have identical indicies. The results I get back are basically and addition of both cores' results. Good luck, please reply to this m

Re: Multicore Solr not returning expects results from search

I am still trying to figure this out... I am thinking maybe I have the shards setup wrong? If I have core0 and core1 with indices, and then I run the query on core0, specifying shards of core0 and core1. Is this how I should be doing it? Or should I have another core just to specify the other shar

Re: solr machine freeze up during first replication after optimization

Thanks for the suggestion, Otis. At this point, we are not sure what the real cause is . We have more than one master-slave groups. Every day the first replication after the optimization causes a random slave machine to freeze; the very same slave succeeded previous replications and would succee

Re: Recover crashed solr index

It sounds like you had never committed while building the original index? Unfortunately, it's not easy to recover an index in that state. It's best to periodically commit if you're building such a large index. Do you have the original exception you hit? I'll fix CheckIndex to be more sane if it

Re: Is it memory leaking in solr?

I think you are describing JVM memory usage you are observing from the outside (e.g. by looking at the top). You can't really tell what's going on inside the JVM like that, though. Like Linux makes use of any available RAM for FS buffering, the JVM tends to make use of the available heap for

Re: Indexing from DB connection issue

I have an update: I played around with it some more and it seems like it's being caused by the ClobTransformer. If I remove the 'clob="true"' from the field part in the data-config, it works fine. The Solr install is a multicore one. I placed the apache-solr-dataimporthandler-1.4-dev.jar from th

Re: grouping response docs together

Matt - what you are describing is about search-time grouping a la SOLR-236. Deduplication stuff I mention is for index-time near-duplicate detection. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Matt Mitchell > To: solr-user@lucene.apa

Re: issues with shards

"WARNING: Could not getStatistics on info bean null"? well i have no idea about that one :( 2009/5/26 Grant Ingersoll > It looks like one of the components failed loading up (see the log during > Solr init) What's that about? > > -Grant > > > On May 25, 2009, at 4:13 PM, Valdir Salgueiro wrote

Document Boosts don't seem to be having an effect

(gentle nudge - feel free to ask me to ask the question differently) Greetings - first post here - hoping someone can direct me - grasping at straws. thank you in advance. Jodi I'm trying to tune the sort order using a combination of document and query time boosts. When searching for the

PNW Hadoop + Apache Cloud Stack Meetup, Wed. May 27th:

Greetings, This is a friendly reminder that the 1st meetup for the PNW Hadoop + Apache Cloud Stack User Group is THIS WEDNESDAY at 6:45pm. We're very excited to have everyone attend! University of Washington, Allen Center Room 303, at 6:45pm on Wednesday, May 27, 2009. I'm going to put together a

Re: grouping response docs together

Thanks Otis. I'll give the dedup a test drive today. I'll explain what I'm trying to do a little better though because I don't think I have yet! So, I'm indexing an XML file. There are different "sections" in the XML file. Each of those sections gets a solr doc (the xml text-only is indexed). Eac

Re: Is it memory leaking in solr?

Hi Otis, The slowness was due to the JVM memory limit set by tomcat.. I have solved this problem. Initially I thought there might be memory leaking because I noticed the following behavior: In the peak of indexing, almost all 4GB of memory was used. Once indexing is done, the memory usage was

Re: recommendation for document store to use alongside Solr?

Otis- Thanks much. This is v. useful! --peter On Tue, May 26, 2009 at 12:16 PM, Otis Gospodnetic wrote: > > Peter, > If you need to sell and selling fewer changes is more likely to succeed, then > you could always propose: > - Offload SQL-based full-text searcher to Solr -- it's going to be f

Re: recommendation for document store to use alongside Solr?

Peter, If you need to sell and selling fewer changes is more likely to succeed, then you could always propose: - Offload SQL-based full-text searcher to Solr -- it's going to be faster, more flexible, require less hardware, etc. - This reduces load on the DB servers, which you can now use as sim

Re: Distributed Architecture

4 big indices if those servers can handle them. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: "Woytowitz, Matthew" > To: solr-user@lucene.apache.org > Sent: Tuesday, May 26, 2009 1:01:48 PM > Subject: Distributed Architecture > > I hav

Re: Is it memory leaking in solr?

Jianbin, If you connect to that Java process with jconsole, do you see a lot of garbage collection activity? What makes you think there is a memory leak? The slowness? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Jianbin Dai > To: s

Distributed Architecture

I have 4 servers each with 8 cores and 32 gigs of ram with 2TB of SAN space for each server. We want to distribute the index across the 4 servers using shards. What would be better: 4 big indexes. One on each server. 4 total shards, one per server. X number of smaller indexes, X on

Re: recommendation for document store to use alongside Solr?

Thanks Otis (for this and your recs on Twitter!). I'm just figuring this stuff out, nice to get a better perspective. Our application (a large digital repository at UT Austin) is, up until now, is all PHP+PostgreSQL. Creating another dependency (like Solr) means a bit of a sell on my part (convin

Re: Index size concerns

Muhammed, It sounds like you are talking about the ratio of original data size vs. index size. The exact ratio depends on things such as: - whether you store fields or just index them - whether you compress fields if you store them - whether you have term vectors enabled or not - analyzers and

Re: grouping response docs together

Matt, The Deduplication feature in Solr does support near-duplicate scenario. It comes with a few components to help you detect near-duplicates, and you should be able to write a custom near-dupe detection component and plug it in. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr

Re: Indexing from DB connection issue

Hello Erik, Yes, the drivers are there. I forgot to mention, this DB indexing was working before on the server when the DB was using a different schema. The schema has changed, so I did all my testing on my local machine. When I saw that it worked fine, I put in the new connection string/user/p

Re: recommendation for document store to use alongside Solr?

Hi Peter, Yeah, familiar recommendations - http://twitter.com/otisg/status/1907773452 I've used BDB for this type of stuff in the past and was pleased with BDBs performance, ability to scale, and the simplicity of the API. I would not use Solr itself for this. This one is from January: http:

Re: term vectors

Aponte - I'm not quite understanding your question. Could you provide some detailed examples of what you're trying to accomplish? Just guessing from your post, it seems what you are after is flattening your structure so that it fits within Solr/Lucene's document & field capabilities and th

Re: Shuffling results

On Mon, May 25, 2009 at 11:29 PM, yaymicro_bjorn wrote: > No, as I was trying to explain, I obviously don't want a totally random > result. I just want to mix it up a "little". Is there a way to achieve this > with solr? It doesn't exist yet, but adding a random function would allow you to combin

Re: Solrits - Problem with Index...

in witch Ducument i must remove the defType parameter.. i found "defType"only in solrConfig.xml. but i found two Configs.. one in "apache-solr-nightly\example\solr\conf". and one in "apache-solr-nightly\contrib\velocity\src\main\solr\conf" i only remove "defType=dismax" ? in with of the Conf?

Re: Solrits - Problem with Index...

On May 26, 2009, at 10:44 AM, Jörg Agatz wrote: in witch Ducument i must remove the defType parameter.. i found "defType"only in solrConfig.xml. but i found two Configs.. one in "apache-solr-nightly\example\solr\conf". and one in "apache-solr-nightly\contrib\velocity\src\main\solr\conf" i only

Re: Solrits - Problem with Index...

On May 26, 2009, at 10:13 AM, Jörg Agatz wrote: ok, now i understand.. the phraser fram solrtis, or the phraser, that solrtis is using... do not work with: (fieldname:term) To be precise... "parser". but how i can change the default search field from solrtis? and how can i search in a specia

Indexing from DB connection issue

Hello all, I am tyring to index directly from an Oracle DB. This is what appears in the stack trace: SEVERE: Full Import failed org.apache.solr.handler.dataimport.DataImportHandlerException: Unable to execute query: select * from ARTICLE Processing Document # 1 at org.apache.solr.handler

Re: recommendation for document store to use alongside Solr?

Depends on how large/many are your documents. Using Solr/Lucene: you can store the documents in the same index as the search index, or you can have "key" in the search index which is a look-up in a second index, which has two fields, the key (indexed) and the document(stored, maybe compressed too).

Re: recommendation for document store to use alongside Solr?

It's not distributed, but I like Tokyo Cabinet. Tokyo Tyrant adds http on top, I've never used Tyrant though. -Steve On Tue, May 26, 2009 at 7:50 AM, Peter Keane wrote: > Hi All- > > I've just recently began playing with Apache Solr, and it seems to be > a perfect fit for our project (http://co

Re: Indexing from DB connection issue

Did you move the Oracle JDBC driver to the other machine also? Erik On May 26, 2009, at 11:37 AM, ahammad wrote: Hello all, I am tyring to index directly from an Oracle DB. This is what appears in the stack trace: SEVERE: Full Import failed org.apache.solr.handler.dataimport.Data

RE: Solr statistics of top searches and results returned

If this is is not done in an async way wouldn't this have a serious performance impact? Plaatje, Patrick wrote: > > Hi all, > > I created a script that uses a Solr Search Component, which hooks into the > main solr core and catches the searches being done. After this it > tokenizes the sear

recommendation for document store to use alongside Solr?

Hi All- I've just recently began playing with Apache Solr, and it seems to be a perfect fit for our project (http://code.google.com/p/dase/). I've been quite surprised at both how easy Solr was to get up and running and how flexible it seems to be. I've been tempted to use it for not just search,

Re: Plugin Not Found

That is exactly what we were missing. As soon as we added the sharedlib in solr.xml it started working. Thank you very much for all the help on this one. It is greatly appreciated. -- Jeff Newburn Software Engineer, Zappos.com jnewb...@zappos.com - 702-943-7562 > From: Mark Miller > Reply-To

Re: Custom sort

The custom Field type similar to RandomSortField is definitely the way to go over a custom component since it's much less invasive - then everything else like distributed search should work. But see SOLR-... I'm in the process of converting Solr to use the new FieldComparatorSource. -Yonik ht

Re: grouping response docs together

Thomas, Here's what I get after patching nightly build (yesterday) and running "ant test" compileTests: [javac] Compiling 1 source file to /Users/mwm4n/Downloads/apache-solr-nightly/build/tests [javac] /Users/mwm4n/Downloads/apache-solr-nightly/src/test/org/apache/solr/search/TestDocSet.j

Re: Plugin Not Found

Mark Miller wrote: Jeff Newburn wrote: Whats your setup there? It looks like everything is getting launched twice. How many cores are you using? You have file:/home/zetasolr/lib/FacetCubeComponent.jar, but then /home/zetasolr/cores/zeta-main/ indicates you are actually trying to use the com

Re: Plugin Not Found

Mark, Overall we are using 3 cores. For the system we are trying to get the plugin working I have disabled all but the one core. If we put the jar in the specific core directory then it would need to be put in each core lib directory or the core will never load. That seems odd to me as well. I

Re: Solrits - Problem with Index...

ok, now i understand.. the phraser fram solrtis, or the phraser, that solrtis is using... do not work with: (fieldname:term) but how i can change the default search field from solrtis? and how can i search in a special field? Change /itas to use "lucene" (or remove the defType parameter altogethe

Re: Plugin Not Found

Jeff Newburn wrote: Whats your setup there? It looks like everything is getting launched twice. How many cores are you using? You have file:/home/zetasolr/lib/FacetCubeComponent.jar, but then /home/zetasolr/cores/zeta-main/ indicates you are actually trying to use the component from that cor

Re: Solrits - Problem with Index...

On May 26, 2009, at 9:59 AM, Jörg Agatz wrote: Sounds like it's just purely working out the basics of what field(s) you want to search on, and indexing them properly. Erik coud bee, but when i use the admin page to search.. it works... I search in the field "summary" the word "p

Re: Solrits - Problem with facing after restart yetty

> >> >> >> > You must be using trunk from last week? A bug got in, but if you grab the > latest, its fixed. > > - Mark > > I use a Nightly Build from 24.05, there is the bug dosent fixet? is it fixed in th enext nightly? i cant use Trunk, because, i cant use SVN behind the Proxy, so i musst use th

Re: Plugin Not Found

We are using solr 1.4 on trunk as of 5/7/2009. What patch did you want us to apply? -- Jeff Newburn Software Engineer, Zappos.com jnewb...@zappos.com - 702-943-7562 > From: Noble Paul നോബിള്‍ नोब्ळ् > Reply-To: > Date: Mon, 25 May 2009 17:16:19 +0530 > To: > Subject: Re: Plugin Not Found

Re: Solrits - Problem with Index...

> > > Sounds like it's just purely working out the basics of what field(s) you > want to search on, and indexing them properly. > >Erik > > coud bee, but when i use the admin page to search.. it works... I search in the field "summary" the word "plesnik" so i doo "*summary:plesnik*" in th

Re: Solrits - Problem with facing after restart yetty

Jörg Agatz wrote: Hi.. I tryed the Original stepp by stepp way from solritas, start indexing, ect.. after this i can use the "cat" links in the left sithe of the webfrontend.. Like "*cat:"electronics"*" but when i shutdown the Server, "*strg+C*" in the shell, and restart it with: "*java -Dsolr.

Solrits - Problem with facing after restart yetty

Hi.. I tryed the Original stepp by stepp way from solritas, start indexing, ect.. after this i can use the "cat" links in the left sithe of the webfrontend.. Like "*cat:"electronics"*" but when i shutdown the Server, "*strg+C*" in the shell, and restart it with: "*java -Dsolr.solr.home=../contrib

Re: Solrits - Problem with Index...

On May 26, 2009, at 9:39 AM, Jörg Agatz wrote: fields in index: extension, title, url, last-modified, size, some oter... 136 Documents in the index when i do this: http://192.168.105.54:8983/solr/itas/?q=summary%3Aplesnik&version=2.2&start=0&rows=10&indent=on&wt=velocity&v.template=browse I

Re: Solrits - Problem with Index...

Fist the Facts... Ubuntu 8.10 2GB RAM Nightlybuild from 24.05.09 fields in index: extension, title, url, last-modified, size, some oter... 136 Documents in the index when i do this: http://192.168.105.54:8983/solr/itas/?q=summary%3Aplesnik&version=2.2&start=0&rows=10&indent=on&wt=velocity&v.tem

Custom sort

Hey there, I want to implement a custom sort coding my own SortComparatorSource. The sorter would sort the results doing some calculations of some document attributes witch I have loaded previously from a DB at the solr core init. I mean, I have a preloaded hash with some attributes per doc id an

Re: Solrits - Problem with Index...

On May 26, 2009, at 5:11 AM, Jörg Agatz wrote: when i go to the Admin Page i can send a Search and get a result... Like tis: URL: http://192.168.105.56:8983/solr/select/?q=summary%3Aplesnik&version=2.2&start=0&rows=10&indent=on Result: It is only one example, the hole relsults to long :-) But

Re: clustering unit test failures on ubuntu

Sorry about that - wrong list. Auto complete foiled me again. - Mark

clustering unit test failures on ubuntu

I'm on the latest Ubuntu and since last week the clustering tests have been failing for me. Not sure why. [junit] Running org.apache.solr.handler.clustering.ClusteringComponentTest [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec [junit] Test org.apache.solr.handler.clus

Re: Lucene Query to Solr query

That is not what I'm looking for because my Lucene query is very complicated and contains many Query sub classes. org.apache.lucene.search.Query luceneQuery = buildLuceneQuery(); // contains many types BooleanQuery, TermQuery etc SolrServer server = new CommonsHttpSolrServer("http://example.c

Re: DIH uses == instead of = in SQL

Argh... I learned a lesson (yet again!)... I spent an hour setting up detailed logging, digging around in lots of DIH source, with no real luck finding the offending == versus =. Mentioned my frustration to a colleague and he pointed out right where I had checked multiple times that I

Re: issues with shards

It looks like one of the components failed loading up (see the log during Solr init) What's that about? -Grant On May 25, 2009, at 4:13 PM, Valdir Salgueiro wrote: hello, im using solr 1.3, and having some problems when i search with the shards parameters, for example: *shards=localhost:

term vectors

Hello!! I´m working with solr, indexing de content of xml files from a digital library. I have the structure of this xml files save in a tree with an optimal form , and in the leave i need to have the reference of the field save in solr. I do this because when a i got the leaves i have do some

Solrits - Problem with Index...

Hallo... I create with a croler a Index, it is not a original Solr index, but a "Lucene" so it works... when i go to the Admin Page i can send a Search and get a result... Like tis: URL: http://192.168.105.56:8983/solr/select/?q=summary%3Aplesnik&version=2.2&start=0&rows=10&indent=on Result: *