Hi Avlesh,
hi Otis,
hi Grant,
hi all,
(enumerating to keep track of all the input)
a) mergeFactor 1000 too high
I'll change that back to 10. I thought it would make Lucene use more RAM
before starting IO.
b) ramBufferSize:
OK, or maybe more. I'll keep that in mind.
c) solrconfig.xml - defau
Philip,
I cannot answer your question, but I do have a question for you. Does
aggregation happen at the primary shard ? For eg : if I have three JVMs
JVM 1 : My application powered by Solr
JVM 2 : Shard 1
JVM 3 : Shard 2
I initialize my SolrServer like this
SolrServer _solrServer = *new* CommonsHt
I am trying to track memory utilization with my Application that uses Solr.
Details of the setup :
-3rd party Software : Solaris 10, Weblogic 10, jdk_150_14, Solr 1.3.0
- Hardware : 12 CPU, 24 GB RAM
For testing during PSR I am using a smaller subset of the actual data that I
want to work with. D
I guess a requesthandler may not be the right component to implement.
try implementing a CoreAdminHandler .
http://wiki.apache.org/solr/CoreAdmin#head-dfab3233db613c1976e05cdf894aa326ca64d312
On Mon, Aug 3, 2009 at 9:11 PM, James Brady wrote:
> Hi,
> I'm creating a custom request handler to retur
I understand the general need here. And just extending what you suggested
(indexing the fields themselves inside a multiValued field), you can perform
a query like this -
/search?q=myquery&facet=true&facet.field=indexedfields&facet.field=field1&facet.field=field2...&facet.sort=true
You'll get face
On Tue, Aug 4, 2009 at 12:16 AM, Mark Bennett wrote:
> Sow just make sure to use rows=1 ?
No, make sure that the query matches one document - rows (the number
of top docs returned) is irrelevant to faceting.
So q=id:some_doc
-Yonik
http://www.lucidimagination.com
>
> datasource.getData("update mytable "); //though the name is getData()
> it can execute update commands also
>
Even when the dataSource is "readOnly", Noble?
Cheers
Avlesh
2009/8/4 Noble Paul നോബിള് नोब्ळ्
> If your are writing a Transformer (or any other component) you can get
> hold o
If your are writing a Transformer (or any other component) you can get
hold of a dataSource instance .
datasource =Context#getDataSource(name).
//then you can invoke
datasource.getData("update mytable ");
//though the name is getData() it can execute update commands also
ensure that you do a
>
> You lost me.
>
Absolutely sorry about that Bill :(
How does boosting change the sort order?
What I really meant here is that if you have more than one "similarity"
fields in you MLT query, you can boost the results found due to one over the
other. It was not at all aimed to be an answer for s
Sow just make sure to use rows=1 ?
--
Mark Bennett / New Idea Engineering, Inc. / mbenn...@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
On Mon, Aug 3, 2009 at 5:51 PM, Yonik Seeley wrote:
> On Mon, Aug 3, 2009 at 8:26 PM, Mark Bennett wrote:
> > Yonik, can you conf
Couple of things -
1. Your dataSource is probably in readOnly mode. It is possible to fire
updates, by specifying readOnly="false" in your dataSource.
2. What you are trying achieve, is typically done using a "select for
update". For MySql, here's the documentation -
http://dev.mysq
>
> I'm sure I have the class name right - changing it to something patently
> incorrect results in the expected "org.apache.solr.common.SolrException:
> Error loading class ...", rather thanthe ClassCastException.
>
You are right about that, James.
Which Solr version are you using?
Can you please
Yes, daemontools or any kind of home-grown process-watching-and-restarting tool
will work.
Regarding those caches - they look too large.
Also, the ramBufferSizeMB is irrelevant on search slaves.
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Had
I checked the javadoc of Lucene's MoreLikeThis and MoreLikeThisQuery and
don't see any method regarding sorting at all. So my guess is we can't sort
the MoreLIkeThis result in Solr until this is supported in Lucene.
Bill
On Mon, Aug 3, 2009 at 5:54 PM, Bill Au wrote:
> Avlesh,
> You lost m
On Mon, Aug 3, 2009 at 8:26 PM, Mark Bennett wrote:
> Yonik, can you confirm reasoning below for 1.4 for a text field?
The bit about warming? Looks right to me - a big base docset can
trigger short-circuit logic in the enum faceting code... using a
docset of size 1 currently avoids this.
-Yonik
On Mon, Aug 3, 2009 at 8:45 PM, Stephen Green wrote:
> On Mon, Aug 3, 2009 at 8:38 PM, Stephen Green wrote:
>> On Mon, Aug 3, 2009 at 8:34 PM, Erik Hatcher
>> wrote:
>>> Is default-search-field stored (as specified in schema.xml)?
>>
>> Yep:
>>
>> > type="html" indexed="true" stored="tru
On Mon, Aug 3, 2009 at 8:38 PM, Stephen Green wrote:
> On Mon, Aug 3, 2009 at 8:34 PM, Erik Hatcher
> wrote:
>> Is default-search-field stored (as specified in schema.xml)?
>
> Yep:
>
> type="html" indexed="true" stored="true"
> termVectors="true" multiValued="true"/>
>
> While t
On Mon, Aug 3, 2009 at 8:34 PM, Erik Hatcher wrote:
> Is default-search-field stored (as specified in schema.xml)?
Yep:
While trying to figure this out, I went and did ant run-examples to
bring up the example in Jetty (I'm using Tomcat), and tried a couple
of queries in the resulting /solr/
Is default-search-field stored (as specified in schema.xml)?
Erik
On Aug 3, 2009, at 8:05 PM, Stephen Green wrote:
Hi, folks. I'm trying to get a very simple example working with Solr
highlighting. I have a default search field (called, unsurprisingly
"default-search-field") with te
Yonik, can you confirm reasoning below for 1.4 for a text field?
( Of course faceting is so much faster in 1.4 anyway, it's probably worth
the upgrade.
https://issues.apache.org/jira/browse/SOLR-475 )
A warning for folks NOT using 1.4:
At the bottom of this wiki page: (very bottom)
htt
KaktuChakarabati wrote:
Hey all,
I was wondering - when running Solr in a master/slaves setup using the 1.3
snap* scripts,
does the slaves' solrconfig.xml mergeFactor value make any difference? as
far as I would assume,
the mergeFactor specified in the master solrconfig.xml dictates the format
of
Hey all,
I was wondering - when running Solr in a master/slaves setup using the 1.3
snap* scripts,
does the slaves' solrconfig.xml mergeFactor value make any difference? as
far as I would assume,
the mergeFactor specified in the master solrconfig.xml dictates the format
of the index and then the s
Hi, folks. I'm trying to get a very simple example working with Solr
highlighting. I have a default search field (called, unsurprisingly
"default-search-field") with text in it and I want query terms to be
highlighted in that field when I do a search.
I'm using an up to date (as of this evening)
Hi all,
I just tried installing LocalSolr and LocalLucene - both the v2.0 tags
- in my Solr 1.3 instance, but when I start up I'm getting a
NoClassDefFoundError for
org.apache.solr.handler.component.SearchComponent. Grepping through
the source it appears that that class does exist in Solr 1.3, so
Thanks!
Any idea why
Miguel : three dimensions : [Exhibitio
parse to: miguel, three,dimensions, exhibitio
BUT
Miguel : three dimensions : [Exhibition]
parses to miguel, three, dimensions, null_1, exhibition
seems quite strange...
--peter
On Mon, Aug 3, 2009 at 4:02 PM, Andrzej Bialecki w
Ah! Looks like that'll work. Thanks Yonik!
For other folks listening in, he's suggesting not using Luke, and instead
reverting to a regular faceted query.
The full facet query URL would then be:
http://localhost:8983/solr/select?facet=true&facet.field=title&facet.limit=1000&q=state:CA
Vs. my a
On Mon, Aug 3, 2009 at 5:45 PM, Stephen Duncan
Jr wrote:
> What will cause me to exceed the limit and fail during faceting using the fc
> method is if the documents within a block of 65536 combine to take up too
> much space. And this (generally speaking) going to be a function on the
> average nu
Avlesh,
You lost me. How does boosting change the sort order? What about
sorting on a field that is not the mlt field?
Bill
On Mon, Aug 3, 2009 at 3:13 AM, Avlesh Singh wrote:
> You can boost the similarity field matches, if you want. Look for mlt.boost
> at http://wiki.apache.org/solr/M
On Mon, Aug 3, 2009 at 4:46 PM, Mohamed Parvez wrote:
> Hello,
>
>I have two fields.
>
>
>
>
> I have documnet, that has a value of ICS for BUS and SSE for ROLE
>
> *When I search for q=BUS:ics i get the result. but if i search for
> q=BUS:ics* i don't get any match*
>
> when I search f
Sounds like faceting?
q=state:CA&facet=true&facet.field=title&facet.limit=1000
-Yonik
http://www.lucidimagination.com
On Mon, Aug 3, 2009 at 5:39 PM, Mark Bennett wrote:
> You can get a nice list of terms for a field using the Luke handler:
> http://localhost:8983/solr/admin/luke?fl=title&num
On Mon, Aug 3, 2009 at 4:56 PM, Yonik Seeley wrote:
> On Mon, Aug 3, 2009 at 4:18 PM, Stephen Duncan
> Jr wrote:
> > On Mon, Aug 3, 2009 at 2:43 PM, Yonik Seeley >wrote:
> > Hmm, that's a hard thing to sell to the user and my boss, as it makes the
> > query time go from nearly always being sub-se
You can get a nice list of terms for a field using the Luke handler:
http://localhost:8983/solr/admin/luke?fl=title&numTerms=1000
But what I'd really like is to get the terms for the docs that match a
particular slice of the index.
For example, let's say I have records for all 50 states, but
Peter Keane wrote:
I've used Luke to figure out what is going on, and I see in the fields that
fail to match, a "null_1". Could someone tell me what that is? I see some
null_100s there as well, which see to separate field values. Clearly the
null_1s are causing the search to fail.
You used t
On Mon, Aug 3, 2009 at 4:18 PM, Stephen Duncan
Jr wrote:
> On Mon, Aug 3, 2009 at 2:43 PM, Yonik Seeley
> wrote:
> Hmm, that's a hard thing to sell to the user and my boss, as it makes the
> query time go from nearly always being sub-second (frequently less than 60
> ms), to ranging up to nearly 4
One task when designing a facet-based UI is deciding which fields to
facet on and display facets for. One possibility that I hope to
explore is to determine which fields to facet on dynamically, based on
the search results. In particular, I hypothesize that, for a somewhat
heterogeneous index (hete
I've used Luke to figure out what is going on, and I see in the fields that
fail to match, a "null_1". Could someone tell me what that is? I see some
null_100s there as well, which see to separate field values. Clearly the
null_1s are causing the search to fail.
--peter keane
On Mon, Aug 3, 20
Is it possible for the DataImportHandler to update records in the table it
is querying? For example, say I have a query like this in my entity:
query="select field1, field2, from someTable where hasBeenIndexed=false"
Is there a way I can mark each record processed by updating the
hasBeenIndexed f
On Mon, Aug 3, 2009 at 2:43 PM, Yonik Seeley wrote:
>
> >
> > I'm getting the following warning in my logs: 2009-08-03 13:41:40,114
> > [http-127.0.0.1-8080-1] WARN org.apache.solr.core.SolrCore - Approaching
> > too many values for UnInvertedField faceting on field 'originaltext' :
> > bucket si
Thanks all, I figured there would be more talk about daemontools if there were
really a need. I appreciate the input and for starters we'll put two slaves
behind a load balancer and grow it from there.
Lovin' Solr So Far! We were using alta vista as our search engine... it was
sooo 90's! hah
Greetings--
I am having a problem with a search that has stumped me. I have a field
(called "source") with the following value:
Miguel : three dimensions : [Exhibition]
when I perform this search:
source:"Miguel \: three dimensions \: \[Exhibition\]"
it fails to match.
Yet when I change the
On Mon, Aug 3, 2009 at 2:46 PM, vivek sar wrote:
> So, if I run only one sort query once in a day there would still be
> 4GB required at all time. Is there any way to tell Solr/Lucene to
> release the memory once the query has been run? Basically I don't want
> cache. I've commented out all the cac
How big are your documents? I haven't benchmarked DIH, so I am not
sure what to expect, but it does seem like something isn't right. Can
you fully describe how you are indexing? Have you done any profiling?
On Aug 3, 2009, at 12:32 PM, Chantal Ackermann wrote:
Hi all,
I'm still struggli
Hi,
Related question to "getting the latest records first". After trying
few suggested ways (function query, index time boosting) of getting
the latest first I settled for simple "sort" parameter,
sort=field+asc
As per wiki, http://wiki.apache.org/solr/SchemaDesign?highlight=(sort),
Lucen
On Mon, Aug 3, 2009 at 2:18 PM, Stephen Duncan
Jr wrote:
> On Fri, Jul 31, 2009 at 5:23 PM, Yonik Seeley
> wrote:
>
>> On Fri, Jul 31, 2009 at 5:06 PM, Stephen Duncan
>> Jr wrote:
>> > I have a couple more questions on the FieldValueCache. I see that the
>> > number of items in the cache is basic
On Fri, Jul 31, 2009 at 5:23 PM, Yonik Seeley wrote:
> On Fri, Jul 31, 2009 at 5:06 PM, Stephen Duncan
> Jr wrote:
> > I have a couple more questions on the FieldValueCache. I see that the
> > number of items in the cache is basically the number of multi-valued
> fields
> > facets have been reque
Hello!
Robert, from my experience with Solr (since 1.2 and running few 1.4
deployments) Solr does not need any kind of mechanism to ensure it will auto
start on crash, because I didn`t see it crash on it`s own fault. Just ensure,
You have not one instance of Solr, and run it behind a proxy or l
For 99.9%, run three copies behind a load balancer. That allows you to
take one down for upgrade, and still be fault-tolerant.
wunder
On Aug 3, 2009, at 10:46 AM, Robert Petersen wrote:
So then would the 'right' thing to do be to run it under something
like
Daemontools so it bounces back u
So then would the 'right' thing to do be to run it under something like
Daemontools so it bounces back up on a crash? Do any other people use
this approach or is there something better to make it come back up?
Speaking of overly large caches, if I have solr running on a machine
with 8GB main memo
Hi,
Thanks for your suggestions!
I'm sure I have the class name right - changing it to something patently
incorrect results in the expected
"org.apache.solr.common.SolrException: Error loading class ...", rather than
the ClassCastException.
I did have some problems getting my class on the app ser
>
> I have a Solr text field and when I use Solr's field analysis tool, it
> shows that wildcard queries are being stemmed. But query results indicate
> that it is not. It looks like there is a bug in the tool.
>
I am in agreement. Seems like a bug to me.
Cheers
Avlesh
On Mon, Aug 3, 2009 at 10
Hi,
I'd have to poke around the machine(s) to give you better guidance, but here is
some initial feedback:
- mergeFactor of 1000 seems crazy. mergeFactor is probably not your problem.
I'd go back to default of 10.
- 256 MB for ramBufferSizeMB sounds OK.
- pinging the DB won't tell you much ab
>
> avg-cpu: %user %nice%sys %iowait %idle
> 1.230.000.030.03 98.71
>
I agree, real bad statistics, actually.
Currently, I've set mergeFactor to 1000 and ramBufferSize to 256MB.
>
To me the former appears to be too high and latter too low (for your machine
configur
I have a Solr text field and when I use Solr's field analysis tool, it shows
that wildcard queries are being stemmed. But query results indicate that it
is not. It looks like there is a bug in the tool.
Bill
On Mon, Aug 3, 2009 at 7:21 AM, gateway0 wrote:
>
> Ok thanks you´re right.
>
> But t
Hi all,
I'm still struggling with the index performance. I've moved the indexer
to a different machine, now, which is faster and less occupied.
The new machine is a 64bit 8Gig-RAM RedHat. JDK1.6, Tomcat 6.0.18,
running with those settings (and others):
-server -Xms1G -Xmx7G
Currently, I've set
Can you cross check the class attribute for your handler in solrconfig.xml?
My guess is that it is specified as "solr.LiveCoresHandler". It should be
fully qualified class name - com.foo.path.to.LiveCoresHandler instead.
Moreover, I am damn sure that you did not forget to drop your jar into
solr.h
Hi,
I'm creating a custom request handler to return a list of live cores in
Solr.
On startup, I get this exception for each core:
Jul 31, 2009 5:20:39 PM org.apache.solr.common. SolrException log
SEVERE: java.lang.ClassCastException: LiveCoresHandler
at
org.apache.solr.core.RequestHandler
On Mon, Aug 3, 2009 at 4:45 AM, Nicolae Mihalache wrote:
> Hello,
>
> I'm using faceted search (perhaps in a dumb way) to collect some statistics
> for my index. I have documents in various languages, one of the field is
> "language" and I simply want to see how many documents I have for each
> lan
> Ok still not working with new field "text_two":
> "text:"Har*" text_two:"Har*""
> ==> result 0
>
> Schema Updates:
> "
> positionIncrementGap="100">
>
>
>
>
>
>
>
>
>
>
>
> multiValued="true"/>
>
>
> "
I'm pretty s
I much prefer "jstack" (which comes with the JDK) which is for exactly this
purpose. I always forget the right number for using kill. When using jstack,
the stack goes to the current terminal session, not Solr's output -- very
convenient.
~ David Smiley
__
Ok still not working with new field "text_two":
"text:"Har*" text_two:"Har*""
==> result 0
Schema Updates:
"
"
This is what you suggested, right?
kind regards, S.
Avlesh Singh wrote:
>
> The easiest thing
Ok still not working with new field "text_two":
"text:"Har*" text_two:"Har*""
==> result 0
Schema Updates:
"
"
This is what you suggested, right?
kind regards, S.
gateway0 wrote:
>
> Hi,
>
Thanks, Mark!
Mark Miller schrieb:
Its the pre-analyzed form thats copied. The field that its copied to will
determine the analyzer/filters for that field.
If you want to check out the code doing it, its
in org.apache.solr.update.DocumentBuilder
--
- Mark
http://www.lucidimagination.com
On M
Its the pre-analyzed form thats copied. The field that its copied to will
determine the analyzer/filters for that field.
If you want to check out the code doing it, its
in org.apache.solr.update.DocumentBuilder
--
- Mark
http://www.lucidimagination.com
On Mon, Aug 3, 2009 at 8:12 AM, Chantal Ac
Dear all,
before searching through the source code - maybe one of you can answer
this easily:
When and based on what are the tokenizer and filters applied when
copying fields? Can it happen that fields are analyzed twice (once when
creating the first field, and a second time when they are co
The easiest thing to do would be to create a new field in your schema which
only has a lowercasefilter applied to it. While searching perform searches
across the two fields. You'll get desired results.
You can use the directive in your schema.xml for copying data
from your original field into the
Ok thanks you´re right.
But the thing is my users will often search for expressions like: "Harr" or
"har" etc..
So I thought I automatically add the wildcard "*" to every request.
If that too gets me into trouble "Harr*"=>no result "harry*"=>no result
What should I do?
Otis Gospodnetic wro
Dear Mark,
Thank you very much for your suggestions. It works really great. I construct
a new plugin based on AnalyzingQueryParser with getPrefixQuery,
getRangeQuery, getWildcardQuery methods from Lucene. With this plugin we can
now search with leading wildcards, case sensitive and by analyzed wi
Thanks, I will try maybe together with the -XX:OnOutOfMemoryError="; ":
http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp
But now I know where the problem came from, so maybe next time.
On Mon, Aug 3, 2009 at 11:08 AM, Chantal Ackermann <
chantal.ackerm...@btelligent.de> wrote:
> ki
kill -3 [JVM-process-id] (SIGQUIT)
sends a signal to the JVM to dump all thread stacks. It does NOT kill
it. It's only for outputting the stacks. You can see whether there are
any threads that lock up.
This produces a LOT of output in the main logfile (e.g. catalina.out for
Tomcat).
(see for
Hi Otis,
thank you for your answer. I will have a look at the link.
It can happen that search queries include several languages, and I
thought at first, that creating a multilingual (sub-)index from a mix of
the existing core indexes could be a way. But that is not a good
solution, isn't it? I
Hello,
I'm using faceted search (perhaps in a dumb way) to collect some statistics
for my index. I have documents in various languages, one of the field is
"language" and I simply want to see how many documents I have for each
language. I have noticed that the search builds a int[maxDoc] array and
Hello,
Thanks for your answers and suggestions. I can get a heap dump also with the
jmap command. The resulting file is so big that jhat gets out of memory
errors itself when reading the dump.
I traced back my problem (using the heap dump analyser from yourkit.com) to
the FieldCache. In fact it h
You can boost the similarity field matches, if you want. Look for mlt.boost
at http://wiki.apache.org/solr/MoreLikeThis
Cheers
Avlesh
On Mon, Aug 3, 2009 at 11:33 AM, Renz Daluz wrote:
> Hi,
>
> I'm looking at changing the result order when searching by MLT. I tried the
> sort=, but it's not wo
73 matches
Mail list logo
