Re: strange performance issue with many shards on one server
Frederik Kraus, il 28/09/2011 23:16, ha scritto: Yep, I'm not getting more than 50-60% CPU during those load tests. I would try reducing the number of shards. A part from the memory discussion, this really seems to me a concurrency issue: too many threads waiting for other threads to complete, too many context switches... recently, on a lots-of-cores database server, we INCREASED speed by REDUCING the number of cores/threads each query was allowed to use (making sense of our customer investment) maybe you can get a similar effect by reducing the number of pieces your distributed search has to merge my 2 eurocents federico
Re: strange performance issue with many shards on one server
Jaeger, Jay - DOT, il 28/09/2011 18:40, ha scritto: That would still show up as the CPU being busy. i don't know how the program (top, htop, whatever) displays the value but when the cpu has a cache miss definitely that thread sits and waits for a number of clock cycles with 130GB of ram (per server?) I suspect caches miss as a rule just a suspicion however, nothing I'll bet on
Re: strange performance issue with many shards on one server
Frederik Kraus, il 28/09/2011 12:58, ha scritto: Hi, I am experiencing a strange issue doing some load tests. Our setup: just because I've listened to JUG mates talking about that at the last meeting, could it be that your CPUs are spending their time getting things from RAM to CPU cache? maybe that, say, 10% CPU power is spent on the bus federico
Re: Lucene->SOLR transition
Hi Scott Smith, il 16/09/2011 02:30, ha scritto: I've been using lucene for a number of years. We've now decided to move to SOLR. I have a couple of questions. 1. I'm used to creating Boolean queries, filter queries, term queries, etc. for lucene. Am I right in thinking that for SOLR my only option is creating string queries (with q and fq components) for solrj? 2. Assuming that the answer to 1 is "correct", then is there an easy way to take a lucene query (with nested Boolean queries, filter queries, etc.) and generate a SOLR query string with q and fq components? one other option is to build your own SearchComponent and handle additional parameters best regards federico
Re: Solr custom plugins: is it possible to have them persistent?
a brief question: is using an IoC framework like spring an option for you? if so, maybe this could help http://lucene.472066.n3.nabble.com/dependency-injection-in-solr-td3292685.html#a3295939
Re: Solr custom plugins: is it possible to have them persistent?
samuele.mattiuzzo, il 31/08/2011 18:22, ha scritto: SEVERE: org.apache.solr.common.SolrException: Error Instantiating UpdateRequestProcessorFactory, ToTheGoCustom is not a org.apache.solr.update.processor.UpdateRequestProcessorFactory btw you can't load classes in the default package from classes in a package: put your class in a package federico
Re: dependency injection in solr
Tomás Fernández Löbbe, il 29/08/2011 20:32, ha scritto:
You can use reflection to instantiate the correct object (specify the class
name on the parameter on the solrconfig and then invoke the constructor via
reflection). You'll have to manage the life-cycle of your object yourself.
If I understand your requirement, you probably have created a
SearchComponent that uses that "retriever", right?
sorry for the delay: I was experimenting.
Raw reflections do not suffice: you can't specify a dependency required
by your reflection-constructed object
I've ended up using spring this way (will cut some code for brevity)
I've enabled spring the usual way, adding a ContextLoaderListener in
web.xml and configuring the spring xml or java configuration files (I do
java configuration)
I've declared a spring bean named myComponentDeclaredInTheSpringConf,
that is an extension of SearchComponent, with it's collaborators
I've created SpringAwareSearchComponent, that is a delegate of
SearchComponent
public SpringAwareSearchComponent() {
this.ctx = ContextLoader.getCurrentWebApplicationContext();
}
...
public void init(NamedList args) {
super.init(args);
inner = ctx.getBean(args.get("__beanname__").toString(),
SearchComponent.class);
inner.init(args);
}
public void prepare(ResponseBuilder rb) throws IOException {
inner.prepare(rb);
}
public void process(ResponseBuilder rb) throws IOException {
inner.process(rb);
}
In solrconfig.xml I've declared the search component as
myComponentDeclaredInTheSpringConf
...other bean specific parameters
and added myComponent to the list of search components
And it works like a charm. Maybe I can implement some other solr class
delegate and add hooks between spring and solr as needed
any comment will be appreciated
best regards
federico
Re: How to send an OpenBitSet object from Solr server?
Satish Talim, il 30/08/2011 14:22, ha scritto:
But how to throw? As a stream of bits?
getBits() return a long[]
add a long[] part to your response
rb.rsp.add("long_array", obs.getBits())
federico
Re: How to send an OpenBitSet object from Solr server?
Satish Talim, il 30/08/2011 05:42, ha scritto: [...] Is there a work-around wherein I can send an OpenBitSet object? JavaBinCodec (used by default by solr) supports writing arrays. you can "getBits()" from openbitset and throw them into the binary response federico
Re: How to send an OpenBitSet object from Solr server?
Satish Talim, il 30/08/2011 05:42, ha scritto: [...] Is there a work-around wherein I can send an OpenBitSet object? JavaBinCodec (used by default by solr) supports writing arrays. you can "getBits()" from openbitset and throw them into the binary response federico
Re: dependency injection in solr
Tomás Fernández Löbbe, il 29/08/2011 17:58, ha scritto: I think I get it. Many of the objects that depend on the configuration are instantiated by using reflection, is that an option for you? yes it is what do you propose?
Re: dependency injection in solr
Tomás Fernández Löbbe, il 29/08/2011 17:58, ha scritto: I think I get it. Many of the objects that depend on the configuration are instantiated by using reflection, is that an option for you? yes it is what do you propose?
Re: dependency injection in solr
Tomás Fernández Löbbe, il 29/08/2011 16:39, ha scritto: You can do a lot of dependency injection though solrconfig.xml and schema.xml, Specify search components, update processors, filters, similarity, etc. Solr doesn't use any DI framework, everything is built-in in a pluggable manner. What kind of customizations do you need to apply? maybe we can point you better. for example, I have a classification component: it can retrieve classification data either from the file system or via a rest call. the "retriever" is a parameter of the component and we want to set it depending if we host the searcher or if the customer choose to install it locally i would like to inject a custom bean into the search component: at init time, it will just call the retrieve method and that will return the classification (ps: we are trying to migrate our existing system to solr) thanks in advance federico
dependency injection in solr
Hello everyone I need to hack solr by adding a couple custom search components. One small inconvenience is about configuring all the stuff. AFAIK solrconfig.xml is not a place where to do dependency injection, not yet at least. Have you ever had the need to use DI on a solr configuration? How have you managed it? Hard coding params in some delegate SearchComponent? Getting a reference of a spring application context via some static method? Any more elegant ways? thanks in advance federico
