How difficult is adding a new aggregate function to Json Facet?

[Pablo Anzorena]

Hey,

it has been awhile since I'm using Solr, but I have never see the code. Now
I'm in the need to add some capability to Solr.

What I need to add is another aggregate function to Json Facet API, for
example "sum(field1)-sum(field2)". How hard do you think it would be? Also,
it would be great if you give me some advice on where to start.

Thanks!

Re: Solr dynamic "on the fly fields"

[Pablo Anzorena]

Thanks Erick for the answer. Function Queries are great, but for my use
case what I really do is making aggregations (using Json Facet for example)
with this functions.

I have tried using Function Queries with Json Facet but it does not support
it.

Any other idea you can imagine?





2017-07-03 21:57 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> I don't know how one would do this. But I would ask what the use-case
> is. Creating such fields at index time just seems like it would be
> inviting abuse by creating a zillion fields as you have no control
> over what gets created. I'm assuming your tenants don't talk to each
> other
>
> Have you thought about using function queries to pull this data out as
> needed at _query_ time? See:
> https://cwiki.apache.org/confluence/display/solr/Function+Queries
>
> Best,
> Erick
>
> On Mon, Jul 3, 2017 at 12:06 PM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> > Thanks Erick,
> >
> > For my use case it's not possible any of those solutions. I have a
> > multitenancy scheme in the most basic level, that is I have a single
> > collection with fields (clientId, field1, field2, ..., field50) attending
> > many clients.
> >
> > Clients can create custom fields based on arithmetic operations of any
> > other field.
> >
> > So, is it possible to update let's say field49 with the follow operation:
> > log(field39) + field25 on clientId=43?
> >
> > Do field39 and field25 need to be stored to accomplish this? Is there any
> > other way to avoid storing them?
> >
> > Thanks!
> >
> >
> > 2017-07-03 15:00 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:
> >
> >> There are two ways:
> >> 1> define a dynamic field pattern, i.e.
> >>
> >> 
> >>
> >> Now just add any field in the doc you want. If it ends in "_sum" and
> >> no other explicit field matches you have a new field.
> >>
> >> 2> Use the managed schema to add these on the fly. I don't recommend
> >> this from what I know of your use case, this is primarily intended for
> >> front-ends to be able to modify the schema and/or "field guessing".
> >>
> >> I do caution you though that either way don't go over-the-top. If
> >> you're thinking of thousands of different fields that can lead to
> >> performance issues.
> >>
> >> You can either put stuff in the field on your indexing client or
> >> create a custom update component, perhaps the simplest would be a
> >> "StatelessScriptUpdateProcessorFactory:
> >>
> >> see: https://cwiki.apache.org/confluence/display/solr/
> >> Update+Request+Processors#UpdateRequestProcessors-
> >> UpdateRequestProcessorFactories
> >>
> >> Best,
> >> Erick
> >>
> >> On Mon, Jul 3, 2017 at 10:52 AM, Pablo Anzorena <
> anzorena.f...@gmail.com>
> >> wrote:
> >> > Hey,
> >> >
> >> > I was wondering if there is some way to add fields "on the fly" based
> on
> >> > arithmetic operations on other fields. For example add a new field
> >> > "custom_field" = log(field1) + field2 -5.
> >> >
> >> > Thanks.
> >>
>

Re: Solr dynamic "on the fly fields"

[Pablo Anzorena]

Thanks Erick,

For my use case it's not possible any of those solutions. I have a
multitenancy scheme in the most basic level, that is I have a single
collection with fields (clientId, field1, field2, ..., field50) attending
many clients.

Clients can create custom fields based on arithmetic operations of any
other field.

So, is it possible to update let's say field49 with the follow operation:
log(field39) + field25 on clientId=43?

Do field39 and field25 need to be stored to accomplish this? Is there any
other way to avoid storing them?

Thanks!


2017-07-03 15:00 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> There are two ways:
> 1> define a dynamic field pattern, i.e.
>
> 
>
> Now just add any field in the doc you want. If it ends in "_sum" and
> no other explicit field matches you have a new field.
>
> 2> Use the managed schema to add these on the fly. I don't recommend
> this from what I know of your use case, this is primarily intended for
> front-ends to be able to modify the schema and/or "field guessing".
>
> I do caution you though that either way don't go over-the-top. If
> you're thinking of thousands of different fields that can lead to
> performance issues.
>
> You can either put stuff in the field on your indexing client or
> create a custom update component, perhaps the simplest would be a
> "StatelessScriptUpdateProcessorFactory:
>
> see: https://cwiki.apache.org/confluence/display/solr/
> Update+Request+Processors#UpdateRequestProcessors-
> UpdateRequestProcessorFactories
>
> Best,
> Erick
>
> On Mon, Jul 3, 2017 at 10:52 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> > Hey,
> >
> > I was wondering if there is some way to add fields "on the fly" based on
> > arithmetic operations on other fields. For example add a new field
> > "custom_field" = log(field1) + field2 -5.
> >
> > Thanks.
>

Solr dynamic "on the fly fields"

[Pablo Anzorena]

Hey,

I was wondering if there is some way to add fields "on the fly" based on
arithmetic operations on other fields. For example add a new field
"custom_field" = log(field1) + field2 -5.

Thanks.

Re: How to use Wordnet in solr?

[Pablo Anzorena]

Thanks to everybody.

I will try first Alessandro and Steve recommendation.

If i don't misunderstood, you are telling me that I have to customize the
prolog files to "solr txt synonyms syntax"? If that is correct, what is the
point of format:wordnet ?

2017-04-21 12:52 GMT-03:00 alessandro.benedetti :

> Hi Pablo,
> with wordnet format , Solr will just parse synonyms from a different file
> format [1] .
> The rest will work exactly the same.
> You will use a managed resource to load the file and then potentially
> update
> it.
> If you were thinking to use directly the online resource, you may need to
> customize it a bit.
>
> Cheers
>
> [1] http://wordnet.princeton.edu/man/prologdb.5WN.html
>
>
>
> -
> ---
> Alessandro Benedetti
> Search Consultant, R Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/How-to-use-Wordnet-in-solr-tp4331273p4331306.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

How to use Wordnet in solr?

[Pablo Anzorena]

Hey,

I'm planning to use Wordnet and I want to know how.

There's a class called *WordnetSynonymParser *, does anybody use it? It
says it is experimental...

I'm using solr 5.2.1

Briefly speaking about my needs:
I have different collections in different languages (fr, pr, sp, en).
When the user search for example in the english collection the word
"furnitures" I want to look for "table", "chair", "furniture"(without the
plural) and all the synonyms of "furnitures". Wordnet already provides me
all this and in different languages, that's why it would be great to have
solr using it.

Regards,
Pablo.

Re: Pagination bug? when sorting by a field (not unique field)

[Pablo Anzorena]

Excellent guns, thank you very much!

El mar. 29, 2017 18:09, "Erick Erickson"  escribió:

> You might be helped by "distributed IDF".
> see: SOLR-1632
>
> On Wed, Mar 29, 2017 at 1:56 PM, Chris Hostetter
>  wrote:
> >
> > The thing to keep in mind, is that w/o a fully deterministic sort,
> > the underlying problem statement "doc may appera on multiple pages" can
> > exist even in a single node solr index, even if no documents are
> > added/deleted between bage requests: because background merges /
> > searcher re-opening may happen in between those page requests.
> >
> > The best practice, if you really care about ensuring no (non-updated) doc
> > is ever returned twice in subsequent pages, is to to use a fully
> > deterministic sort, with a "tie breaker" clause that is unique to every
> > document (ie: uniqueKey field)
> >
> >
> >
> > : Date: Wed, 29 Mar 2017 23:14:22 +0300
> > : From: Mikhail Khludnev 
> > : Reply-To: solr-user@lucene.apache.org
> > : To: solr-user 
> > : Subject: Re: Pagination bug? when sorting by a field (not unique field)
> > :
> > : Great explanation, Alessandro!
> > :
> > : Let me briefly explain my experience. I have a tiny test with 2 shards
> and
> > : 2 replicas, index about a hundred of docs. And then when I fully
> paginate
> > : search results with score ranking, I've got duplicates across pages.
> And
> > : the reason is deletes, which occur probably due to update/failover.
> Every
> > : paging request lands to the different replica. There are a few
> workarounds:
> > : lands consequent requests to the same replicas; also  fixes
> > : duplicates; but tie-breaking is the best way for sure.
> > :
> > : On Wed, Mar 29, 2017 at 7:10 PM, alessandro.benedetti <
> a.benede...@sease.io>
> > : wrote:
> > :
> > : > The reason Mikhail mentioned that, is probably related to :
> > : >
> > : > *The way how number of document calculated is changed (LUCENE-6711)*
> > : > /The number of documents (docCount) is used to calculate term
> specificity
> > : > (idf) and average document length (avdl). Prior to LUCENE-6711,
> > : > collectionStats.maxDoc() was used for the statistics. Now,
> > : > collectionStats.docCount() is used whenever possible, if not
> maxDocs() is
> > : > used.
> > : > Assume that a collection contains 100 documents, and 50 of them have
> > : > "keywords" field. In this example, maxDocs is 100 while docCount is
> 50 for
> > : > the "keywords" field. The total number of tokens for "keywords"
> field is
> > : > divided by docCount to obtain avdl. Therefore, docCount which is the
> total
> > : > number of documents that have at least one term for the field, is a
> more
> > : > precise metric for optional fields.
> > : > DefaultSimilarity does not leverage avdl, so this change would have
> > : > relatively minor change in the result list. Because relative idf
> values of
> > : > terms will remain same. However, when combined with other factors
> such as
> > : > term frequency, relative ranking of documents could change. Some
> Similarity
> > : > implementations (such as the ones instantiated with NormalizationH2
> and
> > : > BM25) take account into avdl and would have notable change in ranked
> list.
> > : > Especially if you have a collection of documents with varying
> lengths.
> > : > Because NormalizationH2 tends to punish documents longer than avdl./
> > : >
> > : > This means that if you are load balancing, the page 2 query could go
> to
> > : > another replica, where the doc is scored differently, ending up on a
> > : > different position ( and maybe appearing again as a final effect).
> > : > This scenario is referred to scored ranking, so it will not affect
> sorting
> > : > (
> > : > and I believe in your initial mail you were referring not to sorting)
> > : >
> > : > Cheers
> > : >
> > : >
> > : > Pablo wrote
> > : > > Mikhall,
> > : > >
> > : > > effectively maxDocs are different and also deletedDocs, but
> numDocs are
> > : > > ok.
> > : > >
> > : > > I don't really get it, but can that be the problem?
> > : >
> > : >
> > : >
> > : >
> > : >
> > : > -
> > : > ---
> > : > Alessandro Benedetti
> > : > Search Consultant, R Software Engineer, Director
> > : > Sease Ltd. - www.sease.io
> > : > --
> > : > View this message in context: http://lucene.472066.n3.
> > : > nabble.com/Pagination-bug-when-sorting-by-a-field-not-unique-field-
> > : > tp4327408p4327461.html
> > : > Sent from the Solr - User mailing list archive at Nabble.com.
> > : >
> > :
> > :
> > :
> > : --
> > : Sincerely yours
> > : Mikhail Khludnev
> > :
> >
> > -Hoss
> > http://www.lucidworks.com/
>

Re: Pagination bug? when sorting by a field (not unique field)

[Pablo Anzorena]

Mikhall,

effectively maxDocs are different and also deletedDocs, but numDocs are ok.

I don't really get it, but can that be the problem?

2017-03-29 10:35 GMT-03:00 Mikhail Khludnev <m...@apache.org>:

> Can it happen that replicas are different by deleted docs? I mean numDocs
> is the same, but maxDocs is different by number of deleted docs, you can
> see it in solr admin at the core page.
>
> On Wed, Mar 29, 2017 at 4:16 PM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
>
> > Shawn,
> >
> > Yes, the field has duplicate values and yes, if I add the secondary sort
> by
> > the uniqueKey it solve the issue.
> >
> > Those 2 situations you mentioned are not occurring, none of them. The
> index
> > is replicated, but not sharded.
> >
> > Does solr sort by an internal id if no uniqueKey is present in the sort?
> >
> > 2017-03-29 9:58 GMT-03:00 Shawn Heisey <apa...@elyograg.org>:
> >
> > > On 3/29/2017 6:35 AM, Pablo Anzorena wrote:
> > > > I was paginating the results of a query and noticed that some
> > > > documents were repeated across pagination buckets of 100 rows. When I
> > > > sort by the unique field there is no repeated document but when I
> sort
> > > > by another field then repeated documents appear. I assume is a bug
> and
> > > > it's not the intended behaviour, right?
> > >
> > > There is a potential situation that can cause this problem that is NOT
> a
> > > bug.
> > >
> > > If the field you are sorting on contains duplicate values (same value
> in
> > > multiple documents), then I am pretty sure that the sort order of
> > > documents with the same value in the sort field is non-deterministic in
> > > these situations:
> > >
> > > 1) A distributed (sharded) index.
> > > 2) When the index contents can change between a request for one page
> and
> > > a request for the next page -- documents being added, deleted, or
> > changed.
> > >
> > > Because the sort order of documents with the same value can change, one
> > > document that may have ended up on the first page on the first query
> may
> > > end up on the second page on the second query.
> > >
> > > Sorting by a field with no duplicate values (the unique field you
> > > mentioned) will always result in the exact same sort order ... but if
> > > you add documents that sort to near the start of the sort order between
> > > queries, the behavior you have noticed can still happen.
> > >
> > > If this is what you are encountering, adding secondary sort on the
> > > uniqueKey field would probably clear up the problem.  If your uniqueKey
> > > field is "id", something like this:
> > >
> > > sort=someField desc,id desc
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: Pagination bug? when sorting by a field (not unique field)

[Pablo Anzorena]

Shawn,

Yes, the field has duplicate values and yes, if I add the secondary sort by
the uniqueKey it solve the issue.

Those 2 situations you mentioned are not occurring, none of them. The index
is replicated, but not sharded.

Does solr sort by an internal id if no uniqueKey is present in the sort?

2017-03-29 9:58 GMT-03:00 Shawn Heisey <apa...@elyograg.org>:

> On 3/29/2017 6:35 AM, Pablo Anzorena wrote:
> > I was paginating the results of a query and noticed that some
> > documents were repeated across pagination buckets of 100 rows. When I
> > sort by the unique field there is no repeated document but when I sort
> > by another field then repeated documents appear. I assume is a bug and
> > it's not the intended behaviour, right?
>
> There is a potential situation that can cause this problem that is NOT a
> bug.
>
> If the field you are sorting on contains duplicate values (same value in
> multiple documents), then I am pretty sure that the sort order of
> documents with the same value in the sort field is non-deterministic in
> these situations:
>
> 1) A distributed (sharded) index.
> 2) When the index contents can change between a request for one page and
> a request for the next page -- documents being added, deleted, or changed.
>
> Because the sort order of documents with the same value can change, one
> document that may have ended up on the first page on the first query may
> end up on the second page on the second query.
>
> Sorting by a field with no duplicate values (the unique field you
> mentioned) will always result in the exact same sort order ... but if
> you add documents that sort to near the start of the sort order between
> queries, the behavior you have noticed can still happen.
>
> If this is what you are encountering, adding secondary sort on the
> uniqueKey field would probably clear up the problem.  If your uniqueKey
> field is "id", something like this:
>
> sort=someField desc,id desc
>
> Thanks,
> Shawn
>
>

Re: Pagination bug? when sorting by a field (not unique field)

[Pablo Anzorena]

Let me try. It is really hard to replicate it, but I will try out and come
back when i got it.

2017-03-29 9:40 GMT-03:00 Erik Hatcher <erik.hatc...@gmail.com>:

> Certainly not intended behavior.  Can you show us a way to replicate the
> issue?
>
>
> > On Mar 29, 2017, at 8:35 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> >
> > Hey,
> >
> > I was paginating the results of a query and noticed that some documents
> > were repeated across pagination buckets of 100 rows.
> > When I sort by the unique field there is no repeated document but when I
> > sort by another field then repeated documents appear.
> > I assume is a bug and it's not the intended behaviour, right?
> >
> > Solr version:5.2.1
> >
> > Regards,
> > Pablo.
>
>

Pagination bug? when sorting by a field (not unique field)

[Pablo Anzorena]

Hey,

I was paginating the results of a query and noticed that some documents
were repeated across pagination buckets of 100 rows.
When I sort by the unique field there is no repeated document but when I
sort by another field then repeated documents appear.
I assume is a bug and it's not the intended behaviour, right?

Solr version:5.2.1

Regards,
Pablo.

Re: Configuring Solr for Maximum Concurrency

[Pablo Anzorena]

Dave,

there is something similar like MAX_CONNECTIONS and
MAX_CONNECTIONS_PER_HOST which control the number of connections.

Are you leaving open the connection to zookeeper after you establish it?
Are you using the singleton pattern?

2016-12-28 14:14 GMT-03:00 Dave Seltzer :

> Hi Erick,
>
> I'll dig in on these timeout settings and see how changes affect behavior.
>
> One interesting aspect is that we're not indexing any content at the
> moment. The rate of ingress is something like 10 to 20 documents per day.
>
> So my guess is that ZK simply is deciding that these servers are dead based
> on the fact that responses are so very sluggish.
>
> You've mentioned lots of timeouts, but are there any settings which control
> the number of available threads? Or is this something which is largely
> handled automagically?
>
> Many thanks!
>
> -Dave
>
> On Wed, Dec 28, 2016 at 11:56 AM, Erick Erickson 
> wrote:
>
> > Dave:
> >
> > There are at least 4 timeouts (not even including ZK) that can
> > be relevant, defined in solr.xml:
> > socketTimeout
> > connTimeout
> > distribUpdateConnTimeout
> > distribUpdateSoTimeout
> >
> > Plus the ZK timeout
> > zkClientTimeout
> >
> > Plus the ZK configurations.
> >
> > So it would help narrow down what's going on if we knew why the nodes
> > dropped out. There are indeed a lot of messages dumped, but somewhere
> > in the logs there should be a root cause.
> >
> > You might see Leader Initiated Recovery (LIR) which can indicate that
> > an update operation from the leader took too long, the timeouts above
> > can be adjusted in this case.
> >
> > You might see evidence that ZK couldn't get a response from Solr in
> > "too long" and decided it was gone.
> >
> > You might see...
> >
> > One thing I'd look at very closely is GC processing. One of the
> > culprits for this behavior I've seen is a very long GC stop-the-world
> > pause leading to ZK thinking the node is dead and tripping this chain.
> > Depending on the timeouts, "very long" might be a few seconds.
> >
> > Not entirely helpful, but until you pinpoint why the node goes into
> > recovery it's throwing darts at the wall. GC and log messages might
> > give some insight into the root cause.
> >
> > Best,
> > Erick
> >
> > On Wed, Dec 28, 2016 at 8:26 AM, Dave Seltzer 
> wrote:
> > > Hello Everyone,
> > >
> > > I'm working on a Solr Cloud cluster which is used in a hash matching
> > > application.
> > >
> > > For performance reasons we've opted to batch-execute hash matching
> > queries.
> > > This means that a single query will contain many nested queries. As you
> > > might expect, these queries take a while to execute. (On the order of 5
> > to
> > > 10 seconds.)
> > >
> > > I've noticed that Solr will act erratically when we send too many
> > > long-running queries. Specifically, heavily-loaded servers will
> > repeatedly
> > > fall out of the cluster and then recover. My theory is that there's
> some
> > > limit on the number of concurrent connections and that client queries
> are
> > > preventing zookeeper related queries... but I'm not sure. I've
> increased
> > > ZKClientTimeout to combat this.
> > >
> > > My question is: What configuration settings should I be looking at in
> > order
> > > to make sure I'm maximizing the ability of Solr to handle concurrent
> > > requests.
> > >
> > > Many thanks!
> > >
> > > -Dave
> >
>

Re: Posting files 405 http error

[Pablo Anzorena]

When I manually copy one collection to another, I copy the core.properties
from the source to the destination with the name core.properties.unloaded
so there is no problem.

So the steps I'm doing are:
1> index to my source collection.
2>Copy the directory of the source collection, excluding the
core.properties.
3>Copy the core.properties under the name of core.properties.unloaded to
the destination.
4>Create the collection in the destination.
5>Use the ADDREPLICA command to add replicas.

With these it throws the error.

They are very similar to those you mentioned, but instead you first create
the destination collection and then copy the data. The problem I face with
your approach is that unless I unload my collection, solr doesn't realize
there is data indexed.

2016-11-03 14:54 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> Wait. What were you doing originally? Just copying the entire
> SOLR_HOME over or something?
>
> Because one of the things each core carries along is a
> "core.properties" file that identifies
> 1> the name of the core, something like collection_shard1_replica1
> 2> the name of the collection the core belongs to
>
> So if you just copy a directory containing the core.properties file
> from one place to another _and_ they're pointing to the same Zookeeper
> then the behavior is undefined.
>
> And if you _don't_ point to the same zookeeper, your copied collection
> isn't registered with ZK so that's a weird state as well.
>
> If your goal is to move things from one collection to another, here's
> a possibility (assuming the nodes can all "see" each other).
>
> 1> index to your source collection
> 2> create a new destination collection
> 3a> use the "fetchindex" command to move the relevant indexes from the
> source to the destination, see
> https://cwiki.apache.org/confluence/display/solr/Index+
> Replication#IndexReplication-HTTPAPICommandsfortheReplicationHandler
> 3b> instead of <3a>, manually copy the data directory from the source
> to each replica.
> 4> In either 3a> or 3b>, it's probably easier to create a leader-only
> (replicationFactor=1) destination collection then use the ADDREPLICA
> command to add replicas, that way they'll all sync automatically.
>
> Best,
> Erick
>
> On Thu, Nov 3, 2016 at 10:28 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> > Thanks Shawn.
> >
> > Actually there is no load balancer or proxy in the middle, but even if
> > there was, how would you explain that I can index if a create a
> completely
> > new collection?
> >
> > I figured out how to fix it. What I'm doing is creating a new collection,
> > then unloading it (by unloading all the shards/replicas), then copy the
> > data directory from the collection in the other solrcloud, and finally
> > creating again the collection. It's not the best solution, but it works,
> > nevertheless I still would like to know what's causing the problem...
> >
> > It's worth to mention that I'm not using jetty, I'm using solr-undertow
> > https://github.com/kohesive/solr-undertow
> >
> > 2016-11-03 12:56 GMT-03:00 Shawn Heisey <apa...@elyograg.org>:
> >
> >> On 11/3/2016 9:10 AM, Pablo Anzorena wrote:
> >> > Thanks for the answer.
> >> >
> >> > I checked the log and it wasn't logging anything.
> >> >
> >> > The error i'm facing is way bizarre... I create a new fresh collection
> >> and
> >> > then index with no problem, but it keeps throwing this error if i copy
> >> the
> >> > collection from one solrcloud to the other and then index.
> >> >
> >> > Any clue on why is this happening?
> >>
> >> Solr's source code doesn't seem to even have a 405 error, so I bet
> >> what's happening is that you have Solr sitting behind a proxy or load
> >> balancer, and that server doesn't like the request you sent, so it
> >> rejects it and Solr never receives anything.
> >>
> >> Here's an excerpt of code from the SolrException class in the master
> >> branch:
> >>
> >>   /**
> >>* This list of valid HTTP Status error codes that Solr may return in
> >>* the case of a "Server Side" error.
> >>*
> >>* @since solr 1.2
> >>*/
> >>   public enum ErrorCode {
> >> BAD_REQUEST( 400 ),
> >> UNAUTHORIZED( 401 ),
> >> FORBIDDEN( 403 ),
> >> NOT_FOUND( 404 ),
> >> CONFLICT( 409 ),
> >> UNSUPPORTED_MEDIA_TYPE( 415 ),
> >> SERVER_ERROR( 500 ),
> >> SERVICE_UNAVAILABLE( 503 ),
> >> INVALID_STATE( 510 ),
> >> UNKNOWN(0);
> >> public final int code;
> >>
> >> private ErrorCode( int c )
> >> {
> >>   code = c;
> >> }
> >> public static ErrorCode getErrorCode(int c){
> >>   for (ErrorCode err : values()) {
> >> if(err.code == c) return err;
> >>   }
> >>   return UNKNOWN;
> >> }
> >>   };
> >>
> >> Thanks,
> >> Shawn
> >>
> >>
>

Re: Posting files 405 http error

[Pablo Anzorena]

Thanks Shawn.

Actually there is no load balancer or proxy in the middle, but even if
there was, how would you explain that I can index if a create a completely
new collection?

I figured out how to fix it. What I'm doing is creating a new collection,
then unloading it (by unloading all the shards/replicas), then copy the
data directory from the collection in the other solrcloud, and finally
creating again the collection. It's not the best solution, but it works,
nevertheless I still would like to know what's causing the problem...

It's worth to mention that I'm not using jetty, I'm using solr-undertow
https://github.com/kohesive/solr-undertow

2016-11-03 12:56 GMT-03:00 Shawn Heisey <apa...@elyograg.org>:

> On 11/3/2016 9:10 AM, Pablo Anzorena wrote:
> > Thanks for the answer.
> >
> > I checked the log and it wasn't logging anything.
> >
> > The error i'm facing is way bizarre... I create a new fresh collection
> and
> > then index with no problem, but it keeps throwing this error if i copy
> the
> > collection from one solrcloud to the other and then index.
> >
> > Any clue on why is this happening?
>
> Solr's source code doesn't seem to even have a 405 error, so I bet
> what's happening is that you have Solr sitting behind a proxy or load
> balancer, and that server doesn't like the request you sent, so it
> rejects it and Solr never receives anything.
>
> Here's an excerpt of code from the SolrException class in the master
> branch:
>
>   /**
>* This list of valid HTTP Status error codes that Solr may return in
>* the case of a "Server Side" error.
>*
>* @since solr 1.2
>*/
>   public enum ErrorCode {
> BAD_REQUEST( 400 ),
> UNAUTHORIZED( 401 ),
> FORBIDDEN( 403 ),
> NOT_FOUND( 404 ),
> CONFLICT( 409 ),
> UNSUPPORTED_MEDIA_TYPE( 415 ),
> SERVER_ERROR( 500 ),
> SERVICE_UNAVAILABLE( 503 ),
> INVALID_STATE( 510 ),
> UNKNOWN(0);
> public final int code;
>
> private ErrorCode( int c )
> {
>   code = c;
> }
> public static ErrorCode getErrorCode(int c){
>   for (ErrorCode err : values()) {
> if(err.code == c) return err;
>   }
>   return UNKNOWN;
> }
>   };
>
> Thanks,
> Shawn
>
>

Re: Posting files 405 http error

[Pablo Anzorena]

Thanks for the answer.

I checked the log and it wasn't logging anything.

The error i'm facing is way bizarre... I create a new fresh collection and
then index with no problem, but it keeps throwing this error if i copy the
collection from one solrcloud to the other and then index.

Any clue on why is this happening?

2016-11-01 17:42 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> What does the solr log say? I'd tail the Solr log while
> sending the query, that'll do two things:
>
> 1> insure that your request is actually getting to the
> Solr you expect.
>
> 2> the details in the solr log are often much more helpful
> than what gets returned to the client.
>
> Best,
> Erick
>
> On Tue, Nov 1, 2016 at 1:37 PM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> > Hey,
> >
> > I'm indexing a file with a delete query in xml format using the
> post.jar. I
> > have two solrclouds, which apparently have all the same configurations.
> The
> > thing is that I have no problem when indexing in one of them, but the
> other
> > keeps giving me this error:
> >
> > SimplePostTool version 5.0.0
> > Posting files to [base] url
> > http://solr2:8983/solr/mycollection/update?separator=| using
> content-type
> > application/xml...
> > POSTing file delete_file.unl.tmp to [base]
> > SimplePostTool: WARNING: Solr returned an error #405 (Method Not Allowed)
> > for url: http://solr2:8983/solr/mycollection/update?separator=|
> > SimplePostTool: WARNING: Response:
> > ErrorHTTP method POST is not
> > supported by this URL
> > SimplePostTool: WARNING: IOException while reading response:
> > java.io.IOException: Server returned HTTP response code: 405 for URL:
> > http://solr2:8983/solr/mycollection/update?separator=|
> > 1 files indexed.
> > Time spent: 0:00:00.253
> >
> > Do I need some extra configuration to support for xml updates?
> >
> > Thanks!
>

Posting files 405 http error

[Pablo Anzorena]

Hey,

I'm indexing a file with a delete query in xml format using the post.jar. I
have two solrclouds, which apparently have all the same configurations. The
thing is that I have no problem when indexing in one of them, but the other
keeps giving me this error:

SimplePostTool version 5.0.0
Posting files to [base] url
http://solr2:8983/solr/mycollection/update?separator=| using content-type
application/xml...
POSTing file delete_file.unl.tmp to [base]
SimplePostTool: WARNING: Solr returned an error #405 (Method Not Allowed)
for url: http://solr2:8983/solr/mycollection/update?separator=|
SimplePostTool: WARNING: Response:
ErrorHTTP method POST is not
supported by this URL
SimplePostTool: WARNING: IOException while reading response:
java.io.IOException: Server returned HTTP response code: 405 for URL:
http://solr2:8983/solr/mycollection/update?separator=|
1 files indexed.
Time spent: 0:00:00.253

Do I need some extra configuration to support for xml updates?

Thanks!

Rollback solrcloud

[Pablo Anzorena]

Hey,

I was trying to make a rollback under solrcloud and foundd that it's not
supported
https://issues.apache.org/jira/browse/SOLR-4895 (I have solr6.1.0)

So my question is, how can I simulate a rollback?
Actually what I'm doing is:

   1. prepareCommit
   2. add documents
   3. try to commit
   4. if success, then exit, else rollback.

I have to point out that it doesn't happen that multiple threads are
preparing commits nor adding documents, just single thread.

Thanks.

Re: Solr 6 Configuration - java.net.SocketTimeoutException

[Pablo Anzorena]

I think it is under solr_path/server/etc/ a file name jetty.xml



2016-08-15 10:01 GMT-03:00 Stan Lee :

> I currently doing a POC with SOLR 6 on my windows 7, with 16GB ram.
> Successfully imported 16 million of documents from SQL Server, where one of
> the SQL column is an XML.
> Whenever I query on the XML datatype (In solr, it's a text field), I keep
> getting SocketTimeoutException if the /select query goes behind the 1min
> mark. Can someone shed some light as to where I can config solr to go
> beyond the 1 min sockettimeout exception?
>
> Many thanks.
>
> Regards, Stan.
>

Re: Consume sql response using solrj

[Pablo Anzorena]

Thanks Joel, that's work perfectly well. I checked some cases and the data
is consistent.



2016-08-11 14:17 GMT-03:00 Joel Bernstein <joels...@gmail.com>:

> Actually try this:
>
> select a from b where _query_='a:b'
>
> *This produces the query:*
>
> (_query_:"a:b")
>
> which should run.
>
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Aug 11, 2016 at 1:04 PM, Joel Bernstein <joels...@gmail.com>
> wrote:
>
> > There are no test cases for this but you can try this syntax:
> >
> > select a from b where _query_=(a:c AND d:f)
> >
> > This should get translated to:
> >
> > _query_:(a:c AND d:f)
> >
> > This link describes the behavior of _query_ https://lucidworks.
> > com/blog/2009/03/31/nested-queries-in-solr/
> >
> > Just not positive how the SQL parser will treat the : in the query.
> >
> >
> >
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Thu, Aug 11, 2016 at 12:22 PM, Pablo Anzorena <
> anzorena.f...@gmail.com>
> > wrote:
> >
> >> Joel, one more thing.
> >>
> >> Is there anyway to use the sql and the lucene query syntax? The thing is
> >> that my bussiness application is tightly coupled with the lucene query
> >> syntax, so I need a way to use both the sql features (without the where
> >> clause) and the query syntax of lucene.
> >>
> >> Thanks.
> >>
> >> 2016-08-11 11:40 GMT-03:00 Pablo Anzorena <anzorena.f...@gmail.com>:
> >>
> >> > Excellent!
> >> >
> >> > Thanks Joel
> >> >
> >> > 2016-08-11 11:19 GMT-03:00 Joel Bernstein <joels...@gmail.com>:
> >> >
> >> >> There are two ways to do this with SolrJ:
> >> >>
> >> >> 1) Use the JDBC driver.
> >> >>
> >> >> 2) Use the SolrStream to send the request and then read() the Tuples.
> >> This
> >> >> is what the JDBC driver does under the covers. The sample code can be
> >> >> found
> >> >> here:
> >> >> https://github.com/apache/lucene-solr/blob/master/solr/solrj
> >> >> /src/java/org/apache/solr/client/solrj/io/sql/StatementImpl.java
> >> >>
> >> >> The constructStream() method creates a SolrStream with the request.
> >> >>
> >> >> Joel Bernstein
> >> >> http://joelsolr.blogspot.com/
> >> >>
> >> >> On Thu, Aug 11, 2016 at 10:05 AM, Pablo Anzorena <
> >> anzorena.f...@gmail.com
> >> >> >
> >> >> wrote:
> >> >>
> >> >> > Hey,
> >> >> >
> >> >> > I'm trying to get the response of solr via QueryResponse using
> >> >> > QueryResponse queryResponse = client.query(solrParams); (where
> client
> >> >> is a
> >> >> > CloudSolrClient)
> >> >> >
> >> >> > The error it thows is:
> >> >> >
> >> >> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrE
> >> xception:
> >> >> > Error
> >> >> > from server at http://tywin:8983/solr/testcol
> >> lection1_shard1_replica1:
> >> >> > Expected mime type application/octet-stream but got text/plain.
> >> >> > {"result-set":{"docs":[
> >> >> > {"count(*)":5304,"d1":2},
> >> >> > {"count(*)":5160,"d1":1},
> >> >> > {"count(*)":5016,"d1":3},
> >> >> > {"count(*)":4893,"d1":4},
> >> >> > {"count(*)":4824,"d1":5},
> >> >> > {"EOF":true,"RESPONSE_TIME":11}]}}
> >> >> > at
> >> >> > org.apache.solr.client.solrj.impl.HttpSolrClient.
> >> >> > executeMethod(HttpSolrClient.java:558)
> >> >> >
> >> >> > Then I tryed to implement a custom ResponseParser that override the
> >> >> > getContentType() and returns "text/plain", but it returns another
> >> error.
> >> >> >
> >> >> > So... Is it a way to get the sql response via this method?
> >> >> >
> >> >> > I make it works via Connection and ResultSets, but I need to use
> the
> >> >> other
> >> >> > way (if possible).
> >> >> >
> >> >> > Thanks!
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>

Re: Consume sql response using solrj

[Pablo Anzorena]

Joel, one more thing.

Is there anyway to use the sql and the lucene query syntax? The thing is
that my bussiness application is tightly coupled with the lucene query
syntax, so I need a way to use both the sql features (without the where
clause) and the query syntax of lucene.

Thanks.

2016-08-11 11:40 GMT-03:00 Pablo Anzorena <anzorena.f...@gmail.com>:

> Excellent!
>
> Thanks Joel
>
> 2016-08-11 11:19 GMT-03:00 Joel Bernstein <joels...@gmail.com>:
>
>> There are two ways to do this with SolrJ:
>>
>> 1) Use the JDBC driver.
>>
>> 2) Use the SolrStream to send the request and then read() the Tuples. This
>> is what the JDBC driver does under the covers. The sample code can be
>> found
>> here:
>> https://github.com/apache/lucene-solr/blob/master/solr/solrj
>> /src/java/org/apache/solr/client/solrj/io/sql/StatementImpl.java
>>
>> The constructStream() method creates a SolrStream with the request.
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Thu, Aug 11, 2016 at 10:05 AM, Pablo Anzorena <anzorena.f...@gmail.com
>> >
>> wrote:
>>
>> > Hey,
>> >
>> > I'm trying to get the response of solr via QueryResponse using
>> > QueryResponse queryResponse = client.query(solrParams); (where client
>> is a
>> > CloudSolrClient)
>> >
>> > The error it thows is:
>> >
>> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> > Error
>> > from server at http://tywin:8983/solr/testcollection1_shard1_replica1:
>> > Expected mime type application/octet-stream but got text/plain.
>> > {"result-set":{"docs":[
>> > {"count(*)":5304,"d1":2},
>> > {"count(*)":5160,"d1":1},
>> > {"count(*)":5016,"d1":3},
>> > {"count(*)":4893,"d1":4},
>> > {"count(*)":4824,"d1":5},
>> > {"EOF":true,"RESPONSE_TIME":11}]}}
>> > at
>> > org.apache.solr.client.solrj.impl.HttpSolrClient.
>> > executeMethod(HttpSolrClient.java:558)
>> >
>> > Then I tryed to implement a custom ResponseParser that override the
>> > getContentType() and returns "text/plain", but it returns another error.
>> >
>> > So... Is it a way to get the sql response via this method?
>> >
>> > I make it works via Connection and ResultSets, but I need to use the
>> other
>> > way (if possible).
>> >
>> > Thanks!
>> >
>>
>
>

Re: Consume sql response using solrj

[Pablo Anzorena]

Excellent!

Thanks Joel

2016-08-11 11:19 GMT-03:00 Joel Bernstein <joels...@gmail.com>:

> There are two ways to do this with SolrJ:
>
> 1) Use the JDBC driver.
>
> 2) Use the SolrStream to send the request and then read() the Tuples. This
> is what the JDBC driver does under the covers. The sample code can be found
> here:
> https://github.com/apache/lucene-solr/blob/master/solr/
> solrj/src/java/org/apache/solr/client/solrj/io/sql/StatementImpl.java
>
> The constructStream() method creates a SolrStream with the request.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Aug 11, 2016 at 10:05 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
>
> > Hey,
> >
> > I'm trying to get the response of solr via QueryResponse using
> > QueryResponse queryResponse = client.query(solrParams); (where client is
> a
> > CloudSolrClient)
> >
> > The error it thows is:
> >
> > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> > Error
> > from server at http://tywin:8983/solr/testcollection1_shard1_replica1:
> > Expected mime type application/octet-stream but got text/plain.
> > {"result-set":{"docs":[
> > {"count(*)":5304,"d1":2},
> > {"count(*)":5160,"d1":1},
> > {"count(*)":5016,"d1":3},
> > {"count(*)":4893,"d1":4},
> > {"count(*)":4824,"d1":5},
> > {"EOF":true,"RESPONSE_TIME":11}]}}
> > at
> > org.apache.solr.client.solrj.impl.HttpSolrClient.
> > executeMethod(HttpSolrClient.java:558)
> >
> > Then I tryed to implement a custom ResponseParser that override the
> > getContentType() and returns "text/plain", but it returns another error.
> >
> > So... Is it a way to get the sql response via this method?
> >
> > I make it works via Connection and ResultSets, but I need to use the
> other
> > way (if possible).
> >
> > Thanks!
> >
>

Consume sql response using solrj

[Pablo Anzorena]

Hey,

I'm trying to get the response of solr via QueryResponse using
QueryResponse queryResponse = client.query(solrParams); (where client is a
CloudSolrClient)

The error it thows is:

org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://tywin:8983/solr/testcollection1_shard1_replica1:
Expected mime type application/octet-stream but got text/plain.
{"result-set":{"docs":[
{"count(*)":5304,"d1":2},
{"count(*)":5160,"d1":1},
{"count(*)":5016,"d1":3},
{"count(*)":4893,"d1":4},
{"count(*)":4824,"d1":5},
{"EOF":true,"RESPONSE_TIME":11}]}}
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)

Then I tryed to implement a custom ResponseParser that override the
getContentType() and returns "text/plain", but it returns another error.

So... Is it a way to get the sql response via this method?

I make it works via Connection and ResultSets, but I need to use the other
way (if possible).

Thanks!

Re: Does solr support two phase commit or any other distributed transaction protocol?

[Pablo Anzorena]

Thanks Shawn, I understood perfectly well. One important thing in my use
case is that I only have one entry point for indexing solr, so I won't have
any problems of multiple threads trying to update the index.

So what can I do if I have to index in solr and also in postgres and I need
to do it transactionally?

I imagine something like:
1)Open a distributed transaction in postgresql and "index" the data with
the global id transaction.
1.1)If some problem occurs, rollback postgres. End of transaction.
2)Index data in solr. If no problem occurs, commit in solr and then commit
in postgres. End of transaction.
2.1)If some problem occurs in solr, rollback solr and rollback postgres.
End of transaction.

2016-08-09 11:24 GMT-03:00 Shawn Heisey <apa...@elyograg.org>:

> On 8/9/2016 7:55 AM, Pablo Anzorena wrote:
> > That's it. Thanks.
>
> Solr doesn't support transactions in the way that most people with a
> database background imagine them.
>
> With a typical database server, all changes to the database that happen
> on a single DB connection can be committed or rolled back completely
> independently from updates that happen on other DB connections.
>
> Solr doesn't work this way.
>
> In a Lucene index (Solr is a Lucene program), a "transaction" is all
> updates made since the last commit with openSearcher=true.  This
> includes ALL updates made, regardless of where they came from.  So if
> you have a dozen different threads/processes making changes to your Solr
> index, then have something do a commit, all of the updates made by those
> 12 sources before the commit will be committed.  There is no concept of
> an individual transaction.
>
> Adding the DB transaction model would be a *major* development effort,
> and there's a good chance that adding it would destroy the blazing
> search performance that Solr and Lucene are known for.
>
> Thanks,
> Shawn
>
>

Does solr support two phase commit or any other distributed transaction protocol?

[Pablo Anzorena]

That's it.

Thanks.

Re: Group and sum in SOLR 5.3

[Pablo Anzorena]

Using the json facet api is quite easy
Check out: http://yonik.com/json-facet-api/#TermsFacet

http://solr:8983/solr/your_collection/select?q=*:*=json=true=
{
  property_codes_group_by : {
type : terms,
field : property_code,
facet : { sum_price : "sum(price)" }
  }
}


2016-07-29 7:47 GMT-03:00 andreap21 :

> Hi,
> is there any way in SOLR 5.3 to achieve grouping and do mathematical
> operations like sum, avg on a certain document property?
>
> Please find below example:
>
> 
>   HIKS
>   Hotel Holiday Inn King's cross
>   4
>   40.99
> 
>
> 
>   HIKS
>   Hotel Holiday Inn King's cross
>   4
>   40.99
> 
>
> 
>   HIKS
>   Hotel Holiday Inn King's cross
>   4
>   40.99
> 
>
> We would need to group by the property_code (HIKS) and get the sum of all
> the prices from the resulting group.
>
>
> Thanks,
> Andrea
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Group-and-sum-in-SOLR-5-3-tp4289556.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Parallel SQL where exists, in

[Pablo Anzorena]

I don't have time now, but I'll keep that in mind.

Thanks.

2016-07-19 12:27 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> There is a lot of activity in the ParallelSQL world, all being done
> by a very few people, so it's a matter of priorities. Can you
> consider submitting a patch?
>
> Best,
> Erick
>
> On Tue, Jul 19, 2016 at 8:12 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> > Hey,
> >
> > Is anyone willing to add the where exists and in clauses into paraller
> sql?
> >
> > Thanks.
>

Parallel SQL where exists, in

[Pablo Anzorena]

Hey,

Is anyone willing to add the where exists and in clauses into paraller sql?

Thanks.

Re: Is SolrCloudClient Singleton Pattern possible with multiple collections?

[Pablo Anzorena]

The thing is that back in solr4.8, when I was using solr standalone and I
had to make a distributed query among multiple shards, I found that for
each shard in the param "shards" it makes a request (which is the correct
behaviour I know) but when I put just one shard in the "shards" param then
it makes two identical requests.
So, now because I'm using SolrCloud I replaced the "shards" with
"collection" param and I was wondering if it would have the same erratic
behaviour.

Now I tried and I found that it has the correct behaviour.

Thanks, and sorry for asking before testing it.

2016-07-14 15:26 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> bq:  if using the param "collection" is the same
>
> Did you just try it? If so what happened?
>
> Not sure what you're asking here. It's the name of the
> collection you want to query against. It's only
> necessary when you want to go against a
> collection that isn't the default which you can set with
> setDefaultCollection()
>
> Best,
> Erick
>
> On Thu, Jul 14, 2016 at 10:51 AM, Pablo Anzorena
> <anzorena.f...@gmail.com> wrote:
> > I was using
> > public QueryResponse query(ModifiableSolrParams params, METHOD method)
> >
> > And my actual code is parsing that object. I can change it to your
> method,
> > but before that let me ask you if using the param "collection" is the
> same.
> >
> > Actually I am using the param "collection" only when I need to request to
> > multiple collections.
> >
> > Thanks.
> >
> >
> >
> > 2016-07-14 14:15 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:
> >
> >> Just use the
> >>
> >> public NamedList request(SolrRequest request, String collection)
> >>
> >> method on the SolrCloudClient?
> >>
> >> Best,
> >> Erick
> >>
> >> On Thu, Jul 14, 2016 at 9:18 AM, Pablo Anzorena <
> anzorena.f...@gmail.com>
> >> wrote:
> >> > Hey,
> >> > So the question is quite simple, Is it possible to use Singleton
> Pattern
> >> > with SolrCloudClient instantiation and then reuse that instance to
> handle
> >> > multiple requests concurrently accessing differente collections?
> >> >
> >> > Thanks.
> >>
>

Re: Is SolrCloudClient Singleton Pattern possible with multiple collections?

[Pablo Anzorena]

I was using
public QueryResponse query(ModifiableSolrParams params, METHOD method)

And my actual code is parsing that object. I can change it to your method,
but before that let me ask you if using the param "collection" is the same.

Actually I am using the param "collection" only when I need to request to
multiple collections.

Thanks.



2016-07-14 14:15 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> Just use the
>
> public NamedList request(SolrRequest request, String collection)
>
> method on the SolrCloudClient?
>
> Best,
> Erick
>
> On Thu, Jul 14, 2016 at 9:18 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> > Hey,
> > So the question is quite simple, Is it possible to use Singleton Pattern
> > with SolrCloudClient instantiation and then reuse that instance to handle
> > multiple requests concurrently accessing differente collections?
> >
> > Thanks.
>

Is SolrCloudClient Singleton Pattern possible with multiple collections?

[Pablo Anzorena]

Hey,
So the question is quite simple, Is it possible to use Singleton Pattern
with SolrCloudClient instantiation and then reuse that instance to handle
multiple requests concurrently accessing differente collections?

Thanks.

Re: Facet in SOLR Cloud vs Core

[Pablo Anzorena]

Sorry for introducing bad information.
Because it happens in the json facet api, I thought it would also happen in
the facet. Soyrry again for the misunderstood.

2016-07-07 16:08 GMT-03:00 Chris Hostetter :

>
> : The problem with the shards appears in the following scenario (note that
> : the problem below also applies in a solr standalone enviroment with
> : distributed search):
> :
> : Shard1: DATA_SOURCE1 (3 docs), DATA_SOURCE2 (2 docs), DATA_SOURCE3 (2
> docs).
> : Shard2: DATA_SOURCE3 (2 docs), DATA_SOURCE2 (1 docs).
> :
> : If you make a distributed search across these two shards, faceting
> : dataSourceName with a limit of 1, it will ask for the top 1 in the first
> : shard (DATA_SOURCE1 (3 docs)) and for the top 1 in the second shard
> : (DATA_SOURCE3
> : (2 docs)). After that it will merge the results and return DATA_SOURCE1
> (3
> : docs), when it should have return DATA_SOURCE3 (4 docs).
>
> That's completley false.
>
> a) in the first pass, even if you ask for "top 1" (ie: facet.limit=1) solr
> will overrequest when comunicating with each shard (the amount of
> overrequest is a function of your facet.limit, so as facet.limit increases
> so does the overrequest amount)
>
> b) if *any* (but not *all*) shards returns DATA_SOURCE3 from the
> initial shard request, a second "refinement" step will request the count
> for DATA_SOURCE3 from all of the other shards to get an accurate count,
> and to accurately sort DATA_SOURCE3 to the top of the facet constraint
> list.
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Facet in SOLR Cloud vs Core

[Pablo Anzorena]

As long as you don't shard your index, you will have no problem migrating
to solrcloud.

The problem with the shards appears in the following scenario (note that
the problem below also applies in a solr standalone enviroment with
distributed search):

Shard1: DATA_SOURCE1 (3 docs), DATA_SOURCE2 (2 docs), DATA_SOURCE3 (2 docs).
Shard2: DATA_SOURCE3 (2 docs), DATA_SOURCE2 (1 docs).

If you make a distributed search across these two shards, faceting
dataSourceName with a limit of 1, it will ask for the top 1 in the first
shard (DATA_SOURCE1 (3 docs)) and for the top 1 in the second shard
(DATA_SOURCE3
(2 docs)). After that it will merge the results and return DATA_SOURCE1 (3
docs), when it should have return DATA_SOURCE3 (4 docs).

Summarizing: if you make a distributed search with a facet.limit, there is
a chance that the count is not correct (it also applies to stats).

2016-07-07 15:28 GMT-03:00 Whelan, Andy :

> Hello,
>
> I have am somewhat of a novice when it comes to using SOLR in a
> distributed SolrCloud environment. My team and I are doing development work
> with a SOLR core. We will shortly be transitioning over to a SolrCloud
> environment.
>
> My question specifically has to do with Facets in a SOLR cloud/collection
> (distributed environment). The core I am working with has a field
> "dataSourceName" defined as following in its schema.xml file.
>
>  required="true"/>
>
> I am using the following facet query which works fine in more Core based
> index
>
>
> http://localhost:8983/solr/gamra/select?q=*:*=0=true=dataSourceName
>
> It returns counts for each distinct dataSourceName as follows (which is
> the desired behavior).
>
> 
>
>   169
>   121
>   68
>
> 
>
> I am wondering if this should work fine in the SOLR Cloud as well?  Will
> this method give me accurate counts out of the box in a SOLR Cloud
> configuration?
>
> Thanks
> -Andrew
>
> PS: The reason I ask is because I know there is some estimating performed
> in certain cases for the Facet "unique" function (as is outlined here:
> http://yonik.com/solr-count-distinct/ ). So I guess I am wondering why
> folks wouldn't just do what I have done vs going throught the trouble of
> using the unique(dataSourceName) function?
>
>
>

Re: No live SolrServers triggered by maxclausecount

[Pablo Anzorena]

Thanks! I will analyze it and let you know.


2016-06-28 11:25 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> OK, but the consider what you can do to keep from having to
> create such things.
> 1> for infix notation (leading and trailing wildcards) you can
> use ngrams to turn them into simple queries. These are
> performance-killers.
> 2> Use reverseWildcardFactory to deal with leading wildcards
> 3> restrict the wildcard search to at least two "real" characters.
>  I can (and have) argued that a* is not useful to a user at all.
>
> The point is that your responsiveness will suffer if you come
> anywhere near 100K OR clauses. Maybe that's OK, but
> it's lurking out there.
>
>
> And no I can't see the image, the Apache server is pretty
> aggressive about stripping those too.
>
> Best,
> Erick
>
> On Tue, Jun 28, 2016 at 7:19 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
>
> > Hi Erick, thanks for answering.
> >
> > I attached the image to the body so you can see it.
> >
> >
> >
> > Why do I need so many clauses?
> > It is because I have two text fields that contains in average 25 words
> > with a lot of typos (which I'm not cleaning it) and on top of that the
> > index consists of 25 million records. And we let the users make queries
> > with phrases, wildcards prefix and suffix. So for example the following
> > query is valid *q=text_field:"*ban*" AND text_field2:"foo* bar*"* (I
> > ommited the query parser syntax that allows this).
> > ​
> >
> > 2016-06-28 2:08 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:
> >
> >> That error sometimes gets reported inappropriately, as long as the
> >> servers are live
> >> you can pretty much ignore it.
> >>
> >> Attachments pretty much all get stripped by the mail server so we can't
> >> see your
> >> screen shot.
> >>
> >> Setting your max clause count to over 100K is pretty much an
> >> anti-pattern, what in the world
> >> are you doing that would require it to be that high? You haven't
> >> really shown us the query
> >> you're sending, but I bet it's amazing. Frankly, anything over the
> >> default of 1K is suspect.
> >>
> >> If this is some clause like id:(1 OR 2 OR 3 OR 4...) you really
> >> want to try using the
> >> TermsQueryParser (note the 's'  as opposed to TermQueryParser (no 's').
> >> See:
> >>
> >>
> https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-TermQueryParser
> >>
> >> And if you use the TermsQueryParser, it's probably a good idea to sort
> >> the list of terms, it's
> >> more efficient.
> >>
> >> Or do a join or... because anything with that many terms will be
> >> rather slow to say the least.
> >>
> >> Best,
> >> Erick
> >>
> >>
> >> On Mon, Jun 27, 2016 at 8:38 AM, Pablo Anzorena <
> anzorena.f...@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > I have an ensemble zookeeper consisting of 3 machines and 2 machines
> >> with
> >> > solrcloud.
> >> >
> >> > With a high frequency I see in the logging:
> >> > No live SolrServers available to handle this
> >> > request:[
> >>
> http://solr2:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica1,
> >> >
> >>
> http://solr3:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica2]
> >> >
> >> > and the state.json is:
> >> >
> >> > {"usa_bills_imp_2016_2016062300":{
> >> > "replicationFactor":"2",
> >> > "shards":{"shard1":{
> >> > "range":"8000-7fff",
> >> > "state":"active",
> >> > "replicas":{
> >> >   "core_node1":{
> >> >
>  "core":"usa_bills_imp_2016_2016062300_shard1_replica2",
> >> > "base_url":"http://solr3:8983/solr;,
> >> > "node_name":"solr3:8983_solr",
> >> > "state":"active"},
> >> >   "core_node2":{
> >> >
>  "core":"usa_bills_imp_2016_2016062300_shard1_replica1",
> >> > "base_url":"http:/

Re: No live SolrServers triggered by maxclausecount

[Pablo Anzorena]

Hi Erick, thanks for answering.

I attached the image to the body so you can see it.



Why do I need so many clauses?
It is because I have two text fields that contains in average 25 words with
a lot of typos (which I'm not cleaning it) and on top of that the index
consists of 25 million records. And we let the users make queries with
phrases, wildcards prefix and suffix. So for example the following query is
valid *q=text_field:"*ban*" AND text_field2:"foo* bar*"* (I ommited the
query parser syntax that allows this).
​

2016-06-28 2:08 GMT-03:00 Erick Erickson <erickerick...@gmail.com>:

> That error sometimes gets reported inappropriately, as long as the
> servers are live
> you can pretty much ignore it.
>
> Attachments pretty much all get stripped by the mail server so we can't
> see your
> screen shot.
>
> Setting your max clause count to over 100K is pretty much an
> anti-pattern, what in the world
> are you doing that would require it to be that high? You haven't
> really shown us the query
> you're sending, but I bet it's amazing. Frankly, anything over the
> default of 1K is suspect.
>
> If this is some clause like id:(1 OR 2 OR 3 OR 4...) you really
> want to try using the
> TermsQueryParser (note the 's'  as opposed to TermQueryParser (no 's').
> See:
>
> https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-TermQueryParser
>
> And if you use the TermsQueryParser, it's probably a good idea to sort
> the list of terms, it's
> more efficient.
>
> Or do a join or... because anything with that many terms will be
> rather slow to say the least.
>
> Best,
> Erick
>
>
> On Mon, Jun 27, 2016 at 8:38 AM, Pablo Anzorena <anzorena.f...@gmail.com>
> wrote:
> > Hi,
> >
> > I have an ensemble zookeeper consisting of 3 machines and 2 machines with
> > solrcloud.
> >
> > With a high frequency I see in the logging:
> > No live SolrServers available to handle this
> > request:[
> http://solr2:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica1,
> > http://solr3:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica2
> ]
> >
> > and the state.json is:
> >
> > {"usa_bills_imp_2016_2016062300":{
> > "replicationFactor":"2",
> > "shards":{"shard1":{
> > "range":"8000-7fff",
> > "state":"active",
> > "replicas":{
> >   "core_node1":{
> > "core":"usa_bills_imp_2016_2016062300_shard1_replica2",
> > "base_url":"http://solr3:8983/solr;,
> > "node_name":"solr3:8983_solr",
> > "state":"active"},
> >   "core_node2":{
> > "core":"usa_bills_imp_2016_2016062300_shard1_replica1",
> > "base_url":"http://solr2:8983/solr;,
> > "node_name":"solr2:8983_solr",
> > "state":"active",
> > "leader":"true",
> > "router":{"name":"compositeId"},
> > "maxShardsPerNode":"1",
> > "autoAddReplicas":"false"}}
> >
> > And the full stacktrace of the error is:
> >
> > null:org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: No live SolrServers
> > available to handle this
> > request:[
> http://solr3:8983/solr/usa_bills_imp_2014_20160603115528_shard1_replica2,
> > http://solr3:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica2
> ,
> > http://solr2:8983/solr/usa_bills_imp_2014_20160603115528_shard1_replica1
> ,
> > http://solr2:8983/solr/usa_bills_imp_2015_20160610125230_shard1_replica1
> ]
> >   at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
> >   at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> >   at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
> >   at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
> >   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
> >   at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
> >   at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
> >

No live SolrServers triggered by maxclausecount

[Pablo Anzorena]

Hi,

I have an ensemble zookeeper consisting of 3 machines and 2 machines with
solrcloud.

With a high frequency I see in the logging:
*No live SolrServers available to handle this
request:[http://solr2:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica1
,
http://solr3:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica2
]*

and the state.json is:

{"usa_bills_imp_2016_2016062300":{
"replicationFactor":"2",
"shards":{"shard1":{
"range":"8000-7fff",
"state":"active",
"replicas":{
  "core_node1":{
"core":"usa_bills_imp_2016_2016062300_shard1_replica2",
"base_url":"http://solr3:8983/solr;,
"node_name":"solr3:8983_solr",
"state":"active"},
  "core_node2":{
"core":"usa_bills_imp_2016_2016062300_shard1_replica1",
"base_url":"http://solr2:8983/solr;,
"node_name":"solr2:8983_solr",
"state":"active",
"leader":"true",
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false"}}
*And the full stacktrace of the error is:*

null:org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers
available to handle this
request:[http://solr3:8983/solr/usa_bills_imp_2014_20160603115528_shard1_replica2,
http://solr3:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica2,
http://solr2:8983/solr/usa_bills_imp_2014_20160603115528_shard1_replica1,
http://solr2:8983/solr/usa_bills_imp_2015_20160610125230_shard1_replica1]
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live
SolrServers available to handle this
request:[http://solr3:8983/solr/usa_bills_imp_2014_20160603115528_shard1_replica2,
http://solr3:8983/solr/usa_bills_imp_2016_2016062300_shard1_replica2,
http://solr2:8983/solr/usa_bills_imp_2014_20160603115528_shard1_replica1,
http://solr2:8983/solr/usa_bills_imp_2015_20160610125230_shard1_replica1]
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:355)
at 
org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:246)
at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:221)
at 
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)
at 

Re: SolrCloud SolrNode stopping randomly for no reason

[Pablo Anzorena]

Sorry for the poor details, but I didn't post the log files because there
was nothing out of the ordinary in the solr.log file, neither in
the solr-8984-console.log, nor in solr_gc.log.

What log do you want me to show you? solr.log.1 (which I think it should be
the last one) for example? You need the tail or the head of the file?

When I say "stopping for no reason" I mean the service is not running
anymore, the process is finished. I tried killing it with kill -9 command
and it does not log that, my first thought was that I restarted the
standalone solr service which try to stop the service and if it can't it
kills it doing SOLR_PROCESS_ID="`ps -eaf | grep -v "grep" | grep
"start.jar";kill -9 ${SOLR_PROCESS_ID}. So sometimes it could kill
solrcloud instead of standalone, but sometimes the datetime does not match.
Another option is that it gives an outofmemoryerror and the oom script is
killing the process, but again I saw nothing in the solr_gc.log.

2016-06-07 10:18 GMT-03:00 Shawn Heisey <apa...@elyograg.org>:

> On 6/7/2016 6:08 AM, Pablo Anzorena wrote:
> > I'am using SolrCloud with two nodes (5.2.1). One or two times a day the
> > node1 is stopping for no reason. I checked the logs but no errors are
> beign
> > logged.
> > I also have a standalone solr service in both nodes running in production
> > (we are doing the migration to SolrCloud, that's why).
>
> https://wiki.apache.org/solr/UsingMailingLists
>
> There are no real details to your message.  What precisely does
> "stopping for no reason" mean?  What does Solr *do*?  We cannot see your
> system, you must tell us what is happening with considerable detail.
>
> It seems highly unlikely that Solr would misbehave without logging
> *something*.  Are you looking at the Logging tab in the admin UI, or the
> actual solr.log file?  The solr.log file is the only reliable place to
> look.  When you restart Solr, the current logfile is renamed and a new
> solr.log will be created.
>
> Thanks,
> Shawn
>
>

SolrCloud SolrNode stopping randomly for no reason

[Pablo Anzorena]

Hey,

I'am using SolrCloud with two nodes (5.2.1). One or two times a day the
node1 is stopping for no reason. I checked the logs but no errors are beign
logged.
I also have a standalone solr service in both nodes running in production
(we are doing the migration to SolrCloud, that's why).

Thanks.

Re: OutOfMemoryError does not fire the script

[Pablo Anzorena]

Perfect, thank you very much.

2016-05-27 12:44 GMT-03:00 Shawn Heisey <apa...@elyograg.org>:

> On 5/27/2016 7:05 AM, Pablo Anzorena wrote:
> > I am using solr 5.2.1 in cloud mode. My jvm arguments for the
> > OutOfMemoryError is
> > -XX:OnOutOfMemoryError='/etc/init.d/solrcloud;restart'
> >
> > In the Solr UI, the event is beign fired, but nothing happens.
>
> In all versions before 5.5.1, that -XX parameter is placed incorrectly
> on the commandline, in the bin/solr script, so it doesn't work.
>
> https://issues.apache.org/jira/browse/SOLR-8145
>
> If you move the parameter so it comes before "-jar start.jar" on the
> commandline (see the newest patch attached to SOLR-8145) and restart
> Solr, it will fix this problem.
>
> Thanks,
> Shawn
>
>

OutOfMemoryError does not fire the script

[Pablo Anzorena]

Hello,

I am using solr 5.2.1 in cloud mode. My jvm arguments for the
OutOfMemoryError is
-XX:OnOutOfMemoryError='/etc/init.d/solrcloud;restart'

In the Solr UI, the event is beign fired, but nothing happens.

What am I missing?

Regards.

Create collections in SolrCloud "Could not get shard id for core"

[Pablo Anzorena]

Hey,
I'm using solr 5.2.1 and yesterday I started migrating to SolrCloud so I
might be quite noobish with it. The thing is that I could create 3
collections with not so many inconvenients, but when trying to create
another collection it throws a timeout and inside solr log it says "Could
not get shard id for core my_core_shard1_replica1".

My SolrCloud has three servers running zookeeper and two of them running
solr in cloud mode.
The command for creating the collection is:

http://localhost:8983/solr/admin/collections?action=CREATE=my_core=1=1=my_core_config=server1:8983_solr

Thanks.

Re: Which line is solr following in terms of a BI Tool?

[Pablo Anzorena]

Thank you very much both of you for your insights!
I really appreciate it.



2016-04-13 11:30 GMT-03:00 Kevin Risden :

> For Solr 6, ParallelSQL and Solr JDBC driver are going to be developed more
> as well as JSON facets. The Solr JDBC driver that is in Solr 6 contains
> SOLR-8502. There are further improvements coming in SOLR-8659 that didn't
> make it into 6.0. The Solr JDBC piece leverages ParallelSQL and in some
> cases uses JSON facets under the hood.
>
> The Solr JDBC driver should enable BI tools to connect to Solr and use the
> language of SQL. This is also a familiar interface for many Java
> developers.
>
> Just a note: Solr is not an RDBMS and shouldn't be treated like one even
> with a JDBC driver. The Solr JDBC driver is more of a convenience for
> querying.
>
> Kevin Risden
>
> On Tue, Apr 12, 2016 at 6:24 PM, Erick Erickson 
> wrote:
>
> > The unsatisfactory answer is that the have different characteristics.
> >
> > The analytics contrib does not work in distributed mode. It's not
> > receiving a lot of love at this point.
> >
> > The JSON facets are estimations. Generally very close but are not
> > guaranteed to be 100% accurate. The variance, as I understand it,
> > is something on the order of < 1% in most cases.
> >
> > The pivot facets are accurate, but more expensive than the JSON
> > facets.
> >
> > And, to make matters worse, the ParllelSQL way of doing some
> > aggregations is going to give yet another approach.
> >
> > Best,
> > Erick
> >
> > On Tue, Apr 12, 2016 at 7:15 AM, Pablo  wrote:
> > > Hello,
> > > I think this topic is important for solr users that are planning to use
> > solr
> > > as a BI Tool.
> > > Speaking about facets, nowadays there are three majors way of doing
> > (more or
> > > less) the same  in solr.
> > > First, you have the pivot facets, on the other hand you have the
> > Analytics
> > > component and finally you have the JSON Facet Api.
> > > So, which line is Solr following? Which of these component is going to
> > be in
> > > constant development and which one is going to be deprecated sooner.
> > > In Yonik page, there are some test that shows how JSON Facet Api
> performs
> > > better than legacy facets, also the Api was way simpler than the pivot
> > > facets, so in my case that was enough to base my solution around the
> JSON
> > > Api. But I would like to know what are the thoughts of the solr
> > developers.
> > >
> > > Thanks!
> > >
> > >
> > >
> > > --
> > > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Which-line-is-solr-following-in-terms-of-a-BI-Tool-tp4269597.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>