shards are going down frequently

[ramesh.pallapo...@fedex.com]

Hi All,

I am using solr-4.10.4 and our shards are frequently going down. When ever
it goes down , we are deleting the data folder and restart server it is
recovering data from replica. Any idea why shards are going down? Any
suggestions ? Any defect in 4.10.4 version?

 



-
Thanks,
Ramesh.
--
View this message in context: 
http://lucene.472066.n3.nabble.com/shards-are-going-down-frequently-tp4340652.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Issue with highlighter

[David Smiley]

> Beware of NOT plus OR in a search. That will certainly produce no
highlights. (eg test -results when default op is OR)

Seems like a bug to me; the default operator shouldn't matter in that case
I think since there is only one clause that has no BooleanQuery.Occur
operator and thus the OR/AND shouldn't matter.  The end effect is "test" is
effectively required and should definitely be highlighted.

Note to Ali: Phil's comment implies use of hl.method=unified which is not
the default.

On Wed, Jun 14, 2017 at 10:22 PM Phil Scadden  wrote:

> Just had similar issue - works for some, not others. First thing to look
> at is hl.maxAnalyzedChars is the query. The default is quite small.
> Since many of my documents are large PDF files, I opted to use
> storeOffsetsWithPositions="true" termVectors="true" on the field I was
> searching on.
> This certainly did increase my index size but not too bad and certainly
> fast.
> https://cwiki.apache.org/confluence/display/solr/Highlighting
>
> Beware of NOT plus OR in a search. That will certainly produce no
> highlights. (eg test -results when default op is OR)
>
>
> -Original Message-
> From: Ali Husain [mailto:alihus...@outlook.com]
> Sent: Thursday, 15 June 2017 11:11 a.m.
> To: solr-user@lucene.apache.org
> Subject: Issue with highlighter
>
> Hi,
>
>
> I think I've found a bug with the highlighter. I search for the word
> "something" and I get an empty highlighting response for all the documents
> that are returned shown below. The fields that I am searching over are
> text_en, the highlighter works for a lot of queries. I have no
> stopwords.txt list that could be messing this up either.
>
>
>  "highlighting":{
> "310":{},
> "103":{},
> "406":{},
> "1189":{},
> "54":{},
> "292":{},
> "309":{}}}
>
>
> Just changing the search term to "something like" I get back this:
>
>
> "highlighting":{
> "310":{},
> "309":{
>   "content":["1949 Convention, like those"]},
> "103":{},
> "406":{},
> "1189":{},
> "54":{},
> "292":{},
> "286":{
>   "content":["persons in these classes are treated like
> combatants, but in other respects"]},
> "336":{
>   "content":["   be treated like engagement"]}}}
>
>
> So I know that I have it setup correctly, but I can't figure this out.
> I've searched through JIRA/Google and haven't been able to find a similar
> issue.
>
>
> Any ideas?
>
>
> Thanks,
>
> Ali
> Notice: This email and any attachments are confidential and may not be
> used, published or redistributed without the prior written consent of the
> Institute of Geological and Nuclear Sciences Limited (GNS Science). If
> received in error please destroy and immediately notify GNS Science. Do not
> copy or disclose the contents.
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

Re: Reg:- StrField Analyzer Issue

[@Nandan@]

Thanks Erick For best Explanation.

The issue with My data is as below. :-
I have few data on my books table.

cqlsh:nandan> select * from books;



 id   | author   | date | isbn |
solr_query | title

--+--+--+--++---

 3910b29d-c957-4312-9b8b-738b1d0e25d0 |  Chandan | 2015 |  1asd33s |
null |  Solr

 d7534021-80c2-4315-8027-84f04bf92f53 | 现在有货 | 2015 | 现在有货 |   null
|  Solr

 780b5163-ca6b-40bf-a523-af2c075ef7df |   在有货 | 2015 |   在有货 |   null
|  Solr

 e6229268-d0fd-485b-ad89-bbde73a07ed6 |   货 | 2015 |   现有货 |   null
|  Solr

 76461e7e-6c31-4a4b-8a36-0df5ce746d50 |   Nandan | 2017 |1 |
null |  Datastax

 9a9c66c2-cd34-460e-a301-6d8e7eb14e55 |   Kundan | 2016 | 12ws |
null | Cassandra

 7e87dc3a-5e4e-4653-84cc-3d83239708d4 |   现有货 | 2015 |   现有货 |   null
|  Solr

 6971976e-2528-4956-94a8-345deefe5796 | 现货 | 2015 | 现货 |   null
|  Solr


When I am trying to select from table based on author  as:-

cqlsh:nandan> SELECT * from books where solr_query = 'author:现有货';



 id   | author   | date | isbn |
solr_query | title

--+--+--+--++---

 d7534021-80c2-4315-8027-84f04bf92f53 | 现在有货 | 2015 | 现在有货 |   null |
Solr

 7e87dc3a-5e4e-4653-84cc-3d83239708d4 |   现有货 | 2015 |   现有货 |   null
|  Solr

 6971976e-2528-4956-94a8-345deefe5796 | 现货 | 2015 | 现货 |   null
|  Solr

 780b5163-ca6b-40bf-a523-af2c075ef7df |   在有货 | 2015 |   在有货 |   null
|  Solr

It should return me one value , but I am getting other records also,


But when I am trying to retrive another way, then it is returning me 0 rows
as :-

cqlsh:nandan> SELECT * from books where solr_query = 'author:*现有货*';



 id | author | date | isbn | solr_query | title

++--+--++---



(0 rows)

cqlsh:nandan> SELECT * from books where solr_query = 'author:*现有货';



 id | author | date | isbn | solr_query | title

++--+--++---



(0 rows)

cqlsh:nandan> SELECT * from books where solr_query = 'author:现有货*';



 id | author | date | isbn | solr_query | title

++--+--++---



(0 rows)


In Some cases, I am getting correct data but in some case, I am getting
wrong data. Please check.

Thanks

Nandan

On Thu, Jun 15, 2017 at 11:47 AM, Erick Erickson 
wrote:

> Back up a bit and tell us why you want to use StrField, because what
> you're trying to do is somewhat confused.
>
> First of all, StrFields are totally unanalyzed. So defining an
>  as part of a StrField type definition is totally
> unsupported. I'm a bit surprised that Solr even starts up.
>
> Second, you can't search a StrField unless you search the whole thing
> exactly. That is, if your title field is "My dog has fleas", there
> only a few ways to match anything in that field
>
> 1> search "My dog has fleas" exactly. Even "my dog has fleas" wouldn't
> match because of the capitalization. "My dog has fleas." would also
> fail because of the period. StrField types are intended for data that
> should be invariant and not tokenized.
>
> 2> prefix search as "My dog*"
>
> 3> pre-and-postfix as "*dog*"
>
> <2> is actually reasonable if you have more than, say, 3 or 4 "real"
> characters before the wildcard.
>
> <3> performs very poorly at any kind of scale.
>
> A search for "dog" would not match. A search for "fleas" wouldn't
> match. You see where this is going.
>
> If those restrictions are OK, just use the already-defined "string" type.
>
> As for the English/Chinese that's actually kind of a tough one.
> Splitting Chinese up into searchable tokens is nothing like breaking
> English up. There are examples in the managed-schema file that have
> field definitions for Chinese, but I know of no way to have a single
> field type shard the two different analysis chains. One solution
> people have used is to have a title_ch and title_en field and search
> both. Or search one or the other preferentially if the input is in one
> language or the other.
>
> I strongly advise you use the admin UI>>analysis page to understand
> the effects of tokenization, it's the heart of searching.
>
> Best,
> Erick
>
> On Wed, Jun 14, 2017 at 6:23 PM, @Nandan@
>  wrote:
> > Hi ,
> >
> > I am using Apache Solr for do advanced searching with my Big Data.
> >
> > When I am creating Solr core , then by default for text field , it is
> > coming as TextField data type and class.
> >
> > Can you please tell me how to change TextField to StrField. My table
> > contains record into English as well as Chinese .
> >
> > 
> >
> > 
> >
> >   
> >
> > 
> >
> >   
> >
> > 
> >
> > 
> >
> >   
> >
> > 
> >
> >  name="UUIDField"/>
> >
> >  > 

Re: Reg:- StrField Analyzer Issue

[Erick Erickson]

Back up a bit and tell us why you want to use StrField, because what
you're trying to do is somewhat confused.

First of all, StrFields are totally unanalyzed. So defining an
 as part of a StrField type definition is totally
unsupported. I'm a bit surprised that Solr even starts up.

Second, you can't search a StrField unless you search the whole thing
exactly. That is, if your title field is "My dog has fleas", there
only a few ways to match anything in that field

1> search "My dog has fleas" exactly. Even "my dog has fleas" wouldn't
match because of the capitalization. "My dog has fleas." would also
fail because of the period. StrField types are intended for data that
should be invariant and not tokenized.

2> prefix search as "My dog*"

3> pre-and-postfix as "*dog*"

<2> is actually reasonable if you have more than, say, 3 or 4 "real"
characters before the wildcard.

<3> performs very poorly at any kind of scale.

A search for "dog" would not match. A search for "fleas" wouldn't
match. You see where this is going.

If those restrictions are OK, just use the already-defined "string" type.

As for the English/Chinese that's actually kind of a tough one.
Splitting Chinese up into searchable tokens is nothing like breaking
English up. There are examples in the managed-schema file that have
field definitions for Chinese, but I know of no way to have a single
field type shard the two different analysis chains. One solution
people have used is to have a title_ch and title_en field and search
both. Or search one or the other preferentially if the input is in one
language or the other.

I strongly advise you use the admin UI>>analysis page to understand
the effects of tokenization, it's the heart of searching.

Best,
Erick

On Wed, Jun 14, 2017 at 6:23 PM, @Nandan@
 wrote:
> Hi ,
>
> I am using Apache Solr for do advanced searching with my Big Data.
>
> When I am creating Solr core , then by default for text field , it is
> coming as TextField data type and class.
>
> Can you please tell me how to change TextField to StrField. My table
> contains record into English as well as Chinese .
>
> 
>
> 
>
>   
>
> 
>
>   
>
> 
>
> 
>
>   
>
> 
>
> 
>
>  name="TrieIntField"/>
>
>   
>
>   
>
>  type="StrField"/>
>
>  type="StrField"/>
>
>  stored="true" type="StrField"/>
>
>  type="StrField"/>
>
>  stored="true" type="UUIDField"/>
>
>  stored="true" type="TrieIntField"/>
>
>   
>
>
> Please guide me for correct StrField.
>
> Thanks.

RE: Issue with highlighter

[Phil Scadden]

Just had similar issue - works for some, not others. First thing to look at is 
hl.maxAnalyzedChars is the query. The default is quite small.
Since many of my documents are large PDF files, I opted to use 
storeOffsetsWithPositions="true" termVectors="true" on the field I was 
searching on.
This certainly did increase my index size but not too bad and certainly fast.
https://cwiki.apache.org/confluence/display/solr/Highlighting

Beware of NOT plus OR in a search. That will certainly produce no highlights. 
(eg test -results when default op is OR)


-Original Message-
From: Ali Husain [mailto:alihus...@outlook.com]
Sent: Thursday, 15 June 2017 11:11 a.m.
To: solr-user@lucene.apache.org
Subject: Issue with highlighter

Hi,


I think I've found a bug with the highlighter. I search for the word 
"something" and I get an empty highlighting response for all the documents that 
are returned shown below. The fields that I am searching over are text_en, the 
highlighter works for a lot of queries. I have no stopwords.txt list that could 
be messing this up either.


 "highlighting":{
"310":{},
"103":{},
"406":{},
"1189":{},
"54":{},
"292":{},
"309":{}}}


Just changing the search term to "something like" I get back this:


"highlighting":{
"310":{},
"309":{
  "content":["1949 Convention, like those"]},
"103":{},
"406":{},
"1189":{},
"54":{},
"292":{},
"286":{
  "content":["persons in these classes are treated like 
combatants, but in other respects"]},
"336":{
  "content":["   be treated like engagement"]}}}


So I know that I have it setup correctly, but I can't figure this out. I've 
searched through JIRA/Google and haven't been able to find a similar issue.


Any ideas?


Thanks,

Ali
Notice: This email and any attachments are confidential and may not be used, 
published or redistributed without the prior written consent of the Institute 
of Geological and Nuclear Sciences Limited (GNS Science). If received in error 
please destroy and immediately notify GNS Science. Do not copy or disclose the 
contents.

CSV output

[Phil Scadden]

If I try
/getsolr? 
fl=id,title,datasource,score=true=9000=unified=Wainui-1=AND=csv

The response I get is:
id,title,datasource,scoreW:\PR_Reports\OCR\PR869.pdf,,Petroleum 
Reports,8.233313W:\PR_Reports\OCR\PR3440.pdf,,Petroleum 
Reports,8.217836W:\PR_Reports\OCR\PR4313.pdf,,Petroleum 
Reports,8.206703W:\PR_Reports\OCR\PR3906.pdf,,Petroleum 
Reports,8.185147W:\PR_Reports\OCR\PR1592.pdf,,Petroleum 
Reports,8.167614W:\PR_Reports\OCR\PR998.pdf,,Petroleum 
Reports,8.161142W:\PR_Reports\OCR\PR2457.pdf,,Petroleum 
Reports,8.155497W:\PR_Reports\OCR\PR2433.pdf,,Petroleum 
Reports,8.152924W:\PR_Reports\OCR\PR1184.pdf,,Petroleum 
Reports,8.124402W:\PR_Reports\OCR\PR3551.pdf,,Petroleum Reports,8.124402

ie no newline separators at all (Solr 6.5.1) (/getsolr is api that proxy to the 
solr server).
Changing it to
/getsolr?csv.newline=%0A=id,title,datasource,score=true=9000=unified=Wainui-1=AND=csv

Makes no difference. What I am doing wrong here? Is there another way to 
specify csv parameters? It says default is \n but I am not seeing that.

Notice: This email and any attachments are confidential and may not be used, 
published or redistributed without the prior written consent of the Institute 
of Geological and Nuclear Sciences Limited (GNS Science). If received in error 
please destroy and immediately notify GNS Science. Do not copy or disclose the 
contents.

Re: Multiple hashJoin or innerJoin

[Zheng Lin Edwin Yeo]

I have found that this is possible, but currently I have problems if the
field name to join in all the 3 collections are different.

For example, if in "people" collection, it is called personId, and in
"pets" collection, it is called petsId. But in "collectionId", it is called
collectionName, but it won't work when I place it this way below. Any
suggestions on how I can handle this?

innerJoin(innerJoin(
  search(people, q=*:*, fl="personId,name", sort="personId asc"),
  search(pets, q=type:cat, fl="pertsId,petName", sort="personId asc"),
  on="personId=petsId"
),
  search(collection1, q=*:*, fl="collectionId,collectionName",
sort="personId asc"),
)on="personId=collectionId"


Regards,
Edwin

On 14 June 2017 at 23:13, Zheng Lin Edwin Yeo  wrote:

> Hi,
>
> I'm using Solr 6.5.1.
>
> Is it possible to have multiple hashJoin or innerJoin in the query?
>
> An example will be something like this for innerJoin:
>
> innerJoin(innerJoin(
>   search(people, q=*:*, fl="personId,name", sort="personId asc"),
>   search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
>   on="personId"
> ),
>   search(collection1, q=*:*, fl="personId,personName", sort="personId
> asc"),
> )
>
> Regards,
> Edwin
>

Re: Solr 6: how to get SortedSetDocValues from index by field name

[Tomas Fernandez Lobbe]

Hi,
To respond your first question: “How do I get SortedSetDocValues from index by 
field name?”, DocValues.getSortedSet(LeafReader reader, String field) (which is 
what you want to use to assert the existence and type of the DV) will give you 
the dv instance for a single leaf reader. In general, a leaf reader is for a 
specific segment, so depending on what you want to do you may need to iterate 
through all the leaves (segments) if you want all values in the index (kind of 
what you’ll see in NumericFacets or IntervalFacets classes). 

SolrIndexSearcher.getSlowAtomicReader() will give you a view of all the 
segments as a single reader, that’s why in that case the code assumes there is 
only one reader that contains all the values. 

Whatever you do, make sure you test your code in cases with multiple segments 
(and with deletes), which is where bugs using this code are most likely to 
occur.

You won’t need the UninvertingReader if you plan to index docValues, that class 
is used to create a docValues-like view of a field that’s indexed=true & 
docValues=false.

Related note, the DocValues API changed from 6.x to 7 (master). See LUCENE-7407.

I hope that helps, 

Tomás

> On Jun 13, 2017, at 10:49 AM, SOLR4189  wrote:
> 
> How do I get SortedSetDocValues from index by field name?
> 
> I try it and it works for me but I didn't understand why to use
> leaves.get(0)? What does it mean? (I saw such using in
> TestUninvertedReader.java of SOLR-6.5.1):
> 
> *Map mapping = new HashMap<>();
> mapping.put(fieldName, UninvertingReader.Type.SORTED);
> 
> SolrIndexSearcher searcher = req.getSearcher();
> 
> DirectoryReader dReader = searcher.getIndexReader();
> LeafReader reader = null;
> 
> if (!dReader.leaves.isEmpty()) {
>  reader = dReader.leaves().get(0).reader;
>  return null;
> }
> 
> SortedSetDocValues sourceIndex = reader.getSortedSetDocValues(fieldName);*
> 
> Maybe do I need to use SlowAtomicReader, like it:
> 
> *
> UninvertingReader reader = new
> UninvertingReader(searcher.getSlowAtomicReader(), mapping)*;
> 
> What is right way to get SortedSetDocValues and why?
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-6-how-to-get-SortedSetDocValues-from-index-by-field-name-tp4340388.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Reg:- StrField Analyzer Issue

[@Nandan@]

Hi ,

I am using Apache Solr for do advanced searching with my Big Data.

When I am creating Solr core , then by default for text field , it is
coming as TextField data type and class.

Can you please tell me how to change TextField to StrField. My table
contains record into English as well as Chinese .





  



  





  







  

  













  


Please guide me for correct StrField.

Thanks.

Re: Swapping indexes on disk

[Erick Erickson]

Why not just use the replication API and fetchindex? See:
https://cwiki.apache.org/confluence/display/solr/Index+Replication#IndexReplication-HTTPAPICommandsfortheReplicationHandler.


It's not entirely obvious from the writeup, but you can specify
masterUrl as part of the command =some_other_solr_core.

So you have your "live" core, and your just-indexed core, call it "new".

You issue the fetchindex command to core on live and masterUrl points
to "new". It'll look something like
http://liveserver:8983/solr/live_core/replication?command=fetchindex=http://newserver:8983/solr/new_core

Solr will
1> copy the index from new to live
2> once that's done, open a new searcher and you're searching live
documents (after any autowarming you've configured).
3> delete the old index

Note that until <2>, incoming searches are served by the old index. So
the user sees no service interruptions at all.

If for any reason the fetch fails, the old index is left intact.

This will require that you have enough disk space to temporarily have
both the old and new index on the live server and there may be some
extra memory consumed while the old searcher is still open and the new
one is autowarming.

Best,
Erick

On Wed, Jun 14, 2017 at 4:10 PM, Mike Lissner
 wrote:
> I figured Solr would have a native system built in, but since we don't use
> it already, I didn't want to learn all of its ins and outs just for this
> disk situation.
>
> Ditto, essentially, applies for the swapping strategy. We don't have a Solr
> expert, just me, a generalist, and sorting out these kinds of things can
> take a while. The hope was to avoid that kind of complication with some
> clever use of symlinks and minor downtime. Our front end has a retry
> mechanism, so if solr is down for less than a minute, users will just have
> delayed responses, which is fine.
>
> The new strategy is to rsync the files while solr is live, stop solr, do a
> rsync diff, then start solr again. That'll give a bit for bit copy with
> very little downtime — it's the strategy postgres recommends for disk-based
> backups, so it seems like a safer bet. We needed a re-index anyway due to
> schema changes, which my first attempt included, but I guess that'll have
> to wait.
>
> Thanks for the replies. If anybody can explain why the first strategy
> failed, I'd still be interested in learning.
>
> Mike
>
> On Wed, Jun 14, 2017 at 12:09 PM Chris Ulicny  wrote:
>
>> Are you physically swapping the disks to introduce the new index? Or having
>> both disks mounted at the same time?
>>
>> If the disks are simultaneously available, can you just swap the cores and
>> then delete the core on the old disk?
>>
>> https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-SWAP
>>
>> We periodically move cores to different drives using solr's replication
>> functionality and core swapping (after stopping replication). However, I've
>> never encountered solr deleting an index like that.
>>
>>
>>
>> On Wed, Jun 14, 2017 at 2:48 PM David Hastings <
>> hastings.recurs...@gmail.com>
>> wrote:
>>
>> > I dont have an answer to why the folder got cleared, however i am
>> wondering
>> > why you arent using basic replication to do this exact same thing, since
>> > solr will natively take care of all this for you with no interruption to
>> > the user and no stop/start routines etc.
>> >
>> > On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
>> > mliss...@michaeljaylissner.com> wrote:
>> >
>> > > We are replacing a drive mounted at /old with one mounted at /new. Our
>> > > index currently lives on /old, and our plan was to:
>> > >
>> > > 1. Create a new index on /new
>> > > 2. Reindex from our database so that the new index on /new is properly
>> > > populated.
>> > > 3. Stop solr.
>> > > 4. Symlink /old to /new (Solr now looks for the index at /old/solr,
>> which
>> > > redirects to /new/solr)
>> > > 5. Start solr
>> > > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
>> > (Solr
>> > > now looks for the index at /old/solr again, and finds it there.)
>> > > 7. Delete the index pointing to /new created in step 1.
>> > >
>> > > The idea was that this would create a new index for solr, would
>> populate
>> > it
>> > > with the right content, and would avoid having to touch our existing
>> solr
>> > > configurations aside from creating one new index, which we could soon
>> > > delete.
>> > >
>> > > I just did steps 1-5, but I got null pointer exceptions when starting
>> > solr,
>> > > and it appears that the index on /new has been almost completely
>> deleted
>> > by
>> > > Solr (this is a bummer, since it takes days to populate).
>> > >
>> > > Is this expected? Am I terribly crazy to try to swap indexes on disk?
>> As
>> > > far as I know, the only difference between the indexes is their name.
>> > >
>> > > We're using Solr version 4.10.4.
>> > >
>> > > Thank you,
>> > >
>> > > Mike
>> > >
>> >
>>

Re: Issue with highlighter

[Erick Erickson]

If the default operator is OR, then you're just matching on the "like"
word and it's being properly highlighted. If you're saying that doc
286 (or whatever) has both "something" and "like" in the content and
you expect to find them both, try increasing the number of snippets
returned.

Otherwise we need to see the _complete_ query and response, preferably
with =true. Plus your schema, plus a sample document and exactly
what you think should be happening that isn't.

Best,
Erick

On Wed, Jun 14, 2017 at 4:11 PM, Ali Husain  wrote:
> Hi,
>
>
> I think I've found a bug with the highlighter. I search for the word 
> "something" and I get an empty highlighting response for all the documents 
> that are returned shown below. The fields that I am searching over are 
> text_en, the highlighter works for a lot of queries. I have no stopwords.txt 
> list that could be messing this up either.
>
>
>  "highlighting":{
> "310":{},
> "103":{},
> "406":{},
> "1189":{},
> "54":{},
> "292":{},
> "309":{}}}
>
>
> Just changing the search term to "something like" I get back this:
>
>
> "highlighting":{
> "310":{},
> "309":{
>   "content":["1949 Convention, like those"]},
> "103":{},
> "406":{},
> "1189":{},
> "54":{},
> "292":{},
> "286":{
>   "content":["persons in these classes are treated like 
> combatants, but in other respects"]},
> "336":{
>   "content":["   be treated like engagement"]}}}
>
>
> So I know that I have it setup correctly, but I can't figure this out. I've 
> searched through JIRA/Google and haven't been able to find a similar issue.
>
>
> Any ideas?
>
>
> Thanks,
>
> Ali

Re: Swapping indexes on disk

[Mike Lissner]

I figured Solr would have a native system built in, but since we don't use
it already, I didn't want to learn all of its ins and outs just for this
disk situation.

Ditto, essentially, applies for the swapping strategy. We don't have a Solr
expert, just me, a generalist, and sorting out these kinds of things can
take a while. The hope was to avoid that kind of complication with some
clever use of symlinks and minor downtime. Our front end has a retry
mechanism, so if solr is down for less than a minute, users will just have
delayed responses, which is fine.

The new strategy is to rsync the files while solr is live, stop solr, do a
rsync diff, then start solr again. That'll give a bit for bit copy with
very little downtime — it's the strategy postgres recommends for disk-based
backups, so it seems like a safer bet. We needed a re-index anyway due to
schema changes, which my first attempt included, but I guess that'll have
to wait.

Thanks for the replies. If anybody can explain why the first strategy
failed, I'd still be interested in learning.

Mike

On Wed, Jun 14, 2017 at 12:09 PM Chris Ulicny  wrote:

> Are you physically swapping the disks to introduce the new index? Or having
> both disks mounted at the same time?
>
> If the disks are simultaneously available, can you just swap the cores and
> then delete the core on the old disk?
>
> https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-SWAP
>
> We periodically move cores to different drives using solr's replication
> functionality and core swapping (after stopping replication). However, I've
> never encountered solr deleting an index like that.
>
>
>
> On Wed, Jun 14, 2017 at 2:48 PM David Hastings <
> hastings.recurs...@gmail.com>
> wrote:
>
> > I dont have an answer to why the folder got cleared, however i am
> wondering
> > why you arent using basic replication to do this exact same thing, since
> > solr will natively take care of all this for you with no interruption to
> > the user and no stop/start routines etc.
> >
> > On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
> > mliss...@michaeljaylissner.com> wrote:
> >
> > > We are replacing a drive mounted at /old with one mounted at /new. Our
> > > index currently lives on /old, and our plan was to:
> > >
> > > 1. Create a new index on /new
> > > 2. Reindex from our database so that the new index on /new is properly
> > > populated.
> > > 3. Stop solr.
> > > 4. Symlink /old to /new (Solr now looks for the index at /old/solr,
> which
> > > redirects to /new/solr)
> > > 5. Start solr
> > > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
> > (Solr
> > > now looks for the index at /old/solr again, and finds it there.)
> > > 7. Delete the index pointing to /new created in step 1.
> > >
> > > The idea was that this would create a new index for solr, would
> populate
> > it
> > > with the right content, and would avoid having to touch our existing
> solr
> > > configurations aside from creating one new index, which we could soon
> > > delete.
> > >
> > > I just did steps 1-5, but I got null pointer exceptions when starting
> > solr,
> > > and it appears that the index on /new has been almost completely
> deleted
> > by
> > > Solr (this is a bummer, since it takes days to populate).
> > >
> > > Is this expected? Am I terribly crazy to try to swap indexes on disk?
> As
> > > far as I know, the only difference between the indexes is their name.
> > >
> > > We're using Solr version 4.10.4.
> > >
> > > Thank you,
> > >
> > > Mike
> > >
> >
>

Issue with highlighter

[Ali Husain]

Hi,


I think I've found a bug with the highlighter. I search for the word 
"something" and I get an empty highlighting response for all the documents that 
are returned shown below. The fields that I am searching over are text_en, the 
highlighter works for a lot of queries. I have no stopwords.txt list that could 
be messing this up either.


 "highlighting":{
"310":{},
"103":{},
"406":{},
"1189":{},
"54":{},
"292":{},
"309":{}}}


Just changing the search term to "something like" I get back this:


"highlighting":{
"310":{},
"309":{
  "content":["1949 Convention, like those"]},
"103":{},
"406":{},
"1189":{},
"54":{},
"292":{},
"286":{
  "content":["persons in these classes are treated like 
combatants, but in other respects"]},
"336":{
  "content":["   be treated like engagement"]}}}


So I know that I have it setup correctly, but I can't figure this out. I've 
searched through JIRA/Google and haven't been able to find a similar issue.


Any ideas?


Thanks,

Ali

Re: Can't upload pdf file to example Core

[Susheel Kumar]

Try using the curl command directly on terminal/console and it will work. I
just tried on 6.6 on a mac.  The upload thru UI would not work for PDF's
unless more parameters are provided.  The upload thru UI though works
directly for  XML/JSON files etc.

curl '
http://localhost:8983/solr/techproducts/update/extract?literal.id=doc1=true'
-F
"myfile=@example/exampledocs/solr-word.pdf"


On Wed, Jun 14, 2017 at 1:30 PM, Vasiliy Boldyrev <
vasiliy.boldy...@gmail.com> wrote:

>  Hello,
>
>  I used Apache Solr™ version 6.6.0 but can't upload pdf file to Core
>
>  Instruction and Example has been get from
> https://cwiki.apache.org/confluence/display/solr/
> Uploading+Data+with+Solr+Cell+using+Apache+Tika
>
>  Add to solconfig.xml additional path to /dist/ and /contrib/extraction jar
> files.
>  ".*\.jar" />
>  />
>
>  Change requestHandler with name=/update/extract:
>  add last_modified but did not add
> optional parameter "tika.config" to requestHandler
>
>  From web interface try Upload pdf doc to Core "techproducts" from example
> but received error - "Unsupported ContentType: application/pdf"
>
>  http://localhost:8983/solr/#/techproducts/documents -> Document type -
> File Upload - Choose  solr-word.pdf and received error
>
>  "Unsupported ContentType: application/pdf Not in: [application/xml,
> application/csv, application/json, text/json, text/csv, text/xml,
> application/javabin]"
>
>  From Core log file:
>  ERROR - 2017-06-14 17:19:01.190; [   x:techproducts]
> org.apache.solr.common.SolrException; org.apache.solr.common.
> SolrException:
> Unsupported ContentType: application/pdf  Not in: [application/xml,
> application/csv, application/json, text/json, text/csv, text/xml,
> application/javabin]
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(
> UpdateRequestHandler.java:90)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(
> ContentStreamHandlerBase.java:68)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(
> ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
> RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(
> ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:589)
> at java.lang.Thread.run(Unknown Source)
>
>
> Try used curl utilite for uploading pdf file but receive same error:
>
> C:\install\solr-6.6.0\example\exampledocs>curl.exe
> 

Re: Parallel SQL - column not found intermittent error

[Susheel Kumar]

Created JIRA  https://issues.apache.org/jira/browse/SOLR-10890

Thank you.

On Wed, Jun 14, 2017 at 1:59 PM, Joel Bernstein  wrote:

> Let's create a jira for this.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, Jun 14, 2017 at 12:26 PM, Susheel Kumar 
> wrote:
>
> > Thanks, Yury.  Indeed that is the issue.
> >
> > Joel,  is that something expected behavior or should i create a JIRA?
> >
> > Thanks,
> > Susheel
> >
> > On Wed, Jun 14, 2017 at 12:16 PM, Yury Kats 
> > wrote:
> >
> > > I have seen this with very few indexed documents and multiple shards.
> > > In such a case, some shards may not have any documents, and when the
> > query
> > > happens to hit such a shard, it does not find the fields it's looking
> for
> > > and turns this into "column not found". If you resubmit the query and
> hit
> > > a different shards (with docs), the query will succeed.
> > >
> > > On 6/14/2017 11:42 AM, Susheel Kumar wrote:
> > > > Yes, Joel.  Kind of every other command runs into this issue. I just
> > > > executed below queries and 3 of them failed while 1 succeeded.   I
> just
> > > > have 6 documents ingested and no further indexing going on.  Let me
> > know
> > > > what else to look for the state of index.
> > > >
> > > >
> > > > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName
> > as
> > > > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > > > dv_sv_userLastName LIMIT 15'
> > > > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > > >
> > > >
> > > > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery
> > 'SELECT
> > > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName
> FROM
> > > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
> > connection
> > > > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> > > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName
> FROM
> > > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1,
> > column 9
> > > > to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> > > > table","EOF":true,"RESPONSE_TIME":85}]}}
> > > >
> > > >
> > > > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName
> > as
> > > > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > > > dv_sv_userLastName LIMIT 15'
> > > > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > > >
> > > >
> > > > {"result-set":{"docs":[{"firstName":"Thiago","lastName"
> > > :"Diego"},{"firstName":"John","lastName":"Jagger"},{"
> > > firstName":"John","lastName":"Jagger"},{"firstName":"John","
> > > lastName":"Johny"},{"firstName":"Isabel","lastName"
> > > :"Margret"},{"firstName":"Isabel","lastName":"Margret"},
> > > {"EOF":true,"RESPONSE_TIME":241}]}}
> > > >
> > > >
> > > > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName
> > as
> > > > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > > > dv_sv_userLastName LIMIT 15'
> > > > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > > >
> > > >
> > > >
> > > > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery
> > 'SELECT
> > > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName
> FROM
> > > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
> > connection
> > > > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> > > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName
> FROM
> > > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1,
> > column 9
> > > > to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> > > > table","EOF":true,"RESPONSE_TIME":87}]}}
> > > >
> > > > On Wed, Jun 14, 2017 at 11:18 AM, Joel Bernstein  >
> > > wrote:
> > > >
> > > >> Are you able to reproduce the error, or is it just appearing in the
> > > logs?
> > > >>
> > > >> Do you know the state of index when it's occurring?
> > > >>
> > > >> Joel Bernstein
> > > >> http://joelsolr.blogspot.com/
> > > >>
> > > >> On Wed, Jun 14, 2017 at 11:09 AM, Susheel Kumar <
> > susheel2...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> I have setup Solr-6.6-0 on local (local ZK and Solr) and then on
> > > servers
> > > >> (3
> > > >>> ZK and 2 machines, 2 shards) and on both the env,  i see this
> > > >> intermittent
> > > >>> error "column not found". The same query works sometime and other
> > time
> > > >>> fails.
> > > >>>
> > > >>> Is that a bug or am I missing something...
> > > >>>
> > > >>>
> > > >>> Console
> > > >>> ===
> > > >>>
> > > >>> -> solr-6.6.0 curl --data-urlencode 'stmt=SELECT
> dv_sv_userFirstName
> > > as
> > > >>> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> > > >>> dv_sv_userLastName LIMIT 15'
> > > >>> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > > >>>
> > > >>> 

Re: Swapping indexes on disk

[Chris Ulicny]

Are you physically swapping the disks to introduce the new index? Or having
both disks mounted at the same time?

If the disks are simultaneously available, can you just swap the cores and
then delete the core on the old disk?
https://cwiki.apache.org/confluence/display/solr/CoreAdmin+API#CoreAdminAPI-SWAP

We periodically move cores to different drives using solr's replication
functionality and core swapping (after stopping replication). However, I've
never encountered solr deleting an index like that.



On Wed, Jun 14, 2017 at 2:48 PM David Hastings 
wrote:

> I dont have an answer to why the folder got cleared, however i am wondering
> why you arent using basic replication to do this exact same thing, since
> solr will natively take care of all this for you with no interruption to
> the user and no stop/start routines etc.
>
> On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
> mliss...@michaeljaylissner.com> wrote:
>
> > We are replacing a drive mounted at /old with one mounted at /new. Our
> > index currently lives on /old, and our plan was to:
> >
> > 1. Create a new index on /new
> > 2. Reindex from our database so that the new index on /new is properly
> > populated.
> > 3. Stop solr.
> > 4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
> > redirects to /new/solr)
> > 5. Start solr
> > 6. (Later) Stop solr, swap the drives (old for new), and start solr.
> (Solr
> > now looks for the index at /old/solr again, and finds it there.)
> > 7. Delete the index pointing to /new created in step 1.
> >
> > The idea was that this would create a new index for solr, would populate
> it
> > with the right content, and would avoid having to touch our existing solr
> > configurations aside from creating one new index, which we could soon
> > delete.
> >
> > I just did steps 1-5, but I got null pointer exceptions when starting
> solr,
> > and it appears that the index on /new has been almost completely deleted
> by
> > Solr (this is a bummer, since it takes days to populate).
> >
> > Is this expected? Am I terribly crazy to try to swap indexes on disk? As
> > far as I know, the only difference between the indexes is their name.
> >
> > We're using Solr version 4.10.4.
> >
> > Thank you,
> >
> > Mike
> >
>

Re: Swapping indexes on disk

[David Hastings]

I dont have an answer to why the folder got cleared, however i am wondering
why you arent using basic replication to do this exact same thing, since
solr will natively take care of all this for you with no interruption to
the user and no stop/start routines etc.

On Wed, Jun 14, 2017 at 2:26 PM, Mike Lissner <
mliss...@michaeljaylissner.com> wrote:

> We are replacing a drive mounted at /old with one mounted at /new. Our
> index currently lives on /old, and our plan was to:
>
> 1. Create a new index on /new
> 2. Reindex from our database so that the new index on /new is properly
> populated.
> 3. Stop solr.
> 4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
> redirects to /new/solr)
> 5. Start solr
> 6. (Later) Stop solr, swap the drives (old for new), and start solr. (Solr
> now looks for the index at /old/solr again, and finds it there.)
> 7. Delete the index pointing to /new created in step 1.
>
> The idea was that this would create a new index for solr, would populate it
> with the right content, and would avoid having to touch our existing solr
> configurations aside from creating one new index, which we could soon
> delete.
>
> I just did steps 1-5, but I got null pointer exceptions when starting solr,
> and it appears that the index on /new has been almost completely deleted by
> Solr (this is a bummer, since it takes days to populate).
>
> Is this expected? Am I terribly crazy to try to swap indexes on disk? As
> far as I know, the only difference between the indexes is their name.
>
> We're using Solr version 4.10.4.
>
> Thank you,
>
> Mike
>

Can't upload pdf file to example Core

[Vasiliy Boldyrev]

 Hello,

 I used Apache Solr™ version 6.6.0 but can't upload pdf file to Core

 Instruction and Example has been get from
https://cwiki.apache.org/confluence/display/solr/Uploading+Data+with+Solr+Cell+using+Apache+Tika

 Add to solconfig.xml additional path to /dist/ and /contrib/extraction jar
files.



 Change requestHandler with name=/update/extract:
 add last_modified but did not add
optional parameter "tika.config" to requestHandler

 From web interface try Upload pdf doc to Core "techproducts" from example
but received error - "Unsupported ContentType: application/pdf"

 http://localhost:8983/solr/#/techproducts/documents -> Document type -
File Upload - Choose  solr-word.pdf and received error

 "Unsupported ContentType: application/pdf Not in: [application/xml,
application/csv, application/json, text/json, text/csv, text/xml,
application/javabin]"

 From Core log file:
 ERROR - 2017-06-14 17:19:01.190; [   x:techproducts]
org.apache.solr.common.SolrException; org.apache.solr.common.SolrException:
Unsupported ContentType: application/pdf  Not in: [application/xml,
application/csv, application/json, text/json, text/csv, text/xml,
application/javabin]
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:90)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Unknown Source)


Try used curl utilite for uploading pdf file but receive same error:

C:\install\solr-6.6.0\example\exampledocs>curl.exe
http://localhost:8983/solr/techproducts/update/extract?literal.id=doc1=true
-F solr-word.pdf


40031org.apache.solr.common.SolrExceptionorg.apache.solr.common.SolrExceptionmissing content stream400

'commit' is not recognized as an internal or external command, operable
program or batch file.

BR, Vasily Boldyrev

Swapping indexes on disk

[Mike Lissner]

We are replacing a drive mounted at /old with one mounted at /new. Our
index currently lives on /old, and our plan was to:

1. Create a new index on /new
2. Reindex from our database so that the new index on /new is properly
populated.
3. Stop solr.
4. Symlink /old to /new (Solr now looks for the index at /old/solr, which
redirects to /new/solr)
5. Start solr
6. (Later) Stop solr, swap the drives (old for new), and start solr. (Solr
now looks for the index at /old/solr again, and finds it there.)
7. Delete the index pointing to /new created in step 1.

The idea was that this would create a new index for solr, would populate it
with the right content, and would avoid having to touch our existing solr
configurations aside from creating one new index, which we could soon
delete.

I just did steps 1-5, but I got null pointer exceptions when starting solr,
and it appears that the index on /new has been almost completely deleted by
Solr (this is a bummer, since it takes days to populate).

Is this expected? Am I terribly crazy to try to swap indexes on disk? As
far as I know, the only difference between the indexes is their name.

We're using Solr version 4.10.4.

Thank you,

Mike

Re: Parallel SQL - column not found intermittent error

[Joel Bernstein]

Let's create a jira for this.

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, Jun 14, 2017 at 12:26 PM, Susheel Kumar 
wrote:

> Thanks, Yury.  Indeed that is the issue.
>
> Joel,  is that something expected behavior or should i create a JIRA?
>
> Thanks,
> Susheel
>
> On Wed, Jun 14, 2017 at 12:16 PM, Yury Kats 
> wrote:
>
> > I have seen this with very few indexed documents and multiple shards.
> > In such a case, some shards may not have any documents, and when the
> query
> > happens to hit such a shard, it does not find the fields it's looking for
> > and turns this into "column not found". If you resubmit the query and hit
> > a different shards (with docs), the query will succeed.
> >
> > On 6/14/2017 11:42 AM, Susheel Kumar wrote:
> > > Yes, Joel.  Kind of every other command runs into this issue. I just
> > > executed below queries and 3 of them failed while 1 succeeded.   I just
> > > have 6 documents ingested and no further indexing going on.  Let me
> know
> > > what else to look for the state of index.
> > >
> > >
> > > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName
> as
> > > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > > dv_sv_userLastName LIMIT 15'
> > > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > >
> > >
> > > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery
> 'SELECT
> > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
> connection
> > > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1,
> column 9
> > > to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> > > table","EOF":true,"RESPONSE_TIME":85}]}}
> > >
> > >
> > > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName
> as
> > > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > > dv_sv_userLastName LIMIT 15'
> > > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > >
> > >
> > > {"result-set":{"docs":[{"firstName":"Thiago","lastName"
> > :"Diego"},{"firstName":"John","lastName":"Jagger"},{"
> > firstName":"John","lastName":"Jagger"},{"firstName":"John","
> > lastName":"Johny"},{"firstName":"Isabel","lastName"
> > :"Margret"},{"firstName":"Isabel","lastName":"Margret"},
> > {"EOF":true,"RESPONSE_TIME":241}]}}
> > >
> > >
> > > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName
> as
> > > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > > dv_sv_userLastName LIMIT 15'
> > > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > >
> > >
> > >
> > > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery
> 'SELECT
> > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
> connection
> > > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> > >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1,
> column 9
> > > to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> > > table","EOF":true,"RESPONSE_TIME":87}]}}
> > >
> > > On Wed, Jun 14, 2017 at 11:18 AM, Joel Bernstein 
> > wrote:
> > >
> > >> Are you able to reproduce the error, or is it just appearing in the
> > logs?
> > >>
> > >> Do you know the state of index when it's occurring?
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >> On Wed, Jun 14, 2017 at 11:09 AM, Susheel Kumar <
> susheel2...@gmail.com>
> > >> wrote:
> > >>
> > >>> I have setup Solr-6.6-0 on local (local ZK and Solr) and then on
> > servers
> > >> (3
> > >>> ZK and 2 machines, 2 shards) and on both the env,  i see this
> > >> intermittent
> > >>> error "column not found". The same query works sometime and other
> time
> > >>> fails.
> > >>>
> > >>> Is that a bug or am I missing something...
> > >>>
> > >>>
> > >>> Console
> > >>> ===
> > >>>
> > >>> -> solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName
> > as
> > >>> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> > >>> dv_sv_userLastName LIMIT 15'
> > >>> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> > >>>
> > >>> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery
> > 'SELECT
> > >>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName
> FROM
> > >>> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
> > connection
> > >>> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> > >>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName
> FROM
> > >>> collection1 ORDEr BY 

Re: Out of Memory Errors

[Susheel Kumar]

The attachment will not come thru.  Can you upload thru dropbox / other
sharing sites etc.

On Wed, Jun 14, 2017 at 12:41 PM, Satya Marivada 
wrote:

> Susheel, Please see attached. There  heap towards the end of graph has
> spiked
>
>
>
> On Wed, Jun 14, 2017 at 11:46 AM Susheel Kumar 
> wrote:
>
>> You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
>> or so and share.
>>
>> Thnx
>>
>> On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada <
>> satya.chaita...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I am getting Out of Memory Errors after a while on solr-6.3.0.
>> > The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/
>> oom_solr.sh
>> > just kills the jvm right after.
>> > Using Jconsole, I see the nice triangle pattern, where it uses the heap
>> > and being reclaimed back.
>> >
>> > The heap size is set at 3g. The index size hosted on that particular
>> node
>> > is 17G.
>> >
>> > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
>> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
>> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
>> > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
>> > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
>> > CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
>> >
>> > Looking at the solr_gc.log.0, the eden space is being used 100% all the
>> > while and being successfully reclaimed. So don't think that has go to do
>> > with it.
>> >
>> > Apart from that in the solr.log, I see exceptions that are aftermath of
>> > killing the jvm
>> >
>> > org.eclipse.jetty.io.EofException: Closed
>> > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.
>> java:383)
>> > at org.apache.commons.io.output.ProxyOutputStream.write(
>> > ProxyOutputStream.java:90)
>> > at org.apache.solr.common.util.FastOutputStream.flush(
>> > FastOutputStream.java:213)
>> > at org.apache.solr.common.util.FastOutputStream.flushBuffer(
>> > FastOutputStream.java:206)
>> > at org.apache.solr.common.util.JavaBinCodec.marshal(
>> > JavaBinCodec.java:136)
>> >
>> > Any suggestions on how to go about it.
>> >
>> > Thanks,
>> > Satya
>> >
>>
>

Re: Out of Memory Errors

[Satya Marivada]

Susheel, Please see attached. There  heap towards the end of graph has
spiked



On Wed, Jun 14, 2017 at 11:46 AM Susheel Kumar 
wrote:

> You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
> or so and share.
>
> Thnx
>
> On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada <
> satya.chaita...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am getting Out of Memory Errors after a while on solr-6.3.0.
> > The
> -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
> > just kills the jvm right after.
> > Using Jconsole, I see the nice triangle pattern, where it uses the heap
> > and being reclaimed back.
> >
> > The heap size is set at 3g. The index size hosted on that particular node
> > is 17G.
> >
> > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
> > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
> > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
> > CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
> >
> > Looking at the solr_gc.log.0, the eden space is being used 100% all the
> > while and being successfully reclaimed. So don't think that has go to do
> > with it.
> >
> > Apart from that in the solr.log, I see exceptions that are aftermath of
> > killing the jvm
> >
> > org.eclipse.jetty.io.EofException: Closed
> > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
> > at org.apache.commons.io.output.ProxyOutputStream.write(
> > ProxyOutputStream.java:90)
> > at org.apache.solr.common.util.FastOutputStream.flush(
> > FastOutputStream.java:213)
> > at org.apache.solr.common.util.FastOutputStream.flushBuffer(
> > FastOutputStream.java:206)
> > at org.apache.solr.common.util.JavaBinCodec.marshal(
> > JavaBinCodec.java:136)
> >
> > Any suggestions on how to go about it.
> >
> > Thanks,
> > Satya
> >
>

Re: Parallel SQL - column not found intermittent error

[Susheel Kumar]

Thanks, Yury.  Indeed that is the issue.

Joel,  is that something expected behavior or should i create a JIRA?

Thanks,
Susheel

On Wed, Jun 14, 2017 at 12:16 PM, Yury Kats 
wrote:

> I have seen this with very few indexed documents and multiple shards.
> In such a case, some shards may not have any documents, and when the query
> happens to hit such a shard, it does not find the fields it's looking for
> and turns this into "column not found". If you resubmit the query and hit
> a different shards (with docs), the query will succeed.
>
> On 6/14/2017 11:42 AM, Susheel Kumar wrote:
> > Yes, Joel.  Kind of every other command runs into this issue. I just
> > executed below queries and 3 of them failed while 1 succeeded.   I just
> > have 6 documents ingested and no further indexing going on.  Let me know
> > what else to look for the state of index.
> >
> >
> > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > dv_sv_userLastName LIMIT 15'
> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> >
> >
> > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
> >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> > to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> > table","EOF":true,"RESPONSE_TIME":85}]}}
> >
> >
> > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > dv_sv_userLastName LIMIT 15'
> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> >
> >
> > {"result-set":{"docs":[{"firstName":"Thiago","lastName"
> :"Diego"},{"firstName":"John","lastName":"Jagger"},{"
> firstName":"John","lastName":"Jagger"},{"firstName":"John","
> lastName":"Johny"},{"firstName":"Isabel","lastName"
> :"Margret"},{"firstName":"Isabel","lastName":"Margret"},
> {"EOF":true,"RESPONSE_TIME":241}]}}
> >
> >
> > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> > firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> > dv_sv_userLastName LIMIT 15'
> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> >
> >
> >
> > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
> >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> >  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> > to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> > table","EOF":true,"RESPONSE_TIME":87}]}}
> >
> > On Wed, Jun 14, 2017 at 11:18 AM, Joel Bernstein 
> wrote:
> >
> >> Are you able to reproduce the error, or is it just appearing in the
> logs?
> >>
> >> Do you know the state of index when it's occurring?
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >> On Wed, Jun 14, 2017 at 11:09 AM, Susheel Kumar 
> >> wrote:
> >>
> >>> I have setup Solr-6.6-0 on local (local ZK and Solr) and then on
> servers
> >> (3
> >>> ZK and 2 machines, 2 shards) and on both the env,  i see this
> >> intermittent
> >>> error "column not found". The same query works sometime and other time
> >>> fails.
> >>>
> >>> Is that a bug or am I missing something...
> >>>
> >>>
> >>> Console
> >>> ===
> >>>
> >>> -> solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName
> as
> >>> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> >>> dv_sv_userLastName LIMIT 15'
> >>> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> >>>
> >>> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery
> 'SELECT
> >>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> >>> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
> connection
> >>> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> >>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> >>> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1,
> column 9
> >>> to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
> >>> table","EOF":true,"RESPONSE_TIME":78}]}}
> >>>
> >>> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName
> as
> >>> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> >>> dv_sv_userLastName LIMIT 15'
> >>> 

Re: Parallel SQL - column not found intermittent error

[Yury Kats]

I have seen this with very few indexed documents and multiple shards.
In such a case, some shards may not have any documents, and when the query
happens to hit such a shard, it does not find the fields it's looking for
and turns this into "column not found". If you resubmit the query and hit
a different shards (with docs), the query will succeed.

On 6/14/2017 11:42 AM, Susheel Kumar wrote:
> Yes, Joel.  Kind of every other command runs into this issue. I just
> executed below queries and 3 of them failed while 1 succeeded.   I just
> have 6 documents ingested and no further indexing going on.  Let me know
> what else to look for the state of index.
> 
> 
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15'
> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> 
> 
> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> table","EOF":true,"RESPONSE_TIME":85}]}}
> 
> 
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15'
> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> 
> 
> {"result-set":{"docs":[{"firstName":"Thiago","lastName":"Diego"},{"firstName":"John","lastName":"Jagger"},{"firstName":"John","lastName":"Jagger"},{"firstName":"John","lastName":"Johny"},{"firstName":"Isabel","lastName":"Margret"},{"firstName":"Isabel","lastName":"Margret"},{"EOF":true,"RESPONSE_TIME":241}]}}
> 
> 
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15'
> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> 
> 
> 
> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> table","EOF":true,"RESPONSE_TIME":87}]}}
> 
> On Wed, Jun 14, 2017 at 11:18 AM, Joel Bernstein  wrote:
> 
>> Are you able to reproduce the error, or is it just appearing in the logs?
>>
>> Do you know the state of index when it's occurring?
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Wed, Jun 14, 2017 at 11:09 AM, Susheel Kumar 
>> wrote:
>>
>>> I have setup Solr-6.6-0 on local (local ZK and Solr) and then on servers
>> (3
>>> ZK and 2 machines, 2 shards) and on both the env,  i see this
>> intermittent
>>> error "column not found". The same query works sometime and other time
>>> fails.
>>>
>>> Is that a bug or am I missing something...
>>>
>>>
>>> Console
>>> ===
>>>
>>> -> solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
>>> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
>>> dv_sv_userLastName LIMIT 15'
>>> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>>>
>>> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>>> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
>>> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>>> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
>>> to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
>>> table","EOF":true,"RESPONSE_TIME":78}]}}
>>>
>>> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
>>> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
>>> dv_sv_userLastName LIMIT 15'
>>> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>>>
>>> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>>> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
>>> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>>>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>>> collection1 ORDEr BY 

Re: Out of Memory Errors

[Susheel Kumar]

You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
or so and share.

Thnx

On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada 
wrote:

> Hi,
>
> I am getting Out of Memory Errors after a while on solr-6.3.0.
> The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
> just kills the jvm right after.
> Using Jconsole, I see the nice triangle pattern, where it uses the heap
> and being reclaimed back.
>
> The heap size is set at 3g. The index size hosted on that particular node
> is 17G.
>
> java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
> -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
> -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
> CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
>
> Looking at the solr_gc.log.0, the eden space is being used 100% all the
> while and being successfully reclaimed. So don't think that has go to do
> with it.
>
> Apart from that in the solr.log, I see exceptions that are aftermath of
> killing the jvm
>
> org.eclipse.jetty.io.EofException: Closed
> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
> at org.apache.commons.io.output.ProxyOutputStream.write(
> ProxyOutputStream.java:90)
> at org.apache.solr.common.util.FastOutputStream.flush(
> FastOutputStream.java:213)
> at org.apache.solr.common.util.FastOutputStream.flushBuffer(
> FastOutputStream.java:206)
> at org.apache.solr.common.util.JavaBinCodec.marshal(
> JavaBinCodec.java:136)
>
> Any suggestions on how to go about it.
>
> Thanks,
> Satya
>

Re: Parallel SQL - column not found intermittent error

[Susheel Kumar]

Also i tried with both docValues and non docValue fields/column.

On Wed, Jun 14, 2017 at 11:42 AM, Susheel Kumar 
wrote:

> Yes, Joel.  Kind of every other command runs into this issue. I just
> executed below queries and 3 of them failed while 1 succeeded.   I just
> have 6 documents ingested and no further indexing going on.  Let me know
> what else to look for the state of index.
>
>
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15' http://server17:8984/solr/collection1/sql\?
> aggregationMode\=facet
>
>
> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> table","EOF":true,"RESPONSE_TIME":85}]}}
>
>
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15' http://server17:8984/solr/collection1/sql\?
> aggregationMode\=facet
>
>
> {"result-set":{"docs":[{"firstName":"Thiago","lastName"
> :"Diego"},{"firstName":"John","lastName":"Jagger"},{"
> firstName":"John","lastName":"Jagger"},{"firstName":"John","
> lastName":"Johny"},{"firstName":"Isabel","lastName"
> :"Margret"},{"firstName":"Isabel","lastName":"Margret"},
> {"EOF":true,"RESPONSE_TIME":241}]}}
>
>
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
> firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15' http://server17:8984/solr/collection1/sql\?
> aggregationMode\=facet
>
>
>
> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>  sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
> table","EOF":true,"RESPONSE_TIME":87}]}}
>
> On Wed, Jun 14, 2017 at 11:18 AM, Joel Bernstein 
> wrote:
>
>> Are you able to reproduce the error, or is it just appearing in the logs?
>>
>> Do you know the state of index when it's occurring?
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> On Wed, Jun 14, 2017 at 11:09 AM, Susheel Kumar 
>> wrote:
>>
>> > I have setup Solr-6.6-0 on local (local ZK and Solr) and then on
>> servers (3
>> > ZK and 2 machines, 2 shards) and on both the env,  i see this
>> intermittent
>> > error "column not found". The same query works sometime and other time
>> > fails.
>> >
>> > Is that a bug or am I missing something...
>> >
>> >
>> > Console
>> > ===
>> >
>> > -> solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
>> > firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
>> > dv_sv_userLastName LIMIT 15'
>> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>> >
>> > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
>> connection
>> > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column
>> 9
>> > to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
>> > table","EOF":true,"RESPONSE_TIME":78}]}}
>> >
>> > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
>> > firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
>> > dv_sv_userLastName LIMIT 15'
>> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>> >
>> > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC
>> connection
>> > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
>> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column
>> 9
>> > to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
>> > table","EOF":true,"RESPONSE_TIME":92}]}}
>> >
>> > ➜  solr-6.6.0 

Re: Parallel SQL - column not found intermittent error

[Susheel Kumar]

Yes, Joel.  Kind of every other command runs into this issue. I just
executed below queries and 3 of them failed while 1 succeeded.   I just
have 6 documents ingested and no further indexing going on.  Let me know
what else to look for the state of index.


➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
dv_sv_userLastName LIMIT 15'
http://server17:8984/solr/collection1/sql\?aggregationMode\=facet


{"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
 sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
 sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
table","EOF":true,"RESPONSE_TIME":85}]}}


➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
dv_sv_userLastName LIMIT 15'
http://server17:8984/solr/collection1/sql\?aggregationMode\=facet


{"result-set":{"docs":[{"firstName":"Thiago","lastName":"Diego"},{"firstName":"John","lastName":"Jagger"},{"firstName":"John","lastName":"Jagger"},{"firstName":"John","lastName":"Johny"},{"firstName":"Isabel","lastName":"Margret"},{"firstName":"Isabel","lastName":"Margret"},{"EOF":true,"RESPONSE_TIME":241}]}}


➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  sr_sv_userFirstName as
firstName, sr_sv_userLastName as lastName FROM collection1 ORDEr BY
dv_sv_userLastName LIMIT 15'
http://server17:8984/solr/collection1/sql\?aggregationMode\=facet



{"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
 sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
 sr_sv_userFirstName as firstName, sr_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
to line 1, column 27: Column 'sr_sv_userFirstName' not found in any
table","EOF":true,"RESPONSE_TIME":87}]}}

On Wed, Jun 14, 2017 at 11:18 AM, Joel Bernstein  wrote:

> Are you able to reproduce the error, or is it just appearing in the logs?
>
> Do you know the state of index when it's occurring?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Wed, Jun 14, 2017 at 11:09 AM, Susheel Kumar 
> wrote:
>
> > I have setup Solr-6.6-0 on local (local ZK and Solr) and then on servers
> (3
> > ZK and 2 machines, 2 shards) and on both the env,  i see this
> intermittent
> > error "column not found". The same query works sometime and other time
> > fails.
> >
> > Is that a bug or am I missing something...
> >
> >
> > Console
> > ===
> >
> > -> solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
> > firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> > dv_sv_userLastName LIMIT 15'
> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> >
> > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> > to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
> > table","EOF":true,"RESPONSE_TIME":78}]}}
> >
> > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
> > firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> > dv_sv_userLastName LIMIT 15'
> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> >
> > {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> > 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
> >  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> > collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> > to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
> > table","EOF":true,"RESPONSE_TIME":92}]}}
> >
> > ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
> > firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> > dv_sv_userLastName LIMIT 15'
> > http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
> >
> > {"result-set":{"docs":[{"firstName":"Thiago","lastName"
> > 

Out of Memory Errors

[Satya Marivada]

Hi,

I am getting Out of Memory Errors after a while on solr-6.3.0.
The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
just kills the jvm right after.
Using Jconsole, I see the nice triangle pattern, where it uses the heap and
being reclaimed back.

The heap size is set at 3g. The index size hosted on that particular node
is 17G.

java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
-XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000

Looking at the solr_gc.log.0, the eden space is being used 100% all the
while and being successfully reclaimed. So don't think that has go to do
with it.

Apart from that in the solr.log, I see exceptions that are aftermath of
killing the jvm

org.eclipse.jetty.io.EofException: Closed
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:90)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:213)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:206)
at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:136)

Any suggestions on how to go about it.

Thanks,
Satya

Re: Parallel SQL - column not found intermittent error

[Joel Bernstein]

Are you able to reproduce the error, or is it just appearing in the logs?

Do you know the state of index when it's occurring?

Joel Bernstein
http://joelsolr.blogspot.com/

On Wed, Jun 14, 2017 at 11:09 AM, Susheel Kumar 
wrote:

> I have setup Solr-6.6-0 on local (local ZK and Solr) and then on servers (3
> ZK and 2 machines, 2 shards) and on both the env,  i see this intermittent
> error "column not found". The same query works sometime and other time
> fails.
>
> Is that a bug or am I missing something...
>
>
> Console
> ===
>
> -> solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15'
> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>
> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
> table","EOF":true,"RESPONSE_TIME":78}]}}
>
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15'
> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>
> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
> table","EOF":true,"RESPONSE_TIME":92}]}}
>
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15'
> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>
> {"result-set":{"docs":[{"firstName":"Thiago","lastName"
> :"Diego"},{"firstName":"John","lastName":"Jagger"},{"
> firstName":"John","lastName":"Jagger"},{"firstName":"John","
> lastName":"Johny"},{"firstName":"Isabel","lastName"
> :"Margret"},{"firstName":"Isabel","lastName":"Margret"},
> {"EOF":true,"RESPONSE_TIME":191}]}}
>
> ➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
> firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
> dv_sv_userLastName LIMIT 15'
> http://server17:8984/solr/collection1/sql\?aggregationMode\=facet
>
> {"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
> to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
> table","EOF":true,"RESPONSE_TIME":83}]}}
>
> ➜  solr-6.6.0
>
>
> solr.log entry
> ===
>
> 2017-06-14 14:57:29.624 ERROR (qtp834133664-17) [c:collection1 s:shard1
> r:core_node1 x:collection1_shard1_replica1] o.a.c.r.CalciteException
> org.apache.calcite.sql.validate.SqlValidatorException: Column
> 'dv_sv_userFirstName' not found in any table
> 2017-06-14 14:57:29.625 ERROR (qtp834133664-17) [c:collection1 s:shard1
> r:core_node1 x:collection1_shard1_replica1] o.a.c.r.CalciteException
> org.apache.calcite.runtime.CalciteContextException: From line 1, column 9
> to line 1, column 27: Column 'dv_sv_userFirstName' not found in any table
> 2017-06-14 14:57:29.628 ERROR (qtp834133664-17) [c:collection1 s:shard1
> r:core_node1 x:collection1_shard1_replica1] o.a.s.c.s.i.s.ExceptionStream
> java.io.IOException: Failed to execute sqlQuery 'SELECT
>  dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
> collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
> 'jdbc:calcitesolr:'.
> Error while executing SQL "SELECT  dv_sv_userFirstName as firstName,
> dv_sv_userLastName as lastName FROM collection1 ORDEr BY dv_sv_userLastName
> LIMIT 15": From line 1, column 9 to line 1, column 27: Column
> 'dv_sv_userFirstName' not found in any table
> at
> org.apache.solr.client.solrj.io.stream.JDBCStream.open(
> JDBCStream.java:216)
> at
> org.apache.solr.client.solrj.io.stream.ExceptionStream.
> open(ExceptionStream.java:51)
> at
> 

Multiple hashJoin or innerJoin

[Zheng Lin Edwin Yeo]

Hi,

I'm using Solr 6.5.1.

Is it possible to have multiple hashJoin or innerJoin in the query?

An example will be something like this for innerJoin:

innerJoin(innerJoin(
  search(people, q=*:*, fl="personId,name", sort="personId asc"),
  search(pets, q=type:cat, fl="personId,petName", sort="personId asc"),
  on="personId"
),
  search(collection1, q=*:*, fl="personId,personName", sort="personId asc"),
)

Regards,
Edwin

Parallel SQL - column not found intermittent error

[Susheel Kumar]

I have setup Solr-6.6-0 on local (local ZK and Solr) and then on servers (3
ZK and 2 machines, 2 shards) and on both the env,  i see this intermittent
error "column not found". The same query works sometime and other time
fails.

Is that a bug or am I missing something...


Console
===

-> solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
dv_sv_userLastName LIMIT 15'
http://server17:8984/solr/collection1/sql\?aggregationMode\=facet

{"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
 dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
 dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
table","EOF":true,"RESPONSE_TIME":78}]}}

➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
dv_sv_userLastName LIMIT 15'
http://server17:8984/solr/collection1/sql\?aggregationMode\=facet

{"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
 dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
 dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
table","EOF":true,"RESPONSE_TIME":92}]}}

➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
dv_sv_userLastName LIMIT 15'
http://server17:8984/solr/collection1/sql\?aggregationMode\=facet

{"result-set":{"docs":[{"firstName":"Thiago","lastName":"Diego"},{"firstName":"John","lastName":"Jagger"},{"firstName":"John","lastName":"Jagger"},{"firstName":"John","lastName":"Johny"},{"firstName":"Isabel","lastName":"Margret"},{"firstName":"Isabel","lastName":"Margret"},{"EOF":true,"RESPONSE_TIME":191}]}}

➜  solr-6.6.0 curl --data-urlencode 'stmt=SELECT  dv_sv_userFirstName as
firstName, dv_sv_userLastName as lastName FROM collection1 ORDEr BY
dv_sv_userLastName LIMIT 15'
http://server17:8984/solr/collection1/sql\?aggregationMode\=facet

{"result-set":{"docs":[{"EXCEPTION":"Failed to execute sqlQuery 'SELECT
 dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
'jdbc:calcitesolr:'.\nError while executing SQL \"SELECT
 dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15\": From line 1, column 9
to line 1, column 27: Column 'dv_sv_userFirstName' not found in any
table","EOF":true,"RESPONSE_TIME":83}]}}

➜  solr-6.6.0


solr.log entry
===

2017-06-14 14:57:29.624 ERROR (qtp834133664-17) [c:collection1 s:shard1
r:core_node1 x:collection1_shard1_replica1] o.a.c.r.CalciteException
org.apache.calcite.sql.validate.SqlValidatorException: Column
'dv_sv_userFirstName' not found in any table
2017-06-14 14:57:29.625 ERROR (qtp834133664-17) [c:collection1 s:shard1
r:core_node1 x:collection1_shard1_replica1] o.a.c.r.CalciteException
org.apache.calcite.runtime.CalciteContextException: From line 1, column 9
to line 1, column 27: Column 'dv_sv_userFirstName' not found in any table
2017-06-14 14:57:29.628 ERROR (qtp834133664-17) [c:collection1 s:shard1
r:core_node1 x:collection1_shard1_replica1] o.a.s.c.s.i.s.ExceptionStream
java.io.IOException: Failed to execute sqlQuery 'SELECT
 dv_sv_userFirstName as firstName, dv_sv_userLastName as lastName FROM
collection1 ORDEr BY dv_sv_userLastName LIMIT 15' against JDBC connection
'jdbc:calcitesolr:'.
Error while executing SQL "SELECT  dv_sv_userFirstName as firstName,
dv_sv_userLastName as lastName FROM collection1 ORDEr BY dv_sv_userLastName
LIMIT 15": From line 1, column 9 to line 1, column 27: Column
'dv_sv_userFirstName' not found in any table
at
org.apache.solr.client.solrj.io.stream.JDBCStream.open(JDBCStream.java:216)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.open(ExceptionStream.java:51)
at
org.apache.solr.handler.StreamHandler$TimerStream.open(StreamHandler.java:535)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:78)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:193)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at

Solr NLS custom query parser

[aruninfo100]

Hi,

I am trying to configure NLP search with Solr. I am using OpenNLP for the
same.I am able to index the documents and extract named entities and POS
using OpenNLP-UIMA support and also by using a UIMA Update request processor
chain.But I am not able to write a query parser for the same.Is there a
query parser already written to satisfy the above features(nlp search).

Thanks and Regards,
Arun



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-NLS-custom-query-parser-tp4340511.html
Sent from the Solr - User mailing list archive at Nabble.com.

install solr service possible bug

[Susheel Kumar]

Hi,

Can anyone confirm if this  "service --version" command works ? For me to
install in SUSE distribution,  "service --version" commands always fail and
abort the solr installation with printing the error  "Script requires the
'service' command"

To make it work, i had to change "service --version" to "service --help".

If someone can confirm, i'll raise a JIRA to have this minor fix.

install_solr_service.sh =>  Solr 6.6.0
---
---
---
if [[ $SOLR_START == "true" ]] ; then
  service --version &>/dev/null || print_error "Script requires the
'service' command"
  java -version &>/dev/null || print_error "Solr requires java, please
install or set JAVA_HOME properly"
fi
lsof -h &>/dev/null || echo "We recommend installing the 'lsof'
command for more stable start/stop of Solr"

Re: Multi tenant setup

[Susheel Kumar]

I'll suggest to raise a JIRA and link to
https://issues.apache.org/jira/browse/SOLR-7759 but before that see if
updating the settings in Solrcofig for statsCache as described works here
https://issues.apache.org/jira/browse/SOLR-1632

Thanks,
Susheel

On Tue, Jun 13, 2017 at 5:16 PM, Zisis T.  wrote:

> We are talking about fewer collections,so that won't be an issue.
>
> The problem comes when - using the proposed setup - I want to send a query
> across all those collections and get properly ranked results. Each
> collection has its own IDF etc, so the scores are not comparable. This
> means
> that most probably results from one collection will dominate the results.
>
> This led me to try the /DistributedIDF/ configuration but this did not work
> either due to the issues described in the link of the original post.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Multi-tenant-setup-tp4340377p4340421.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Setting q parameter for fetch Streaming expression

[Zheng Lin Edwin Yeo]

Hi Joel,

Thanks for your reply.

For the "having" expression, it says that it requires a BooleanEvaluator.
Which means I can only use it to filter if there is Boolean Evaluator, and
not if it is just a normal query?

And yes, it will be good to support filter queries with fetch. Should I put
that as a possible improvement in Jira?

Regards,
Edwin


On 13 June 2017 at 23:42, Joel Bernstein  wrote:

> Currently you cannot specify a query for fetch. You can filter tuples
> emitted by fetch by wrapping it in a "having" expression.
>
> In the future I think it makes sense to support filter queries with fetch.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Tue, Jun 13, 2017 at 4:46 AM, Zheng Lin Edwin Yeo  >
> wrote:
>
> > Hi,
> >
> > For the Streaming expression on Fetch, is it possible to have set the q
> > parameter for the "addresses" collection?
> > In the below example from the Solr Documentation, it is only setting the
> q
> > parameter for the "people" collection.
> >
> > I'm using Solr 6.5.1.
> >
> > fetch(addresses,
> >   search(people, q="*:*", fl="username, firstName, lastName",
> > sort="username
> > asc"),
> >   fl="streetAddress, city, state, country, zip",
> >   on="username=userId")
> >
> > Regards,
> > Edwin
> >
>